Aerial Images Meet Crowdsourced Trajectories: A New Approach to Robust Road Extraction

Earth science [1, 2] is a crucial subject that studies the physical, chemical, and biological characterizations of our earth for better understanding various physical phenomena and natural systems. Earth science is also a complex subject and it contains a lot of research branches [30]. For instance, meteorologists [31] study the atmosphere for dangerous storm warnings and hydrologists [32] examine hydrosphere for flood warnings. Seismologists [33] study earthquakes and forecast where they will strike, while geologists [34] study rocks and help to locate useful minerals. Among all the subbranches of geoscience, geoinformatics [3] recently has attracted widespread interests with the rapid development of satellite and computer technologies, since it can greatly facilitate other research branches, e.g., monitoring storm/flood from remote sensing data and forecasting their evolutionary trend. In this work, we inherit the research content of geoinformatics and apply computer technologies to land remote sensing analysis, e.g, extracting the traffic road network from aerial images and some complementary modalities. This problem has important applications in transportation navigation and public management. Moreover, we can also compare the road networks at different times and estimate the urban expansion tendency, thereby analyzing the potential impacts of human activities on earth lands.

As a crucial foundation in intelligent transportation systems, automatic road extraction has been studied for decades [35]. On the basis of the modality of input data, previous approaches can be divided into four categories and we would investigate the related works of each category.

In the field of machine learning, message passing [62, 63] refers to information interactions between different entities. A large number of works have shown that such a mechanism can effectively facilitate deep representation learning. For instance, Wang et al. [64] introduced an inter-view message passing module to enhance the view-specific features for action recognition, while Liu et al. [65] propagated information among multiscale features to model the scale variations of people. In graph convolution networks, the message passing mechanism is usually embedded to aggregate information from neighboring nodes [66, 67, 68, 69, 70]. Recently, this mechanism has also been adopted for cross-modal representation learning. For instance, Wang et al. [71] addressed the text-image retrieval problem by transferring multimodal features and computing their matching scores. Nevertheless, most of these previous methods propagated information in a local manner (e.g., at short range). Without capturing global information, these methods may fail to discover the occluded roads that meanwhile have very few trajectories, as shown in Fig. 1-(d). Therefore, more effective approaches are desired to fully exploit the complementary information of aerial images and crowdsourced trajectories for traffic road extraction.

As mentioned above, aerial images and vehicle crowdsourced trajectories are complementary for traffic road extraction. To recognize unconstrained roads effectively and robustly, we propose a Cross-Modal Message Propagation Network (CMMPNet), which mutually enhances the hierarchical features of different modalities for better capturing their complementary information. As shown in Fig. 3, our CMMPNet is composed of (i) two deep AutoEncoders for modality-specific feature learning and (ii) a Dual Enhancement Module (DEM) for cross-modal feature refinement. In this subsection, we mainly introduce the architecture of CMMPNet, whose specific components are described in the following subsections.

Specifically, given an aerial imagery $I$ and a trajectory heat-map $T$ with a resolution ${H{\times}W}$, we first explicitly learn modality-specific representations by feeding them into different AutoEncoders, each of which consists of four encoding blocks and four decoding blocks. As shown in Fig. 3, the first AutoEncoder takes $I$ as input and extracts a group of image features:

where the first four features are the outputs of encoding blocks and the remaining four features are the output of decoding blocks. With the same architecture, the second AutoEncoder extracts a group of trajectory features:

from the input trajectory heat-map $T$.

Rather than directly fuse image and trajectory features with concatenation [28] or weighted addition [29], we fully capture the multimodal complementary information through enhancing their features mutually with a message passing mechanism. For each pair of multimodal feature $\{f_{I}^{i},f_{T}^{i}\}$, we employ the proposed DEM to generate two enhanced features $\{\widehat{f}_{I}^{i},\widehat{f}_{T}^{i}\}$ with their complementary information. This process can be formulated as:

These enhanced features are then fed into the next block of individual AutoEncoder respectively for further representation learning. For convenience, the final outputs of image AutoEncoder and trajectory AutoEncoder are denoted as $f_{I}^{o}$ and $f_{T}^{o}$, and they have the same resolution ${H{\times}W}$. Here, $f_{I}^{o}$ and $f_{T}^{o}$ are jointly utilized to predict a probability map $M\in R^{H{\times}W}$ for traffic roads with the following formulation:

where $\oplus$ denotes feature concatenation and $\mathbb{W}_{1*1}$ refers to the parameters of a 1*1 convolutional layer. For each position $(x,y)$, it can be regarded as a road region only when $M(x,y)$ is greater than a given threshold.

It’s worth noting that our method is universal for multi-modal road extraction. Except for image+trajectory data, the proposed CMMPNet can also be directly employed to recognize traffic roads with image+lidar data. The university of our method would be verified in Section V and VI.

In the previous work [27], aerial images and trajectory heat-maps were directly concatenated to feed into the same network, which caused that their features were over mixed and their complementarities were missed to some extent. To address this problem, we feed the given aerial image and the corresponding trajectory heat-map into different networks to learn modality-specific features. Optimized with individual parameters, these features well preserve the specific information of each modality, thus can be further utilized for mutual refinement.

To maintain the high resolution of final outputs, two AutoEncoders are adopted intentionally to extract modality-specific features. Notice that various AutoEncoders (e.g, Res-UNet [43], LinkNet [45], and D-LinkNet [44]) are suitable to serve as the backbone network of our framework. Since these networks have similar architectures, we take D-LinkNet based AutoEncoder as an example to demonstrate the details of modality-specific feature learning. As shown in Table I, both the image AutoEncoder and trajectory AutoEncoder are mainly composed of four encoding blocks and four decoding blocks. Specifically, we first use a convolutional layer to extract initial features and then feed them into the following four encoding blocks, each of which consists of a $2\times 2$ max-pooling layer and multiple residual units. As shown in Fig. 4-(a), each residual unit contains two $3\times 3$ convolutional layers and a skip layer. After the encoding stage, an interim unit is adopted to capture more spatial context by expanding the receptive field with four dilated convolutional layers. At the decoding stage, four decoding blocks are utilized to enlarge the resolutions of features progressively. Specifically, each decoding block is developed as an upsampling unit, which consists of two convolutional layers for channel adjustment and a transposed convolutional layer for feature upsampling, as shown in Fig. 4-(c). To simultaneously exploit the lower-level information and high-level information, we incorporate the features of encoding blocks and decoding blocks with element-wise addition. Finally, we fully restore the resolution of feature to $H{\times}W$ with a transposed convolution and apply a $3\times 3$ convolutional layer to generate the final modality-specific feature $f^{o}\in R^{H{\times}W{\times}32}$. Note that our image AutoEncoder and trajectory AutoEncoder have individual parameters, thus they can effectively capture and preserve the specific information of each modality.

After modality-specific feature learning, we refine these features mutually with a Dual Enhancement Module (DEM) based on the message passing mechanism. In this module, a Non-Local Message Propagator and a Gated Message Propagator are integrated to dynamically transmit the local and global message of each unimodal feature to complement the feature of another modality. Absorbing the complementary information of other modalities, each unimodal feature becomes more reasonable and robust. In this subsection, we take the refinement of features $f^{i}_{I}$ and $f^{i}_{T}\in R^{h{\times}w{\times}c}$ as an example to demonstrate the working mechanism of the tailor-designed DEM. Note that $h$, $w$, and $c$ are the height, width, and channel number of these features.

In this work, we implement the proposed CMMPNet on the representative deep learning platform PyTorch [79]. First, we perform data augmentation to alleviate the overfitting issue. Specifically, all training samples including the satellite images, trajectory heat-maps and ground-truth maps are i) flipped horizontally or vertically, ii) rotated by 90, 180, 270 degrees, iii) randomly cropped with a size range of [0.7, 0.9] and resized to the original resolution. After augmentation, the number of training samples is enlarged by 7 times. We then determine the hyper-parameters of our framework. The filter weights of all convolutional layers and Fully-Connected layers are uniformly initialized by Xavier [80]. The batch size is set to 4 and the learning rate is set to 0.0002. Finally, we apply the Adam [81] optimizer to train our CMMPNet for 30 epochs by minimizing the Binary Cross-Entropy Loss between the generated road maps and the corresponding ground-truth maps.

Datasets: In this work, our experiments are mainly conducted on the BJRoad dataset [27], which is captured in Beijing, China. Specifically, this benchmark consists of 350 high-resolution aerial images that cover a large geographic area of about 100 square kilometers and around 50 million trajectory records of 28 thousand vehicles. The resolution of aerial images is $1024\times 1024$ and each pixel denotes a 0.5m$\times$0.5m region in the real world. For each aerial image, a $1024\times 1024$ trajectory heat-map is generated with the preprocessing described in Section III, and the corresponding ground-truth (GT) map is manually created by masking out the pixel of traffic roads. Finally, this dataset is officially divided into three partitions: 70% samples are adopted for training, 10% for validation, and the rest 20% for testing.

Following the previous work [29], we also perform experiments on the Porto dataset, which covers a geographic area of about 209 square kilometers in Porto, Portugal. This dataset contains a mass of crowdsourced trajectories generated by 442 taxis from 2013 to 2014. On this dataset, we adopt a five-fold cross-validation setting, since the details of training/testing sets are not provided in [29]. Specifically, the aerial image of the whole area is first cut into 6,048 non-overlapping sub-images with a resolution of $512\times 512$. These sub-images are then randomly divided into five equal parts. For the $i$-th validation, the $i$-th part is used for testing, and the remaining parts are used for training. Finally, the mean and variance of five validations are reported.

Evaluation Details: Given a probability map $M$, we need to determine an estimated road map $M_{e}\in R^{H{\times}W}$ before evaluation. Same to [27], a pixel $(x,y)$ is predicted as a road region in our work, if the response value of $M(x,y)$ is greater than 0.5. Following previous works [82, 44], we adopt Intersection over Union¹¹1https://en.wikipedia.org/wiki/Jaccard_index (IoU) to evaluate the performance for road extraction. Specifically, the IoU score between an estimated map $M_{e}$ and its corresponding ground-truth map $M_{g}$ is computed by:

where $|M_{e}{\cap}M_{g}|$ denotes the pixel number in the intersection set of $M_{e}$ and $M_{g}$, and $|M_{e}{\cup}M_{g}|$ is the pixel number in their union set. There are two manners for computing the IoU of all testing samples. The first manner is to compute the IoU of each sample and then average the IoU of all samples. Such a metric is termed as average IoU (A_IoU). The second manner is to stitch the estimated maps of all samples into a global map and then compute an IoU score. This metric is termed as global IoU (G_IoU). Since different IoU metrics were used in previous works, we would report the results of both A_IoU and G_IoU in the following sections.

In this section, we compare our CMMPNet with seven deep learning-based approaches, including DeepLab (v3+) [83], UNet [61], Res-UNet [43], LinkNet [45], D-LinkNet [44], Sun et al. [27], and DeepDualMapper [29]. Specifically, these compared methods are reimplemented for multimodal road extraction. In particular, DeepDualMapper feeds aerial images and trajectory heat-maps into different backbone networks²²2In DeepDualMapper, the original backbone network is UNet. However, our reimplemented DeepDualMapper based on UNet performs poorly. Thus in this work, we adopt D-LinkNet as the backbone to reimplement DeepDualMapper and this model can obtain competitive performance on different datasets. and then fuses their features with a gated fusion module, while other methods directly take the concatenation of aerial images and trajectory heat-maps as input. Moreover, all the compared methods except DeepLab (v3+) and DeepDualMapper are equipped with 1D transpose convolution to better model traffic roads [27]. Notice that the first six compared methods were implemented by [27], and we utilize the official code of [27] to implement DeepDualMapper and our method with the same data partition. As mentioned above, our CMMPNet can be developed with various AutoEncoders. We hence evaluate multiple implementations of CMMPNet based on different AutoEncoders, such as Res-UNet, LinkNet, and D-LinkNet.

The performance of all methods on the BJRoad dataset is summarized in Table II. We can observe that DeepLab (v3+) obtains the worst A_IoU 50.81% probably because of parameter overfitting. Sun et al. [27] utilized various techniques (e.g., different sampling intervals and GPS augmentation) to obtain an improved A_IoU 59.18%. However, just directly feeding the concatenation of aerial images and trajectory heat-maps into networks, these methods have limited capabilities to capture the multimodal information, thus none of them can acquire an A_IoU above 60%. By fusing image and trajectory features with a gated fusion module, DeepDualMapper obtains a competitive A_IoU 60.91% and G_IoU 61.54%. Despite the progress, DeepDualMapper only use a fusion strategy rather than a mutual refinement strategy, thereby cannot address this task well. In contrast, when learning modality-specific features explicitly and enhancing cross-modal features mutually, our method can fully exploit the complementary information of aerial images and crowdsourced trajectories. For this reason, the proposed CMMPNet outperforms all previous methods with large margins. For instance, Res-UNet+CMMPNet achieves a competitive A_IoU 62.58% and obtains a relative improvement of 15.37%, compared with the original Res-UNet. By improving the A_IoU from 57.96% to 62.85%, our D-LinkNet+CMMPNet also obtains a substantial improvement of 8.4%, compared with the baseline D-LinkNet. Finally, with an impressive A_IoU 63.09% and a G_IoU 63.46%, our LinkNet+CMMPNet becomes the best-performing model. We notice that the performance of D-LinkNet+CMMPNet is slightly lower than that of LinkNet+CMMPNet. This is probably because D-LinkNet+CMMPNet contains two extra interim units and suffers from certain overfitting, although data augmentation has been performed.

Moreover, we compare the performance of our CMMPNet with three competitive models including D-LinkNet [44], Sun et al. [27], and DeepDualMapper [29] on the Porto dataset. As shown in Table III, all methods obtain much better results on this dataset, compared with their performance on the BJRoad dataset. The main reason is that the aerial images of Porto are clearer and the noises of trajectories are smaller [29]. Despite the existing benchmarks are high, our CMMPNet still can boost the IOU with substantial margins, ranking first in performance on the Porto dataset. In summary, these comparisons greatly demonstrate the effectiveness of the proposed CMMPNet for image+trajectory based road extraction.

After external comparison, we then perform extensive internal experiments to analyze the effectiveness of each module in the proposed CMMPNet. In this subsection, D-LinkNet is adopted as the backbone network and our implementation details have been described in Section IV-D.

The effect of global message: In previous works [84, 85], local information is widely adopted, but global information is neglected. In this section, we implement several variants of CMMPNet to verify the effectiveness of global information. As shown in Table IV, when propagating the global information extracted by SPP and FC layer, “Local+Global” model obtains an A_IoU 61.98% and a G_IoU 62.43%, and is better than “Local” model. With an A_IoU 62.85% and a G_IoU 63.39%, “Local+Global+Gate” model also outperforms “Local+Gate” model, whose A_IoU is 62.32% and G_IoU is 62.78%. Except for quantitative results, we also visualize some feature maps and traffic road maps generated by “Local+Gate” model and “Local+Global+Gate” model in Fig. 6. Note that those visualized feature maps are the channel-wise average of image features and trajectory features after DEM. We can observe that incorporating global information can generate more discriminative features and better recognize traffic roads, especially when the roads are occluded/unimpressive and the vehicle trajectories are rare in local regions. These quantitative and qualitative experiments show that global information is greatly effective for traffic road extraction.

The effect of Gated Message Propagator: In this propagator, multiple gate functions are employed to dynamically propagate the complementary information. In this subsection, we also implement several variants to verify the effectiveness of this mechanism. As shown in Table IV, after applying gate functions on “Local” model, A_IoU increases from 61.62% to 62.32% and G_IoU increases from 62.09% to 62.78%. Further, we can obtain a more substantial improvement (around 1% on both A_IoU and G_IoU), when performing gate functions on “Local+Global” model. These comparisons show that this proposed propagator can facilitate robust road extraction using multimodal information.

The configuration of Spatial Pyramid Pooling: In Non-Local Message Propagator, we employ a $N$-level SPP and an FC layer to extract global information. In this section, we explore the effect of the level number for road extraction using multimodal information. As shown in Fig. 8, when applying a global max-pooling ($N$=1), our CMMPNet obtains an A_IoU 62.38% and is slightly better than the “Local+Gate” model in Table IV, since global pooling can only provide some coarse and limited information. As the level number increases, the performance also gradually increases, and our method achieves the best A_IoU 62.85% and G_IoU 63.39% when $N$ is equal to 3. When $n$ increases to 4, the performance slightly drops, probably because of overfitting, i.e., the amount of parameters of the FC layer in DEM increases sharply as the level number increase. Therefore, the level number $N$ of Spatial Pyramid Pooling is uniformly set to 3 for road extraction.

Unimodal Data vs. Multi-modal Data: We first explore whether multimodal data is reliably useful for traffic road extraction. As shown in Table V, when only feeding trajectory heat-maps into a D-LinkNet, we obtain a poor performance (A_IoU 52.38%, G_IoU 52.90%) on the BJRoad dataset. When only utilizing aerial images, we obtain an A_IoU 59.79% and a G_IoU 60.24%, which indicates that image data is more crucial than trajectory data. In contrast, when using the aerial images and trajectory heat-maps simultaneously, our CMMPNet and the early/late fusion models described in the next paragraph outperform the unimodal models consistently with an improvement of at least 1% on IoU. This comparison demonstrates that multimodal data is more effective for traffic road extraction, because aerial images and vehicle crowdsourced trajectories have rich complementarities.

Which multimodal learning manner is better? We then explore the effects of different multimodal learning manners. Except for the proposed CMMPNet, we also implement another two commonly-used manners, i.e., early fusion model and late fusion model. Specifically, the former feeds the concatenation of aerial images and trajectory heat-maps into a D-LinkNet. In the latter, aerial images and trajectory heat-maps are respectively fed into individual D-LinkNet, and their final features are concatenated to estimate the road maps. As shown in Table V, the early fusion model obtains an A_IoU 61.11% and a G_IoU 61.53%, slightly outperforming the late fusion model (A_IoU 60.78%, G_IoU 61.24%). This is because the multimodal information is utilized at different layers in the former, but just utilized once in the latter. Compared with these two models, our CMMPNet is more reasonable to learn modality-specific features and propagate cross-modal information hierarchically. For this reason, our method achieves an impressive A_IoU 62.85% and G_IoU 63.39%, and outperforms early/late fusion models with a large margin. This comparison shows the effectiveness of our CMMPNet for multimodal representation learning.

Significance of crowdsourced trajectories: Although the IoU of aerial images is much better than that of trajectories, we argue that vehicle trajectories are crucial for the robustness of road extraction, especially when some cities (e.g, Chongqing and Chengdu, China) are greatly covered by fog and mist in aerial images. So here we explore to extract traffic roads from foggy images and crowdsourced trajectories. Since there are no foggy images in the BJRoad dataset, we need to generate some aerial images with heavy fog in advance. Specifically, for each cloudless image in BJRoad, we employ a fog effect renderer of Photoshop to generate a foggy image. After augmenting the training samples as described in Section IV-D, we reimplement the proposed CMMPNet and three other compared methods, including (1) two unimodal models which feed foggy images or trajectory heat-maps into D-LinkNet, and (2) an early fusion D-LinkNet model which takes the concatenation of foggy images and trajectory heat-maps as input.

The results of all methods are summarized in Table VI. We can observe that the unimodal D-LinkNet only obtains an A_IoU 54.54% and a G_IoU 55.27% when only using foggy images. Compared with the corresponding model using cloudless images, this model has a dramatic drop in performance, since traffic roads may be invisible in foggy images. When utilizing foggy images and trajectories simultaneously, the early fusion model obtains an A_IoU 57.98% and a G_IoU 58.49%. Based on the same D-LinkNet, our CMMPNet achieves a competitive A_IoU 60.45% and G_IoU 61.06%, having a performance improvement of at least 3% compared with other models. Moreover, the visualizations in Fig. 9 show that our CMMPNet can still generate high-quality road maps in foggy weather conditions. This is attributed to the fact that the vehicle trajectories can provide rich information to remedy the limitation of aerial images, and our method can fully capture their complementary information. In summary, crowdsourced trajectories are very crucial and beneficial for robust road extraction.

In this subsection, we compare our CMMPNet with six state-of-the-art methods on the TLCGIS dataset. The details of these compared methods are described as follows. SegNet [60]: As a fully convolutional AutoEncoder, SegNet takes the concatenation of aerial images and Lidar images as input. PreConv [59]: The Lidar images are first fed into a depth convolution unit (DepthCNN) implemented with two convolutional layers. The Lidar features and aerial images are then concatenated and fed into SegNet. RFNorm [59]: Given aerial and Lidar images, some Random Forest classifiers [86] are first trained to estimate the road probability score at each location. The aerial-based and Lidar-based score maps are concatenated and fed into SegNet. ElePreConv [59]: In this model, Lidar images are first encoded with two convolutions (8 and 4 filters), while aerial images are extended with an extra zero-initialized channel. The element-wise addition of 4-channel images and Lidar features are fed into FuseNet [87]. TriSeg [59]: This model consists of three SegNets. The first two SegNets respectively take aerial or Lidar images to generate the road probability maps, which are concatenated and fed into the third SegNet for final estimation. DeepDualMapper [29]: This model has been described above and here we adopt D-LinkNet as the backbone network to reimplement this model.

The performance of all approaches is summarized in Fig. 10. We can observe that the previous best-performing methods are TriSeg and DeepDualMapper, whose G_IoU are 78.4% and 81.1%, respectively. Thanks to the cross-modal mutual refinement strategy, our CMMPNet achieves a new state-of-the-art G_IoU 83.1% on the TLCGIS dataset and greatly outperforms DeepDualMapper with an absolute improvement of 2%. Moreover, we also visualize some results in Fig. 11. As can be observed, the traffic road maps generated by our method are more accurate in complex scenarios. In summary, these quantitative and qualitative comparisons demonstrate that our CMMPNet is universal and effective to extract traffic roads from aerial images and Lidar data.

In this subsection, we verify the effectiveness of each component in the proposed CMMPNet for image+Lidar based road extraction. We first explore which manner can better exploit the information of these modalities. As shown in Table VII, we can obtain a G_IoU of 69.12%, when only feeding the rendered Lidar images into D-LinkNet. When only using aerial images, the G_IoU of D-LinkNet is 80.96%, which indicates that aerial images are more important. Incorporating the information of aerial and Lidar images simultaneously, the early fusion model obtains a G_IoU of 81.62%, while the late fusion model has a comparable G_IoU of 81.50%. When fully exploring their complementary information with cross-modal message propagators, our CMMPNet achieves an impressive G_IoU 83.10%, outperforming the early/late fusion models with an absolute improvement of 1.5%. This demonstrates that the proposed CMMPNet can also effectively capture the complementary information among aerial images and Lidar data.

We then explore the effect of global information and gate functions. Similar to Section V-C, we implement several variants of CMMPNet. As shown in Table VIII, when only propagating local information with gate functions, “Local+Gate” model obtains a G_IoU 82.31%. When incorporating global information, “Local+Global+Gate” model has a better G_IoU 83.10%, which indicates that the global information is also useful for image+Lidar based road extraction. Moreover, by comparing the performance of “Local+Global” model and “Local+Global+Gate” model, we can observe that the gate functions help to make an absolute improvement of 1.04% on G_IoU, which also demonstrates the effectiveness of Gated Message Propagator for image+Lidar based road extraction.