CVLNet: Cross-View Semantic Correspondence Learning for Video-based Camera Localization

Prior methods often resort to a satellite to ground projection to bridge the cross-view domain gap. This is achieved either by a polar transform [6, 8, 10] or a projective transform [51, 25, 52]. However, both transforms need to know the query camera location with respect to the satellite image center. In the CVUSA and CVACT dataset where polar transform performs excellent, the query images accidentally align with their matching satellite image center, which however does not occur in practice. When there is a large offset between the real camera location and its assumed location with respect to its matching satellite image (e.g., satellite image center in polar transform), the performance will be impeded significntly. Hence, instead of projecting satellite images to ground views, we introduce a Geometry-driven View Projection (GVP) module to transform ground-view images to overhead view.

Starting from a blank canvas in the overhead view with its center corresponds to the geospatial location of the query camera, we aim to fill it with features collected from ground-view images. We set the origin of the world coordinate system to the geo-spatial query camera location as well, with its $x$-axis pointing to the south direction, $y$ axis pointing to the east direction, and the $z$-axis vertically upward. Different ground-view images in a video sequence are projected to the same overhead-view coordinate system so that they are geographically aligned after projection. Fig. 3 provides a visual illustration of the coordinate systems.

Denote $\mathbf{F}_{t_{i}}^{g}\in\mathbb{R}^{H\times W\times C}$ as ground-view image features by a CNN backbone, where $H$, $W$ and $C$ are the height, width and channels of the features, respectively, and $\text{GVP}(\cdot)$ as the geometry-driven view projection operation illustrated in Eq. (1). The projected features in overhead view are then obtained by $\mathbf{F}_{t_{i}}^{p}=\text{GVP}(\mathbf{F}_{t_{i}}^{g}),~{}~{}\mathbf{F}_{t_{i}}^{p}\in\mathbb{R}^{S\times S\times C}$, where $S$ indicates the overhead-view feature map resolution.

This projection establishes the exact geometric correspondences between the ground and overhead views for scene contents on the ground plane. For scene objects with higher heights, projecting features rather than image pixels can alleviate the strict constraint while providing a cue that corresponding objects exist between the views. As shown in Fig. 4, for pixel $(u^{s},v^{s})$, the projected feature from the ground-view at $t_{n-1}$ represents the tree trunk, but the feature in the satellite image corresponds to the tree canopy. Both tree canopy and tree trunk indicate there is a tree at location $p_{2}$. Then, by applying a matching loss between the two features (tree trunk and tree canopy), the network will be trained to learn viewpoint invariant features, i.e., both tree trunk and tree canopy are mapped to the semantic features of “tree”.

The coverage of the canvas for ground-to-satellite projection is set to the reference satellite image coverage, i.e., around $100\text{m}\times 100\text{m}$, with its center corresponding to the query camera location. When the sequence is too long with some previous image contents exceeding the canvas’s pre-set coverage, the exceeded contents will not be collected. This is because scene contents that are too far from the query camera location are less important for localization, and it is better to cover most of the synthetic overhead-view feature map by referencing satellite images.

We leverage photo consistency among different ground-views for the video sequence fusion. For a satellite pixel, when its corresponding features in several (more than two) ground views are similar, the existence of a scene object at this geographical location is highly reliable for these ground-views. We should highlight these corresponding features when generating descriptors for scene contents. Driven by this, we design a Photo-consistency Constrained Sequence Fusion (PCSF) module. Our PCSF module employs an attention mechanism [54] to emphasize reliable features in fusing a video sequence and obtaining a global descriptor for the video.

Our GVP block has aligned the original ground-view features at different time steps in a unified overhead-view coordinate. When the features of a geographical location observed by different ground views are similar, those features should be more reliable for localization. We leverage the self-attention mechanism [54] to measure the photo-consistency/similarity across different views and find reliable features. Specifically, for each projected feature map $\mathbf{F}^{p}_{t_{i}}$ at time step $t_{i}$, we compute its query, key and value by two stacked convolutional layers, denoted by $\mathbf{Q}_{t_{i}},\mathbf{K}_{t_{i}},\mathbf{V}_{t_{i}}\in\mathbb{R}^{S\times S\times C}$, respectively. The stacked convolutional layers increase the receptive field and the representative ability of the key, query, and value features at each spatial location. Next, we compute the similarities between each projected feature map at $t_{i}$ and other projected feature maps at $t_{j}$, $i,j=1,...,N$, and normalize them across all possible $j$ by a softmax operation, expressed as,

The final fused feature is obtained by,

In this way, we highlight the common features across the views and make the global descriptor reliable.

We want to address the location misalignment between a query camera location and its matching satellite image center by ground-to-satellite projection and spatial correlation between the projected features and the real satellite features. Hence, the satellite feature descriptors should be translational equivariant, which is an inherent property of conventional CNNs. Following most previous works [2, 6, 7, 8, 11], we use VGG16 [55] as our backbone for satellite (and ground) feature extraction. The extracted satellite features, denoted as $\mathbf{F}^{s}\in\mathbb{R}^{S\times S\times C}$, share the same spatial scale as the global representation of the query video. Next, we adopt a Normalized spatial Cross-Correlation (NCC) to estimate latent alignment between the query location and a satellite image center.

Denote $[\mathbf{F}^{s}]_{m,n}$ as a shifted version of a satellite feature map with its center at $(m,n)$ in the original satellite feature map, and $m=0,n=0$ correspond to the center of the original satellite feature map. The similarity between $\mathbf{F}^{s}$ and $\widetilde{\mathbf{F}}^{g}$ aligned at $(m,n)$ computed by NCC is,

where $\mathbf{D}_{0}(\mathbf{F}^{s},\widetilde{\mathbf{F}}^{g})\in\mathbb{R}^{h\times w}$ denotes the similarity matrix between $\mathbf{F}^{s}$ and $\widetilde{\mathbf{F}}^{g}$ at all possible spatial-aligned locations, $m\in[-\frac{h}{2},\frac{h}{2}]$, and $n\in[-\frac{w}{2},\frac{w}{2}]$. A potential spatial-aligned location of the satellite map lies in a region of $10\times 10\ \text{m}^{2}$ in our KITTI-CVL dataset, as the database satellite image is collected very ten meters.

To exclude impossible query camera locations, e.g., top of trees, we estimate an uncertainty map from the satellite semantic features, $\mathbf{U}(\mathbf{F}^{s})=\mathcal{U}(\mathbf{F}^{s})$, $\mathbf{U}(\mathbf{F}^{s})\in\mathbb{R}^{h\times w}$, where $\mathcal{U}(\cdot)$ is the uncertainty net, composed of a set of convolutional layers. The value of each element in $\mathbf{U}(\mathbf{F}^{s})$ is within the range of $[0,1]$, forced by a Sigmoid layer. By encoding the uncertainty, The similarity between $\mathbf{F}^{s}$ and $\widetilde{\mathbf{F}}^{g}$ aligned at $(m,n)$ is then written as,

When the uncertainty at $(m,n)$ is large, the similarity between $\mathbf{F}^{s}$ and $\widetilde{\mathbf{F}}^{g}$ aligned at this location will be decreased. We do not have explicit supervisions for the uncertainty map. Rather, it is learned statistically from training. The relative displacement between $\mathbf{F}^{s}$ and $\widetilde{\mathbf{F}}^{g}$ is obtained by,

During inference, we have no idea which one is the matching reference image for a query image. Thus the uncertainty-guided similarity matching is applied to all reference features (including non-matching ones). Furthermore, it is more challenging when a similarity score between non-matching ground and satellite features is high. Hence, we apply the similarity matching scheme to the pairs of query and non-matching reference images as well during training and minimize their maximum similarity, making the learned features more discriminative.

We employ the soft-weighted triplet loss [2] to train our network. The loss includes a positive term to maximize the similarity between the matching query and reference pairs and a negative term to minimize the similarity between non-matching pairs. The non-matching term also prevents our view projection module from trivial solutions. Therefore, it is formulated as,

where $\mathbf{F}^{s}$ is the matching satellite image feature to the ground feature $\mathbf{F}^{g}$, $\mathbf{F}^{s^{*}}$ is the non-matching satellite image feature, $d(\cdot,\cdot)$ is the ${L}_{2}$ distance between its two inputs after alignment, and $\alpha$ is set to 10.

Since there are no existing video-based cross-view localization algorithms, we conduct extensive experiments to dissect the effectiveness and necessity of each component in our framework.

Single image-based localization is a special case of video-based localization, i.e., when the image frame count in the video is one. In this section, we compare the performance of our method with the recent state-of-the-art (SOTA) that are invented for cross-view single image-based localization, including CVM-NET [2], CVFT [7], SAFA [6], Polar-SAFA [6], DSM [8], Zhu et al. [11], and Toker et al. [10]. The results are presented in Tab. 5. It can be seen that our method significantly outperforms the recent SOTA algorithms.

Among the compared algorithms, DSM [8] achieves the best performance, because it explicitly addresses the challenge of limited FoV problem of query images while the others assume that query images are full FoV panoramas. By comparing SAFA and Polar-SAFA, we can observe that the polar transform boosts the performance on Test-1 (one-to-one matching) while impairs the performance on Test-2 (one-to-many matching). This is consistent with the conclusion in Shi et al. [6] and Zhu et al. [11].

Based on SAFA, Zhu et al. [11] proposes two training losses: (1) an IoU loss and (2) a GPS loss. However, we do not found the two items work well on the KITTI-CVL dataset. We guess that the IoU loss is only suitable for the panorama case. The limited FoV images in the KITTI-CVL dataset have a smaller overlap with satellite images than panoramas, and thus the original IoU loss may not provide correct guidance for training. The GPS loss does not help mainly because of the inaccuracy of the GPS data in our dataset. We provide the GPS accuracy analysis of the KITTI dataset in the supplementary material. In contrast, our method does not rely on the accurate GPS tags of ground or satellite images.

Our method assumes that the north direction is provided by a compass, following previous works [3, 6, 8, 10, 11], and the absolute scale of camera translations can be estimated roughly from the vehicle velocity. We have not investigated how significant tilt and roll angle changes will affect the performance, because the tilt and roll angles in the KITTI dataset are very small and we set them to zero. In autonomous driving scenarios, the vehicle-mounted cameras are usually perpendicular to the ground plane. Thus there are only slight changes in tilt and toll during driving.