Revisiting Radar Camera Alignment by Contrastive Learning for 3D Object Detection

The central focus of recent works on 3D object detection has shifted from PV-based methods [13, 14, 15, 16, 17] to BEV-based methods [1, 3, 18, 19, 20, 21] due to various challenges encountered in PV, such as occlusion, viewpoint transformation and so on. The BEV-based methods can be divided into dense BEV-based methods [4, 20, 22, 21, 23, 24, 25], and query-based methods [2, 3, 18, 19, 26]. The dense BEV-based methods usually employ the Lift-Splat-Shoot (LSS)[27] to transform image features into BEV space. The BEVDet series [1, 28, 29] successfully introduces LSS to 3D object detection and achieves excellent performance. Considering the inaccuracy of LSS for depth estimation, BEVDepth [21] introduces lidar to supervise the predicted depth. After that, in order to get more accurate information about speeds, the temporal information [28, 20, 21] is introduced and further improves the performance of 3D object detection. The query-based methods usually predefine some sparse queries to model the objects and then use the queries to index the related image features. After the introduction of sparse queries for 3D object detection by DETR3D [19] firstly, numerous methods based on sparse queries have been proposed. PETR [2] uses 3D positional encoding to transform image features into 3D position-aware features for end-to-end 3D object detection. In StreamPETR [3], a memory queue which also consists of sparse queries is designed to store temporal information. Considering that query-based methods do not need to model the background and are more efficient compared to dense BEV-based methods. In the work, we design the RCAlign based on the query-based methods.

The radar excels in acquiring both the Doppler velocity and distance of an object, remains virtually unaffected by challenging environmental conditions and is cost-effective. Therefore, radar can serve as an excellent auxiliary sensor to the camera for low-cost 3D object detection. Recently, 3D object detection algorithms based on radar and camera fusion [11, 8, 30, 31, 32, 33] have demonstrated to be effective and have achieved remarkable performance. Similar to the research direction in 3D object detection for multi-view images, researchers initially focused on studying PV space[30, 31, 32]. They establish methods for associating radar points with objects in images. The CenterFusion [30] uses the frustum to associate the radar points with the object. Considering the inaccuracy of the radar projection position, RADIANT [31] corrects the radar point projection position and designs specific rules to associate radar points with objects in the image using Euclidean coordinates, while CRAFT [32] uses polar coordinates for this association. For the BEV-based methods, in order to obtain more accurate image BEV features, the CRN [11] utilizes radar assistance to transform image features from PV into BEV. Afterwards they concat the radar features and image features to serve as sparse queries and employ deformable attention to align features from both modalities. Different from the CRN of obtaining sparse queries, the SparseFusion3D [10] uses pre-defined sparse queries as an intermediate state to align the features from the two modalities. This type of method does not perform inter-modal interactions during the alignment process. RCBEVDet [8] designs a new backbone to extract radar features, and then uses radar features or image features as sparse queries to query features of another modality to align the features. However, they fail to achieve true alignment between the features from different modalities. In order to alleviate the above problems, we design a dual-route alignment module with contrastive loss which effectively aligns features representing the same object across modalities.

The contrastive learning [34, 35, 36] aims to pull samples of the same class closer while pushing samples of different classes further apart. It has been applied to the field of object detection to pull in features that represent the same object. SoCo [37] adopts contrastive learning to maximise the similarity of features representing the same object acquired through different augmentation methods. In order to obtain contrastive-aware proposal embeddings, FSCE [38] designs contrastive branch using contrastive learning to maximize the within-category agreement and cross-category disagreement. CAT-Det [39] utilises contrastive learning for data enhancement at the point and object levels, significantly improving detection accuracy. Thus, using contrastive learning enables the similarity of features that represent the same object. Here, we utilize the comparative learning to align the updated sparse queries following the two routing paths.

As shown in Fig. 2, the proposed RCAlign includes an image backbone, a radar backbone, a radar head, a dual-route alignment module, a radar feature enhancement module and a 3D detection head. During the multimodal feature extraction stage, the image backbone and the radar backbone are used to extract multi-view image PV features and radar BEV features respectively. Subsequently, the radar BEV features are passed through the radar head to get the centre of 3D boxes and radar heatmaps. Then, the radar points corresponding to the top $k$ probability values obtained from the radar heatmaps are taken out as part of the sparse queries. In addition to radar queries, sparse queries also contain initial queries and temporal queries. The sparse queries, image PV features and radar BEV features are input into the proposed DRA module to align and fuse the features of the two modalities. After that, the fused features are used to predict the categories and 3D boxes. Finally, considering the sparsity of the radar BEV features, we project the centre of the predicted 3D boxes into the BEV grid as the occupancy features, the occupancy features and the radar BEV features are fed into the RFE module to get the enhanced radar BEV features. The enhanced radar features are used to distill the original radar features, guiding the network to optimize for acquiring denser radar features.

For image branch, given the images $I\in R^{N\times C\times H\times W}$ from $N$ views, we use the backbone and the FPN [40] to extract multi-view image PV features. For radar branch, given the radar data corresponding to the image, we first aggregate five consecutive frames of radar data. During the aggregation process, we introduce velocity to compensate for the radar points position coordinates in the previous frames. Specifically for the radar point at the $t$-1 frames, we compensate the position using the following equation:

where $p_{xy}^{t}$ and $p_{xy}^{t-1}$ denote $x,y$ coordinates of the radar point at time $t$ and $t-1$, $v_{xy}^{t-1}$ and $\Delta{t}$ denote the velocity of the radar point at moment $t-1$ and the time interval between $t$ and $t-1$, respectively. Finally, we use PointPillar [41] to extract radar BEV features.

In order to better align the features of the two modalities and to perform the inter-modal features interaction during the alignment process, we propose the Dual-Route Alignment (DRA) moudle. The DRA achieves alignment and fusion by aggregating image PV features, radar BEV features to the sparse queries. As shown in Fig. 3 (a), DRA includes two routing paths, each undergoing two deformable attention [12] for inter-modal features interaction. Subsequently, the updated sparse queries from the two paths are aligned and fused using the fusion block. Specifically, for the above path, given the sparse queries $z_{q}$, the radar BEV features $f_{r}$ and the reference point $p_{q}$. The process of aggregate radar BEV features can be formulated as:

where $M$ and $K$ indicate the number of attention heads and sample points respectively. $A_{mqk}$ and $\Delta p_{mqk}$ stand for attention weight and sampling offset respectively. $W_{m}$ and $W_{m}^{\dagger}$ denote the weights of the linear. We denote the updated sparse queries as $z_{r}$. For the image PV features $f_{i}$, the aggregation of $f_{i}$ using $z_{r}$ can be represented as follows:

where $W_{m}^{\prime}$ and $W_{m}^{\dagger\prime}$ indicate the weights of the linear, $A_{mqk}^{\prime}$ and $\Delta p_{mqk}^{\prime}$ indicate attention weight and sampling offset respectively. The updated sparse queries are denoted as $z_{ri}$. For the below path, the same operation is performed, but the image PV features are aggregated before the radar BEV features. The updated sparse queries are denoted as $z_{ir}$. After the aforementioned operation, $z_{ri}$ and $z_{ir}$ are intended to represent the same object at corresponding positions, and their features should have similar semantics. Therefore, we introduce the contrastive learning [42] to align the features of the corresponding positions of $z_{ri}$ and $z_{ir}$, which can be calculated as:

where $\widehat{z}_{ri}$ and $\widehat{z}_{ir}$ stand for the normalized features. $\otimes$, $\tau$, and $I$ denote matrix multiplication, logit scale, and target matrix respectively. $\mathcal{L}_{CE}$ represents the cross-entropy loss. Finally, we use element-wise addition to obtain fusion queries $z_{f}$, which are utilized for 3D object detection task.

The radar data is obtained by aggregating five consecutive frames, but they are still sparse compared to the lidar. Considering that the number of fused queries exceeds the count of ground truth boxes by a considerable margin, there will be cases where multiple queries predict the same box. Due to most of the predicted boxes representing the same object having low classification probabilities, they have some bias relative to the ground truth boxes but also surround them. Therefore, it is feasible to utilize the predicted centre of the 3D boxes for enhancing radar features. Based on this, we designed the radar feature enhancement module.

As illustrated in Fig. 3 (b), the original radar BEV features and occupancy features are fed into RFE to obtain the enhanced radar features. Specifically, we first transform the centre of 3D boxes predicted by the fusion queries $z_{f}$ into BEV grids to get the occupancy features. Later, the radar BEV features and occupancy features are concatenated on the channel dimension, and the concatenated features are fed into a 3-layer conv block to obtain the enhanced radar BEV features. The conv block is composed of a convolutional layer, followed by batch normalization [43], and ReLU [44]. After obtaining the enhanced radar BEV features, on the one hand, they are input to the radar head as the second auxiliary task, sharing parameters with the radar head mentioned in Sec. III-A. On the other hand, they are used to distil the original radar features by knowledge distillation [45], guiding the network to optimize towards the direction where enhanced radar features can be acquired. The knowledge distillation process can be calculated as follows:

where $H$, $W$ and $C$ represent the height, width, and the number of channels of the radar BEV features, respectively. $f_{er}$ denotes enhanced radar BEV features.

Here, we conclude the loss function including the 3D object detection task losses $\mathcal{L}_{task}$, the losses corresponding to the two radar heads $\mathcal{L}_{RH1}$ and $\mathcal{L}_{RH2}$, the contrastive loss $\mathcal{L}_{CL}$ and the knowledge distillation loss $\mathcal{L}_{KD}$. The final optimization objective can be expressed as:

where $\lambda_{1},\lambda_{2},\lambda_{3}$ and $\lambda_{4}$ are hyper-parameters.

Datasets and Metrics. The nuScenes dataset [49] is a publicly available large-scale dataset for autonomous driving, comprising 1000 driving scenes. Among these, 750 scenes are designated for training, 150 for validation, and the remaining 150 for testing. Each scene in the nuScenes dataset spans a duration of 20 seconds and is annotated using a frequency of 2 Hz. The data in each scene is captured using 6 cameras that provide a full 360-degree field of view, along with 5 radars and 1 lidar. The annotated 3D boxes are categorized into 23 classes initially and aggregated into 10 categories for the 3D object detection task during evaluation. The mean Average Precision (mAP), nuScenes Detection Score (NDS), mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE) and mean Average Attribute Error (mAAE) are used as the evaluation metrics for 3D object detection. We compare the Average Multi-Object Tracking Accuracy (AMOTA), Average Multi-Object Tracking Precision (AMOTP), number of False Positives (FP), number of False Negatives (FN) and number of Identity Switches (IDS) with those of other methods for 3D object tracking.

Implementation Details. We use ResNet50 [50], ResNet101 and V2-99 [51] backbones to extract image features. The R50 and R101 are used for nuScenes val set, which are pre-train on ImageNet [52] and nuImages [49]. The V2-99 is used for nuScenes test set and initialized from DD3D [53]. We use the [$x$, $y$, $z$, $vx_{comp}$, $vy_{comp}$, $timestamp$] as the radar features. The point cloud range is set to [-51.2m, 51.2m] for X and Y axis, and [-5.0m, 3.0m] for Z axis. The radar backbone follows the architecture of FUTR3D [9]. The radar head adopts CenterNet [14], and as an auxiliary task, we simplify it by retaining only the prediction of the heatmap and the centers of 3D boxes. The 3D detection head is the same as DETR3D [19].

During training, 2D supervised losses [54] and query denoies [55] are introduced. All experiments were conducted without the use of CBGS [56] and TTA strategies. For image and BEV data augmentation, we follow the PETR [2], the radar data corresponds to the transformation when performing BEV data augmentation. Following StreamPETR [3] when compared with other state-of-the-art methods, the model is trained for 60 epochs. During ablation experiments, it is trained for 24 epochs. AdamW [57] optimizer is used to optimize the model with the learning rate set to 4e-4. The learning rate is updated by the cosine annealing strategy. The models are trained end-to-end. The number of radar queries is set to 30. The $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are set to 1, the $\lambda_{4}$ is set to 5. We implement RCAlign through the mmdetection3D framework and train it on NVIDIA 3090 GPUs.

In this section, we first present the results of RCAlign and other state-of-the-art methods for 3D object detection on the nuScenes val set and test set. Following that, visualisation results of RCAlign on the nuScenes val set are presented. Finally, we give the results of RCAlign in 3D object tracking.

3D Object Detection on val set. We compare the RCAlign with the state-of-the-art methods on nuScenes val for 3D object detection in Tab. I. (1) With the R50 [50] backbone and the input resolution of 256$\times$704, RCAlign outperforms both existing camera-based methods and radar-camera fusion methods, including the latest state-of-the-art methods CRN [11] and RCBEVDet [8]. The NDS and mAP are significant improvements compared to RCBEVDet, with increases of 4.3% and 8.4%, respectively. (2) To the best of our knowledge, this may be the first radar-camera fusion algorithm to achieve an NDS exceeding 60% in real-time 3D object detection. (3) Furthermore, RCAlign with R50 backbone can even outperform CRN using the R101 backbone with the input resolution of 512$\times$1408. (4) When the model size and image size increase to R101 and 512$\times$1408, RCAlign still can outperform all previous camera-based or radar and camera fusion methods. Alongside the significant improvements in NDS and mAP, mAVE has also made remarkable strides. The improvement of mAVE can be attributed to the ability of radar to capture information about the speed of objects.

Pre-class AP on val set. In order to better demonstrate the performance of RCAlign, we analyse the pre-class AP for the radar and camera fusion models in Tab. II. From the table we can see that all the classes except barriers have been improved. Especially for pedestrians, which always is small or clustered, there has been significant improvement, with increasing by 11.5% compared to CRN. In the case of barriers, comparison with CRN shows almost equal performance. In terms of mAP, we have also seen a significant improvement compared to other radar and camera fusion methods.

3D Object Detection on test set. For the nuScenes test set, we use the V2-99 [51] backbone and the input resolution is 640$\times$1600. Table III demonstrates that RCAlign has a notable advancement over existing state-of-the-art methods. When compared to RCBEVDet, there is an improvement of 3.4% NDS and 5.6% mAP. Additionally, RCBEVDet [8] improves baseline (BEVDepth) results by 3.4% NDS and 3.5% mAP. In contrast, RCAlign enhances baseline (StreamPETR) results by 3.7% NDS and 5.6% mAP, which is surpassing RCBEVDet. The above analyses show that RCAlign can achieve outstanding results, thus validating the effectiveness of the proposed model.

Visualisation Results. We show the visualisation results in the camera view (left) and BEV view (right) in Fig. 4. In the BEV view, RCAlign can present excellent detection results, particularly for the densely gathered crowd highlighted in the orange circle. However, as indicated by the green circle, there are some instances of failure, where some objects are either misdetected or missed. This may be due to the distance from the vehicle resulting in poorer detection results. More visualisation results for different weather conditions are provided in Fig. 5. From the figure, we can see that RCAlign can obtain detection results closer to the ground truth in various weather conditions.

3D Object Tracking on test set. As shown in Tab. IV, when compared to the other state-of-the-art methods with the V2-99 backbone, RCAlign achieves the best performance in both AMOTA and AMOTP. The AMOTA can be improved by 2.7% compared to BEVNext. With the addition of the ConvNext-B backbone, RCAlign still performs best on AMOTA, which is the most dominant evaluation metric in nuScenes test set for 3D object tracking. RCAlign improves by 3.6% in AMOTA compared to CRN. For other metrics, RCAlign achieves the best or just below the best performance.

In order to verify the validity of important components of RCAlign, in this part, we give ablation experiments for DRA, REF and their containing sub-modules.

Effectiveness of DRA. In Tab. V (a), we validate the effectiveness of each sub-module of the DRA. (1) Firstly, we present the results of StreamPETR [3] with deformable attention (baseline) in the first line. Then, a radar branch is added to the baseline (Rbaseline), and the features from both modalities are aggregated using deformable attention, resulting in a significant improvement over baseline by 5.1% NDS and 7.4% mAP. This part proves that radar can indeed assist 3D object detection for better performance. (2) Afterwards, the contrastive loss or the second deformable attention is added to Rbaseline, and it is clear from lines 3 and 4 that adding any of the sub-module of DRA can improve the performance of Rbaseline on both NDS and mAP. The same results are observed for mATE and mAOE. The above results demonstrate that the proposed alignment strategy and the inter-modal features interaction are beneficial for Rbaseline. (3) Finally, both sub-modules are added to the Rbaseline, resulting in further performance improvements. (4) In summary, by adding the DRA, the model outperforms the baseline by 6.6% NDS and 9.5% mAP and surpasses the Rbaseline by 1.5% NDS and 2.1% mAP. The experimental results on nuScenes dataset validate the effectiveness of DRA.

Effectiveness of RH and RFE. We conducted ablation experiments on the radar head (RH) and RFE module, the results are presented in Tab. V (b). (1) We first introduce the RH module based on the DRA module, which resulted in a 0.2% gain in NDS. (2) After that, we use the second radar head (SRH) to classify and regress the enhanced radar BEV features, the NDS and mAP can be further improved, which increased by 0.2% NDS and 0.4% mAP. (3) Furthermore, by exploiting knowledge distillation loss, NDS and mAP are improved by 0.4% and 0.7%, respectively. After adding the RH and RFE, Both NDS and mAP increased by 0.8%. Thus, the effectiveness of RH and RFE is verified. (4) It is worth noting that when compared with the baseline, the proposed RCAlign enhances NDS by 7.4% and mAP by 10.3%. Even for the designed Rbaseline, RCAlign has achieved significant improvements, with NDS and mAP increasing by 2.3% and 2.9%, respectively.

Here, we analyse the important parameters in the model. This includes the Impact of different radar point sampling methods, the impact of different alignment losses, and the impact of contrastive loss and distillation loss weights.

Impact of sampling methods. As depicted in Fig. 2, radar points are selected to be part of the sparse queries after the radar head. Here, we analyze the impact of different sampling methods for radar points on the experimental results. As shown in Tab. VII, (1) we start with a random sampling approach, the NDS and mAP are both terrible. (2) After that, we use the farthest point sampling (FPS), which is employed in PointNet++ [60], we take the radar point corresponding to the highest probability value as the initial point. Compared with the random sampling, there are some improvements in NDS and mAP, but some reduction in mATE and mAOE. (3) Finally, we attempt to select radar points corresponding to the top $k$ probability values (Topk) obtained from the radar heatmap. Compared to the previous two methods, this approach led to improvements in NDS, mAP, mATE, and mAOE. (4) Therefore, we choose Topk as the selection method for radar points.

Impact of alignment Loss. We experiment with three different alignment losses to align sparse queries after the two route paths. As shown in Tab. VII, (1) we initially use the L1 loss to align the two sparse queries in DRA, but it produce poorer results. (2) When the alignment loss is changed to cosine loss, there is a significant improvement in overall metrics. This is due to that although the features sampled by the two routing paths are the same, the different order of sampling means that the final feature representations should only have semantic similarity, rather than forcing the feature values to be identical as well. (3) After that, we modify the alignment loss to a contrastive loss, and we can see that all the metrics are further improved. In contrast to the cosine loss, contrastive loss not only pulls the similarity of features at the same location but also simultaneously pushes apart features at different locations to some extent. (4) Based on the above analysis, we choose contrastive loss for the alignment of the two route paths.

Impact of $\lambda$. We conduct experiments on how the weight of contrastive loss and knowledge distillation loss would affect the performance of the model in the Eq. 6. As shown in Tab. IX, with the $\lambda_{3}$ increases, all metrics first increase and then decrease, and the model achieves optimal performance when the weight of the contrastive loss $\lambda_{3}$ is set to 1. Similarly, Tab. IX shows that the model performs optimally when the weight of the knowledge distillation loss $\lambda_{4}$ is set to 5.

A robust detection algorithm is crucial for enhancing vehicle safety in autonomous driving. Therefore, we conducte robustness experiments under different weather and lighting conditions. Following the CRN [11], we use the R101 backbone and the input resolution of 512$\times$1408, as shown in Tab. X, when compared to the radar camera fusion methods, RCAlign achieves state-of-the-art performance across various weather and lighting scenes. Additionally, due to radar is not affected by weather or lighting conditions, RCAlign even outperforms lidar-based method (CenterPoint) in rainy and night scenes. The above experimental results demonstrate that RCAlign is both effective and robustness in different weather conditions.