CalibDNN: Multimodal Sensor Calibration for Perception Using Deep Neural Networks

Figure 1 shows the system overview of the proposed CalibDNN. The output is the 6-DoF parameters, including a 3-DoF rotation vector and a 3-DoF translation vector. We take the point cloud obtained from the a LiDAR and the correspondent RGB image obtained from a camera as the input. We do not know the extrinsic parameters between the LiDAR and camera in the real world, but we do need pre-calibrated data samples to generate ground truth data for training network model. Therefore, when projecting the point cloud onto the image plane, the so-called depth map, we add a random transformation in the extrinsic parameters to get a miscalibrated depth map used for model training. After which, we feed the miscalibrated depth map and the correspondent RGB image into the calibration network to predict the rotation and translation vectors, which will be used to compute the transformation loss. We also apply the predicted transformation matrix, converted from predicted vectors, onto the input depth maps and ground truth transformation onto the input depth maps to compute the depth map loss. We can also obtain the ground truth and predicted point clouds through back-projection from the correspondent depth maps to calculate the point cloud loss.

As discussed in the last subsection, the inputs are pairs of the point clouds and RGB images. We need to convert the 3-D point cloud to the 2-D depth map so that each pixel in the depth map represents the distance information, from the camera to the correspondent point in the real world. Given intrinsic parameters $P$ and extrinsic parameters $T$ the projection formula is defined as:

Where $x$ represents 3-D points in a point cloud and $y$ represents correspondent 2-D points in the converted depth map.

Our method requires calibrated data pairs of image-point clouds, or image-depth map, for training network. To obtain the uncalibrated depth map, we intendedly miscalibrate the calibrated depth map by applying a random transformation ${{T}_{rand}}$ and use as the input depth map. Therefore, the extrinsic parameters after adding the random transform are ${{T}_{rand}}\cdot T$, and the input depth map, projected from the point cloud, is defined as:

Given the miscalibrated depth map as input, the ground truth depth map is defined as:

Therefore, the target is to regress on the ground truth transformation ${{T}_{gt}}={{T}_{rand}}^{-1}$

Since the converted depth map is often too sparse to extract feature information, we apply data interpolation of max-pooling operation onto the sparse depth map to produce a semi-sparse depth map.

As shown in Figure 2, there are two parts in the calibration network, including feature extraction and feature aggregation. Each part will be introduced in detail in this section.

Feature Extraction: There are two network branches to extract features from the RGB image and depth map separately. Thanks to the recent success of ResNet[18], we use ResNet-18 as the architecture to extract features from the two branches. The two branches are symmetric and initialized by pre-trained weights since, as mentioned in the research works[19, 20], pre-trained weights can also boost feature extraction on the depth map. Additionally, feature relevance can be preserved using the same initialization and architecture, which contributes to feature matching in the aggregation part. To apply the pre-trained weights on the one channel depth map, we initialize the filter weights of the first convolutional layer by the mean weights along three channels.

Feature Aggregation: Obtaining features from two branches, we concatenate the extracted features along the channel dimension and input them into the aggregation network. This part is as important as the former part, as accurate prediction is highly dependent on the feature matching power, so exquisite network design is critical. Inspired by the architecture of ResNet, we concatenate two layers with a similar structure as layers in ResNet. Unlike ResNet, to reduce dimension, we use half the number of channels in the second layer. Then we input the features into a convolutional layer for further feature matching and dimension reduction. Finally, we decouple the output into two identical branches to predict rotation and translation vectors separately to predict extrinsic parameters. In each branch, we use a convolutional layer, with $1\times 1$ filters, to keep structural feature information and a fully connected layer to predict the vectors. Compared with using one branch to predict the dual quaternion, the performance of separate vector prediction is better since the separate convolutional layer and fully connected layer can automatically learn different translation and rotation information. Without pre-trained weights, we initialize the weights by He-Normal[21], which gives a more efficient prediction.

Model generalization: The system can also be generalized into different input image sizes by image downsampling or change network architecture. For example, given a larger input image size, we can add some average pooling layers in the feature aggregation part. Instead of using max-pooling layers, correspondent information along the channel is kept by average pooling. Some different input image size experiments are conducted on the RELLIS-3D dataset[22]. Given a smaller input image size, we can eliminate the gray convolutional layer in Figure 2 to fit the image size.

Parameters conversion: With the predicted rotation vector $r={{({{r}_{x}},{{r}_{y}},{{r}_{z}})}^{T}}$ and translation vector $t={{({{t}_{x}},{{t}_{y}},{{t}_{z}})}^{T}}$, we need to convert them to the transformation matrix for further loss function calculation. The translation vector can be directly used as a translation term in the transformation matrix. However, the rotation vector needs to be converted to rotation matrix $R\in SO(3)$ by the well-known Rodrigues’ rotation formula:

Where $I$ is an identity matrix, $\overset{\scriptscriptstyle\frown}{r}$ is the antisymmetric matrix of rotation vector $r$, and $\theta$ is a rotation angle. Combined with the translation vector $t$, we get the predicted transformation matrix ${{T}_{pred}}\in SE(3)$:

We use the weighted sum of three types of loss functions as the total loss, including transformation loss ${{L}_{t}}$, depth map loss ${{L}_{d}}$ and point cloud loss ${{L}_{p}}$. The total loss function is defined as follow:

Where ${{\lambda}_{t}}$, ${{\lambda}_{d}}$, and ${{\lambda}_{p}}$ denote the respective loss weight.

Transformation loss: The target is to regress on the rotation and translation vectors, which are the output of the calibration network. Therefore, we compute the L-2 norm between the prediction and the ground truth separately on the rotation vector and translation vector.

Where ${{r}_{pred}}$ is the predicted rotation vector, ${{r}_{gt}}$ is the ground truth rotation vector, ${{t}_{pred}}$ is the predicted translation vector, and ${{t}_{gt}}$ is the ground truth translation vector. Since there is a deviation between the scale of rotation L-2 norm and translation L-2 norm, we add a scalar $\alpha$ to control the rotation term.

Depth map loss: Given predicted transformation matrix ${{T}_{pred}}$, we apply the transformation to the input depth map and obtain the predicted depth map. For the input depth map with a significant miscalibration deviation, if we directly apply the predicted transformation to them, there will be a large blank area because those points projected outside the image are missing. We apply the transformation to the conversion between the point cloud data and depth map to recover the missing points. Employing this approach, we can directly input the point cloud into the system instead of converting the point cloud to the depth map first. Below formula shows the relationship between the predicted and ground truth depth maps.:

Where $x$ is the 3D points in point cloud, ${{y}_{pred}}$ is the correspondent 2D points in predicted depth map, ${{y}_{gt}}$ is the correspondent 2D points in ground truth depth map, and ${{T}_{rand}}$ is the random transformation.

The depth map loss between predicted and ground truth depth maps is defined as:

Where $N$ is the number of pixels in the depth map.

Point cloud loss: Given intrinsic camera parameters, we can back-project the depth map to the point cloud data. Thus, we can get predicted point cloud and ground truth point cloud by back-projection from predicted depth map and ground truth depth map. We use the Chamfer Distance[23] (CD) between these two point clouds as the loss function. Compared with the testing accuracy on Earth Mover’s distance loss (EMD), the performance of using Chamfer Distance loss is better since it preserves unordered information. To keep point cloud loss the same order of magnitudes with transformation loss, we also add scalars to compute the mean minimum distance between two point clouds. The loss function is defined as:

Where ${{S}_{pred}}$ is the predicted point cloud, ${{S}_{gt}}$ is the ground truth point cloud, $N$ is the number of points in ${{S}_{pred}}$ and $M$ is the number of points in ${{S}_{gt}}$.

We find that there is a limitation on the optimization process during training. The mean rotation and translation errors can only be reduced by a specific range, which means, when the miscalibration value is large enough, the mean errors cannot be reduced to a value small enough. However, although our model is a single iteration, it still can be extended to multi-iteration to boost performance when miscalibration ranges are significant. We can apply iterative refinement to improve the prediction model. We train different networks by different miscalibration ranges, specifically from large to small ranges. After that, given the input pairs with large miscalibration, we test it by feeding them into the models pre-trained from large miscalibration to small miscalibration, step by step. The process of iterative refinement is shown in Figure 3.

In Fig. 3, $({{r}_{i}},{{t}_{i}})$ are the miscalibration ranges in the form of rotation and translation vectors that meet the conditions of ${{r}_{0}}>{{r}_{1}}>{{r}_{2}}>\cdot\cdot\cdot>{{r}_{i}}$ and ${{t}_{0}}>{{t}_{1}}>{{t}_{2}}>\cdot\cdot\cdot>{{t}_{i}}$. ${{T}_{i}}$ is the predicted transformation of each network. Thus, the final predicted transformation is defined as:

We use the KITTI dataset[24], the most popular dataset used in autonomous driving research. We utilize the raw data from the KITTI dataset, including RGB images obtained from the PointGray Flea2 color camera and point clouds obtained from the Velodyne HDL-64E rotating 3D laser scanner. We use the synchronized and rectified version of sequence ‘2011_09_26’, with image resolution $1242\times 375$. There are different types of scenes in this sequence, including city, residential, and road, which give considerable information for network training. The dataset is already calibrated by a traditional method[2]. As discussed in subsection 3.2, to get the uncalibrated image pairs, we apply a random transformation to the depth map. To be consistent with the CalibNet[13], we generate 24,000 pairs of training images and 6,000 pairs of testing images, and use miscalibration in the range of $(-{{10}^{\circ}},+{{10}^{\circ}})$ on rotation and $(-0.25m,+0.25m)$ on translation. To test the generalization power of our method, we also use the sequence ‘2011_09_30’, with image resolution $1226\times 370$, on which we apply zero paddings to warp the image into the same size with sequence ‘2011_09_26’. Different sequences are captured from different dates and scenes, with individual extrinsic parameters.

Compared with the urban scene in the KITTI dataset, the RELLIS-3D dataset is collected in an off-road environment, which provides strong texture, complex terrain, and unstructured features, with grass, bush, forest, and soil. It is challenging for most current algorithms that are mainly trained for urban scenes. We use RGB images obtained from the Basler acA1920-50gc camera and correspondent point clouds obtained from the Ouster OS1 LiDAR. To keep consistent with the image size of the KITTI dataset, we downsample the original images and depth maps, with the size of $1200\times 1920$, into the size of $1242\times 375$. We generate a training set of 20,000 pairs and a testing set of 5,000 pairs by the same miscalibration range as in the KITTI dataset.

We use the Tensorflow library[25] to build the network. To accelerate network training and prevent overfitting, we use Batch normalization[26] after each convolutional block and Dropout[27] at fully connected layers. We set the dropout parameter as 0.5. We also add an L-2 regularization term when training on the Rellis-3D dataset. We use Adam Optimizer[28] and set the parameters as the suggested values ${{\beta}_{\text{1}}}\text{=0}\text{.9}$, ${{\beta}_{\text{2}}}\text{=0}\text{.999}$ and $\varepsilon\text{=1}e-8$. We use an initial learning rate $\text{1}e-3$ and reduce it every a few epochs. For the loss function, we set the initial ${{\lambda}_{t}}$ equal to 4, ${{\lambda}_{d}}$ equal to 1 and ${{\lambda}_{p}}$ equal to 40 to keep them on a similar scale. Then, we keep ${{\lambda}_{d}}$ unchanged and slowly reduce ${{\lambda}_{t}}$ and ${{\lambda}_{p}}$. We train 25 epochs in total. We train our model on a 2 GPUs sever machine.

Figure 4 shows examples of calibration results in different scenes, which shows that our model can accurately calibrate the point cloud and RGB image in different scenes, from small to large miscalibration ranges.

To fairly compare with CalibNet, we evaluate the model performance by the geodesic distance over rotation transformation and absolute error over translation separately, which are defined as:

Where ${{\varepsilon}_{r}}$ is the geodesic distance of the rotation matrix, ${{R}_{pred}}$ is the predicted rotation matrix, and ${{R}_{gt}}$ is the ground truth rotation matrix; ${{\varepsilon}_{t}}$ is the absolute error of the translation vector, ${{t}_{pred}}$ is the predicted translation vector, and ${{t}_{gt}}$ is the ground truth translation vector.

We report a mean absolute error (MAE) for rotation prediction of (Roll: 0.11, Pitch: 0.35, Yaw: 0.18) and translation prediction of (X: 0.038, Y: 0.018, Z: 0.096) on the testing set. Figure 5 shows the evaluation curve, in which (a) shows the low rotation absolute errors over the large range of miscalibration, and (c) shows the geodesic distances of most instances concentrates near 0. (b) shows the low translation absolute errors against the large range of miscalibration, and (d) shows the absolute translation errors of most instances concentrates near 0.

Comparison with the state-of-the-art methods: Table 1 shows the performance comparison among our methods, RegNet[11], CalibNet[13], and CalibRCNN[14]. In RegNet, authors use iterative refinement, for which they estimate the extrinsic parameters from different networks, trained with miscalibration ranges from large to small, iteratively. Also, the whole miscalibration range of RegNet is larger than ours, which is (-1.5m, 1.5m)/(-20°,20°); therefore, it is hard to make a rigorous comparison between our CalibDNN and RegNet. We try to keep consistency with CalibNet on the experiment conditions, including the same miscalibration ranges and the same number of training samples, which lead to comparable results between our method and the CalibNet. Although it can be extended to multi-iteration, our current method uses only one model with one iteration, but CalibNet uses multiple models with two strategies to fine-tune the translation values after obtaining the rotation values in the first model. One strategy is given ground truth rotation parameters, they predict the translation parameters separately (the 2nd row in Table 1); the other is given rotation estimation, they predict the translation parameters separately by an iterative re-alignment methodology (the 3rd row in Table 1). It is fair to compare our results with the second strategy of CalibNet. We also list the result of CalibRCNN, which is also a single model, and they use the same miscalibration ranges as us.

Therefore, we focus on the comparison among the results in the last three rows in Table 1, in which the best results are indicated by boldface. Our method CalibDNN in the last row outperforms the other two methods in almost all axes, especially the pitch and X-axis. To be more specific, compared with CalibNet, our method is simpler with just a single model and single iteration but achieves better performance. Compared with CalibRCNN, our method is still far beyond its performance. The excellent performance comes from the exquisite design of network architecture and reasonable training strategy.

Testing on urban scene dataset: To evaluate the generalization ability of the proposed model, we use the KITTI dataset of sequence ‘2011_09_30’ and Rellis-3D dataset as the testing sets and evaluate the model performance trained on the KTTI sequence ‘2011_09_26. Figure 6 shows the results of KITTI sequence ‘2011_09_30’ in different scenes, with the same results layout with Figure 5. We observe that the misalignment is rectified in prediction, which implies the accurate prediction of extrinsic parameters. For sequence ‘2011_09_30’, we report the MAE rotation prediction of (Roll: 0.15, Pitch: 0.99, Yaw: 0.20) and translation prediction of (X: 0.055, Y: 0.032, Z: 0.096). The MAE is slightly larger because of the different scenes. For Rellis-3D, the MAE rotation prediction is (Roll: 1.40, Pitch: 3.44, Yaw: 2.33) and translation prediction is (X: 0.101, Y: 0.121, Z: 0.186). The performance is worse since the training scenes are significantly different with the testing scenes, from urban to the field. This scene is challenging for calibration because of the complex and unstructured feature.

Training on terrain scene dataset: Since the scene of KITTI and Rellis-3D is totally different, one is the urban environment, and the other is the off-road environment, we re-train the model on Rellis-3D and show the testing results in different scenes in Figure 7, with the same layout as Figure 5. Although there is a visually detected deviation between prediction and ground truth, the misalignment of the input depth map is still rectified significantly. We report the MAE rotation prediction of (Roll: 1.00, Pitch: 2.57, Yaw: 1.94) and translation prediction of (X: 0.092, Y: 0.074, Z: 0.082). Compared with the MAE of Rellis-3D in last subsection, the performance is much better. Although not as good as on the KITTI dataset, the performance is still acceptable since the original miscalibration is still reduced significantly. Also, we find the original calibration of Rellis-3D is not very accurate, which also leads to a large MAE. We believe our CalibDNN rectifies the inaccurate calibration to some extent.