Improvement on LiDAR-Camera Calibration Using Square Targets

Target-based approaches use specially designed objects to facilitate feature detection and matching. In [3], a checkerboard is used for the calibration of a 2D laser range finder and a camera. Laser points are manually selected and the parameters are estimated using plane-line correspondences. A checkerboard is also utilized in [4], where the segmentation of target point clusters in LiDAR is not discussed. The extraction of checkerboard edges in point cloud is affected by the measurement quantization error, especially at large distances, leading to degraded calibration accuracy [2]. In [2], LiDAR vertices are estimated indirectly by optimizing the transformation between LiDAR and target coordinate frames, where LiDAR targets are detected based on intensities. A good detection performance relies on the target material, target range and the point cloud density, limiting the application scope of the algorithm, as shown in Fig. 1(a). Both [5] and [6] design a rectangular calibration board with four circular holes with different maker patterns. A v-shaped target [7] has also been explored as the calibration target, where extrinsic parameters are estimated via solving the perspective-n-point (PnP) problem. A sphere target has also been leveraged for calibration in [8]. In this paper, square boards are chosen as the calibration target due to their ease of fabrication, enabling easy deployment of the proposed method in real world scenarios.

Image and LiDAR intensity map are used within the mutual information framework for extrinsic calibration [9]. Issues arise when the range sensor does not provide reflectivity information or the observed scene has low reflectivity. In [10], extrinsic parameters are estimated by the optimization of the image intensity variation. This approach assumes constant lighting and suffers from significantly degraded performance in case of illumination variation.

LiDAR-camera calibration can be accomplished via registration of LiDAR depth discontinuities and image edges, either using the mutual information framework [11], the grid search method [12] or the optimization method minimizing line projection error [13]. LiDAR edges can be extracted from a single LiDAR scan [12] or from a point cloud map built using LiDAR odometry algorithms [14]. A primary limitation of edge based methods is their assumption that depth discontinuities corresponds mostly to image edges which may be not valid in practice. Moreover, these algorithms require edges in various directions to avoid degenerate optimization.

Reg-Net[15] is the pioneering work using deep learning for LiDAR-camera calibration. The techniques in [15] are followed by subsequent works, including the automatic generation of training data by adding perturbation to known calibration, the dual branch architecture for camera and LiDAR feature extraction and the training of several networks for iterative calibration refinement. Lcc-Net[16] is another representative work which uses a cost volume layer for LiDAR-camera feature matching. DXQ-Net [17] learns the calibration flow defined as the offset of the current LiDAR projection in image space. A differentiable optimizer is employed for solving extrinsic parameters. Compared to [16], its generalization ability is greatly enhanced with respect to different environments and sensor installation. However, its accuracy and stability declines noticeably when applied to different sensor models. The generalization issue is common to learning based methods. Besides, the model training requires ground truth calibrations which come from other calibration approaches such as the target-based methods.

A rigid transformation between two different coordinate systems can be represented by a matrix $\mathit{T}=\begin{bmatrix}\mathit{R}&\mathit{t}\\ \textbf{0}&1\end{bmatrix}\in\mathit{SE}(3)$. As shown in Fig. 2, $\mathit{L}$, $\mathit{C}$, $\mathit{B}$ stand for the LiDAR, camera and calibration target (square board) coordinate systems respectively. The right superscript and subscript denote the two coordinate systems involved in the transformation. In our target application scenarios, the coarse extrinsic parameters ${\mathit{T}_{C}^{L}}^{*}$, camera intrinsic parameters and the square target length $l$ are assumed to be known. In particular, the coarse extrinsic parameter can be obtained from vehicle mechanical design. Additionally, the sensors are assumed to be time synchronized. The algorithm begins with calibration target detection from LiDAR and camera. A grid search is then performed to identify the extrinsic candidate which allows the best alignment between LiDAR and camera detection results. At last, a direct optimization method is designed to deliver the final estimation of the extrinsic parameters. Fig. 3 shows the pipeline of the proposed method.

AprilTag[18] is widely used in robotic applications and is chosen for camera target detection in this paper. $M=\{m_{j}\},j\in\{0,1,2,3,4\}$ represents the center and the four vertices of the AprilTag pattern in the target coordinate system $B$, and $\mathit{D}=\{d_{j}\},j\in\{1,2,3,4\}$ represents the corners detected in a image. Due to AprilTag’s encoding system, corners detected in the image can establish one-to-one correspondence with the target vertices. The transformation between the camera and a calibration target $\mathit{T}_{B}^{C}$ can then be estimated using the PnP algorithm. Fig. 2 shows an example of the 2D-3D correspondence. For each target, the camera detection algorithm outputs the four image corners and $\mathit{T}_{B}^{C}$.

This section presents our two-stage algorithm for detecting markers on LiDAR point cloud. Prior to finding potential markers, the point cloud is segmented into several clusters. Then a descriptor representing each cluster is extracted. Considering that only a small portion of clusters contain markers, a descriptor-based verification is adopted to identify board-like, rectangular clusters.

In practice, the translation installation accuracy is of millimeter level. However, it is common to see rotation installation error on the order of one and even a few degrees. Rotation calibration error has a large impact on LiDAR-camera data association, especially for distant objects. Thus a coarse search step is introduced to obtain a more precise initial rotation for the subsequent LiDAR extrinsic optimization.

The coarse grid search step produces initial extrinsic parameters ${\mathit{T}_{C}^{L}}^{*}$ as well as the LiDAR-camera target correspondences. As in [2], for each target, the transformation $\mathit{T}_{B}^{L}$ is first optimized using $\mathit{T}_{B}^{C}*{\mathit{T}_{C}^{L}}^{*}$ as the initial value. For each target point cloud cluster $\mathit{P}={\{{}^{L}x_{k},{}^{L}y_{k},{}^{L}z_{k}\}}_{k=1}^{N}$, the cost function $C({\mathit{T}_{B}^{L}})$ is defined as below:

Then the target vertices in the LiDAR coordinate system is easily calculated by the transformation of $M=\{m_{j}\},j=1,2,3,4$ with $\mathit{T}_{B}^{L}$. After the vertices coordinates of calibration targets in all LiDAR scans are estimated, the final extrinsic can be optimized using the $\mathit{PnP}$ and $\mathit{RANSAC}$ algorithms. This approach is termed as the indirect optimization method in this paper.

In [2], the parameter $\alpha$ in z-axis is tuned according to LiDAR measurement noise, however the effect of different parameter settings is not discussed. If the value is overestimated (larger than real noise level), it will result in insufficient constraints for $\mathit{T}_{B}^{L}$ optimization, thereby leading to inaccurate vertex estimation. Fig. 4(a) shows such an example where the estimated vertex does not agree with the LiDAR point cloud. The parameter tuning will be further discussed in the experimental part.

For the indirect optimization method, the accuracy of LiDAR vertex fitting is prone to sparse and sometimes partial LiDAR observation of calibration boards. Fig.5 shows two common scenarios encountered in data acquisition: the calibration targets are either too far or too colse to the sensors. In both cases, the incomplete point cluster can not provide enough constraints to fit precise vertices. Those sub-optimal vertices can adversely affect the precision and robustness of calibration. Inspired by the well-known direct method used in visual odometry [24], a direct optimization method is proposed which still uses the cost in Eq. 2. However, instead of optimizing $\mathit{T}_{B}^{L}$ for each individual LiDAR target, the extrinsic parameters are directly optimized using raw LiDAR board points, avoiding the intermediate vertex fitting step. For direct optimization, these incomplete point clouds combined together are able to give sufficient constraints for extrinsic optimization. Assuming $K$ target correspondences in total, the cost can be written as

Square calibration boards used in the experiment have a fixed length of 0.6 m and the inner AprilTag marker pattern has a length of 0.48 m. These standard foam boards are mounted on movable stands, with no additional material specifications required. Our camera has a resolution of 3840x1920 pixels. To verify the generalization performance of our proposed algorithms, Robosense¹¹1http://robosense.ai/, Innovusion²²2https://www.seyond.cn/ and Hesai³³3https://www.hesaitech.com/ are selected for data collection. The vehicle is required to move back and forth within the range of 6 to 30 meters from the calibration targets. Each trajectory lasts around 20 seconds, including 200 LiDAR-camera frame pairs. For each dataset, 5 trajectories are included. Fig. 7 illustrates the data collection scenes.

As III-C presented, we proposed a novel, general and robust LiDAR marker detection algorithm based on geometric descriptor matching without any requirement for board materials. In order to verify the versatility and robustness of the algorithm, the marker centers from different types of LiDAR data in our dataset are annotated. A detection is considered a True Positive (TP) when its center is close to a manually labeled center, otherwise a False Positive (FP). Meanwhile, a labeled center without any detection attached will be regarded as a False Negative(FN). Fig. 6 illustrates curves of different types of LiDAR, including two MEMS LiDAR (Innovusion Falcon, Robosense M1) and one mechanical LiDAR (Hesai OT128), showing that our proposal can adapt to different LiDAR scanning patterns. In this figure, the descriptor matching score threshold for binary classification is set ranging from 0.7 to 0.99, and the threshold used in this paper is 0.94 and same for different types of LiDAR. The square markers in this figure correspond to a score threshold of 0.94.

The work of [2] did not study the influence of $\alpha$ on the calibration accuracy. Table I shows the average projection error (in pixels) between fitted LiDAR vertices and detected camera corners using the indirect optimization method. It is seen that $\alpha=0$ is the best setting. It is also noted that the projection errors of Robosense are larger that the of Innovusion, caused by its larger measurement error compared to Innovusion. Fig. 7 shows an example of projected LiDAR points in the image with the optimized extrinsic parameters.

The collected data was down-sampled by distance to compare the robustness of the two optimization methods. That is, a data pair is selected as input for subsequent algorithms only when the vehicle has moved a certain distance. Fig. 8 illustrates the standard derivations (STD) of rotational calibrations. The STD is calculated for 5 different trajectories and the average result (in degree) for roll, pitch and yaw is shown. As the sampling distance increases, the indirect method becomes much more unstable than direct method. Table. II further compares the calibration results of the two optimization methods using data collected at only close or far distances from the calibration targets, shown in Fig.5.

In order to show the effectiveness of the coarse grid search, a dataset is randomly selected and rotation perturbations are added to the accurately calibrated parameters as the initial value. A rotation perturbation is generated by randomly sampling roll, pitch and yaw angles respectively in the range of -10 to 10 degrees. The search range for our grid search algorithm is from -9 to 9 degrees with the search step being 1.5 degrees. A total of 200 tests are performed, where 100 tests use the proposed grid search to provide initial calibration parameters while the remaining 100 tests directly perform extrinsic optimization. All of the tests using the grid search successfully converged. In contrast, only 6 tests succeed for the 100 tests without using grid search.

To measure the timing of the algorithm, experiments were performed on a computer equipped with an Intel(R) Core(TM) i7-10750H processor (2.60 GHz, 12 cores) and 32 GB of DDR4 memory. The processing time for 200 image and LiDAR frame pairs is around 18.7 s, 4.93 s, 0.12 s and 1.25 s respectively for LiDAR marker detection, image marker detection, grid search and direct optimization. Note that the indirect optimization algorithm takes 1.63 s. The results demonstrate that the algorithm proposed in this paper is highly efficient and is capable of meeting the real-time requirements of production scenarios.