KDD-LOAM: Jointly Learned Keypoint Detector and Descriptors Assisted LiDAR Odometry and Mapping

We treat the joint learning of 3D keypoint detection and description as a multi-task learning paradigm and follow its fundamental neural network style, i.e., plugging several separated task-specific prediction heads into a shared feature extraction backbone. Inspired by KPConv [10], we propose a fully convolutional network for 3D keypoint detection and description. KPConv directly operates on irregular point sets by interpolating point features to uniformly distributed kernel points for regular convolution, i.e., linear mapping, and summation of kernel responses.

Utilizing the normalized KPConv, TCKDD can construct a fully convolutional network that directly consumes point sets, as depicted in Fig. 2. The backbone can extract multi-scale 3D features at different encoder layers consisting of a stack of residual bottleneck blocks. The locality of KPConv enables strided convolution for downsampling. The decoder can recover the resolution via nearest upsampling and aggregate the multi-scale 3D features via skip connections and $1\times 1$ convolution (unary blocks). Different from the original KPFCNN [10], we replace the batch normalization with group normalization [25], which is robust w.r.t. batch size and group-wise features. Finally, TCKDD densely predicts both point-wise descriptors and saliency uncertainty based on the shared 3D features from the backbone via two independent all-MLP heads, i.e., $1\times 1$ convolutional blocks.

Metric learning is widely used to train descriptors. Essentially it maps low-level geometric representations to a high-dimensional feature space, where descriptors of correctly associated point pairs are close, while those of other pairs differ by at least a margin. The correspondence set $\mathbf{C}$ of two partially overlapped point clouds $P,Q$ is a set of the mutually nearest points $(p_{i},q_{j})$ satisfying $\|p_{i}-q_{j}\|_{2}\leq R_{p}$. The negative point set $\mathbf{N}_{i}$ of a point $p_{i}\in P$ is a set of points $q_{k}\in Q$ satisfying $\|p_{i}-q_{k}\|_{2}\geq R_{n}(R_{n}\geq R_{p})$. Denote $d_{i},d_{j}$ as the descriptors of $p_{i}\in P,q_{j}\in Q$, then the self-supervised descriptor loss can be designed as a hardest quadruplet contrastive loss according to metric learning strategies:

	$\displaystyle\mathcal{L}$	$\displaystyle{}_{desc}=\frac{1}{|\mathbf{C}|}\sum_{(i,j)\in\mathbf{C}}\bigg{\{}\lambda_{p}\left[D(d_{i},d_{j})-m_{p}\right]_{+}$		(1)
		$\displaystyle+\left[m_{n}-\min_{k\in\mathbf{N}_{i}}D(d_{i},d_{k})\right]_{+}+\left[m_{n}-\min_{k\in\mathbf{N}_{j}}D(d_{k},d_{j})\right]_{+}\bigg{\}},$

where $D(\cdot,\cdot)$ is a distance metric and $m_{p},m_{n}$ are positive and negative margins, respectively. This loss can maximize the similarity between descriptors of true correspondences and minimize the maximum similarity otherwise.

The principle of joint learning of 3D keypoint detection and description is to fully adapt the detector and the descriptors to each other. Therefore, we treat the detection head of TCKDD as a saliency scoring network based on the matchability of descriptors. According to (1), we can directly design a metric named matchability index to characterize the matchability of a given descriptor $d_{i}$ quantitatively:

$$m_{i}=\left[D(d_{i},d_{j})-m_{p}\right]_{+}+\left[m_{n}-\min_{k\in\mathbf{N}_{i}}D(d_{i},d_{k})\right]_{+},$$

(2)

where $d_{j}$ is the descriptor of the correctly associated point of $p_{i}$ in point cloud $Q$. This matchability index is actually the hardest triplet loss of a single descriptor in metric learning, which describes the distinctness of a descriptor in the feature space. Particularly, we use the hardest negative pairs for negative mining so that the matchability index directly models the decision margin of descriptor matching and optimizes the decision boundary during metric learning. A lower matchability index $m_{i}$ indicates greater matchability of the descriptor $d_{i}$, i.e., the 3D point $p_{i}$ is more salient. Therefore, all we need is to learn a matchability index estimator as a keypoint detector. We propose to learn a probabilistic model rather than a regressive model for matchability index estimation since the descriptor is predicted from a specific point cloud sampled from the surfaces of a dense 3D scene in a specific perspective, and so as the descriptors of the associated points. Ideally, metric learning would construct a fully discriminative feature space where the matchability index of each descriptor is zero. Hence, we choose an exponential distribution to model the matchability index $m_{i}$ with a parameter $\sigma_{i}$, or namely saliency uncertainty:

$$p(m_{i}|\sigma_{i})=\frac{1}{\sigma_{i}}\exp{\left(-\frac{m_{i}}{\sigma_{i}}\right)}.$$

(3)

It is notable that the detection head of our TCKDD directly predicts point-wise saliency uncertainty as outputs. Hence, we can design a probabilistic detection loss based on maximum likelihood estimation (MLE) to robustly fit this exponential distribution model:

	$\displaystyle\mathcal{L}_{det}$	$\displaystyle=-\frac{1}{|\mathbf{C}|}\sum_{(i,j)\in\mathbf{C}}\left(\ln p(m_{i}|\sigma_{i})+\ln p(m_{j}|\sigma_{j})\right)$		(4)
		$\displaystyle=\frac{1}{|\mathbf{C}|}\sum_{(i,j)\in\mathbf{C}}\left(\ln\sigma_{i}+\frac{m_{i}}{\sigma_{i}}+\ln\sigma_{j}+\frac{m_{j}}{\sigma_{j}}\right).$

Theoretically, the first derivative of the log-likelihood indicates that the global optimality conditions for detector learning are $\sigma_{i}=m_{i}$, making the probabilistic detection loss effective for training the detection head as a robust matchability estimator. Remarkably, this loss is also a weighted form of the hardest contrastive loss, with the descriptor losses of keypoints having higher weights than those of non-salient points. When the detector meets global optimality, this detection loss turns out to be a logarithmic contrastive loss, prioritizing the descriptors of keypoints. This approach enhances the matchability of keypoints, mitigating the negative effects of mining geometric features in non-salient areas like planar surfaces and disorganized regions. Hence, this probabilistic detection loss can not only train a keypoint detector as a robust matchability estimator that fully accommodates the jointly learned descriptors but also promote the learning of keypoint descriptors via weighted metric learning.

As illustrated in Fig. 3, we integrate TCKDD into a LiDAR odometry and mapping system, i.e., KDD-LOAM, consisting of the following key components.

Scan deskewing and scan-to-scan registration. Similar to [26], we first utilize the most generally applicable constant velocity model for scan deskewing, which requires no extra sensors involving time synchronization. As a powerful front-end of point cloud registration, TCKDD can densely predict global and local context-aware descriptors and detect keypoints by sorting the saliency uncertainty. We leverage RANSAC to establish sparse correspondences based on the descriptors and minimize the sum of point-to-point distances between inlier correspondences based on SVD. This pipeline achieves real-time scan-to-scan registration to provide a solid initial guess for incremental ego-motion estimation.

Keypoint subsampling and mapping. With a reliable relative pose guess from scan-to-scan registration, we refine the ego-pose estimate by aligning the deskewed and subsampled scan with the accumulated local map. Unlike methods such as [12] that create sparse feature maps of edge points or surface points with noisy directions or normals, we propose a voxel grid map capturing detailed geometry akin to surface reconstruction. We keep it simple, approximating complex 3D surfaces in any topology with ample scan points. Utilizing a voxel hash map with voxel size $v\times v\times v$ that stores up to $N_{max}$ points per voxel, we achieve efficient insertion, indexing, deletion, and nearest neighbor search compared to 3D arrays [12] or KD-trees [24]. Instead of high-resolution panoptic mapping [26], KDD-LOAM accumulates a voxel hash map for only salient regions with low saliency uncertainty. For better surface representation, we fit voxels with $N_{max}$ points to planes via least square regression. Voxels meeting error criteria of regression are replaced with surfels represented by the point closest to their center, their normal vector, and radius $v$. Our approach, in contrast to [26], adapts voxel grids for adaptive 3D salient region reconstruction, combining dense points and sparse surfels for a memory-efficient representation applicable to global mapping.

Inspired by CT-ICP [27], we adopt a two-stage voxel grid-based subsampling for sequential map update and ego-pose estimation. In the first stage, we use voxel size $\alpha v$ ($\alpha\in(0,1]$) to downsample by reserving an original scan point per voxel to prevent discretization errors. After scan-to-map registration, these subsampled points are transformed using the global pose estimate and added to the voxel hash map. In the second stage, we propose a saliency-aware voxel grid subsampling using voxel size $\beta v$ ($\beta\in[1,2]$) to select keypoints for faster scan-to-map registration. Non-salient points are discarded, while salient regions accommodate more points per voxel for accurate point cloud registration.

Robust scan-to-map registration. We use scan-to-map registration for more accurate odometry as it proves more reliable and robust than scan-to-scan registration [12, 1]. Our scan-to-map registration builds on the classic ICP algorithm, which typically establishes correspondences between two point clouds via nearest neighbor search. With a reliable scan-to-scan relative pose guess grounded in geometrically consistent correspondences, it is more likely to avoid sub-optimal convergence and reduce ICP iterations. However, ICP requires a hand-crafted maximum distance threshold for outlier rejection, which depends on the expected initial error.

To this end, we treat scan-to-map registration as compensation for scan-to-scan pose estimation, determining the maximum distance threshold by assessing deviations from the scan-to-scan pose estimate over time. Denote this deviation as $\Delta T\in SE(3)$, the upper bound of the point deviation is

$$\delta(\Delta T)=\|\Delta t\|_{2}+2r\sin\left(\frac{1}{2}\arccos{\frac{tr(\Delta R)-1}{2}}\right),$$

(5)

where $r$ is the maximum range of LiDAR scans, $\Delta R\in SO(3)$ and $\Delta t\in\mathbb{R}^{3}$ refer to the rotation and translation components of $\Delta T$, respectively. Inspired by KISS-ICP [26], we adopt a Gaussian distribution over $\delta(\Delta T)$ and compute its standard deviation $\sigma_{t}$ to robustly set the maximum distance threshold of ICP as the three-sigma bound $\tau_{t}=3\sigma_{t}$.

Given a point from the current scan during data association, we first search the nearest point and the nearest surfel separately in the voxel hash map based on point-to-point distances. Next, we evaluate the point-to-point distance from the nearest point and the point-to-plane distance from the nearest surfel to determine its correspondence. Finally, the maximum distance threshold $\tau_{t}$ determines whether to accept this correspondence as an inlier. This process establishes a set of point-to-point and point-to-plane correspondences for each ICP iteration. A robust ego-pose (i.e., rotation $R\in SO(3)$ and translation $t\in\mathbb{R}^{3}$) is estimated by minimizing the sum of point-to-point residuals and point-to-plane residuals:

$$\min_{R,t}\sum_{p,q\in\mathcal{C}}\rho(e)=\frac{e(Rp+t,q)^{2}/2}{\sigma_{t}/3+e(Rp+t,q)^{2}},$$

(6)

where $\rho$ is the Geman-McClure kernel with a strong outlier rejection property. The optimal pose can be estimated via the Gauss-Newton method. The Jacobian can be derived by applying the left perturbation model with $\delta\xi=[\delta\rho^{T}\;\delta\phi^{T}]^{T}\in\mathbb{R}^{6},\;\delta\xi^{\wedge}\in\mathfrak{se}(3)$. For a point-to-point residual $e=\|Rp+t-q\|_{2}^{2}$, the Jacobian of $\mathbf{e}=Rp+t-q$ w.r.t. $\delta\xi$ is

	$\displaystyle J_{p}$	$\displaystyle=\frac{\partial(Rp+t-q)}{\partial\delta\xi}=\lim_{\delta\xi\to 0}\frac{exp(\delta\phi^{\wedge})Rp+\delta\rho-Rp}{\delta\xi}$		(7)
		$\displaystyle=\begin{bmatrix}I_{3\times 3}&-(Rp+t)^{\wedge}\end{bmatrix}_{3\times 6}.$

For a point-to-plane residual $e=|n^{T}(Rp+t-q)|^{2}$, the Jacobian of $\mathbf{e}=nn^{T}(Rp+t-q)$ w.r.t. $\delta\xi$ is derived as

$$J_{s}=\frac{\partial nn^{T}(Rp+t-q)}{\partial\delta\xi}=\begin{bmatrix}nn^{T}&-nn^{T}(Rp+t)^{\wedge}\end{bmatrix}.$$

(8)

According to the chain rule of derivative, making the derivative of the objective function w.r.t. the disturbance $\delta\xi$ equal to 0 will lead to the following equation:

$$\sum_{i}\frac{1}{(\sigma_{t}/3+e_{i}^{2})^{2}}J_{i}^{T}J_{i}\delta\xi=-\sum_{i}\frac{1}{(\sigma_{t}/3+e_{i}^{2})^{2}}J^{T}_{i}\mathbf{e}_{i}.$$

(9)

The registration is performed by repeating data association and solving (9) until convergence.

3DMatch is a widely used 3D reconstruction benchmark including 62 indoor scenes collected by RGB-D cameras. We use the training data preprocessed by [18] and evaluate our TCKDD against both hand-crafted and learning-based descriptors on the official test set including scan pairs with $>30\%$ overlap using two metrics: feature matching recall (FMR) [14] and registration recall (RR) [2].

TCKDD is evaluated with different numbers of keypoints in Table I, compared with state-of-the-art learning-based descriptors PerfectMatch [13], FCGF [16], D3Feat [9], SpinNet [17], Predator [18], YOHO [28] and coarse-to-fine registration methods CoFiNet [19], GeoTransformer [20]. For TCKDD, we compare random sampling (rand) with three saliency-based keypoint selection strategies: probabilistic sampling (prob), non-maximum suppression (NMS), and probabilistic NMS (NMS-prob). In terms of FMR, TCKDD without keypoints consistently outperforms all 3D descriptors and performs on par with GeoTransformer. When sampled points are fewer than 1000, TCKDD with keypoints achieves higher FMR, showing more stable performance against existing descriptors. In terms of RR, TCKDD with probabilistic keypoints outperforms all the descriptors and CoFiNet consistently by 2$\sim$39%. With over 250 sampled points, our probabilistic keypoints even surpass GeoTransformer notably. Furthermore, the effectiveness and robustness of our keypoints are validated through consistent RR improvement, especially with fewer than 1000 sampled points.

We demonstrate the robustness of TCKDD (prob) by varying the inlier distance threshold $\tau_{1}$ and the inlier ratio threshold $\tau_{2}$ in FMR. As shown in Fig. 4, we report the performance of hand-crafted descriptors SpinImages [29], SHOT [8], FPFH [7] and ealier learning-based descriptors CGF [30], 3DMatch [2], PPFNet [14], PPF-FoldNet [31]. TCKDD consistently outperforms other methods with $\tau_{1}\geq 5$cm and significantly surpasses them across all inlier ratio thresholds. Under a stricter condition $\tau_{2}=0.2$, TCKDD maintains a high FMR of 89.3%, while SpinNet, D3Feat, and FCGF drop to 85.7%, 75.8%, and 67.4%, respectively, which highlights that TCKDD is more robust to maintain higher inlier ratio in challenging scenarios.

For the KITTI [11] dataset, we use sequences 0 to 5 for training, 6 to 7 for validation, and 8 to 10 for testing. We refine the GPS localization results via ICP [3] as ground truth. Additionally, only point cloud pairs at least 10m away from each other are selected. Following [18], we use three metrics for evaluation: relative translation error (RTE), relative rotation error (RRE) and registration recall (RR).

In Table II, TCKDD is compared with the state-of-the-art descriptors 3DFeat-Net [22], FCGF [16], D3Feat [9], SpinNet [17], Predator [18] and coarse-to-fine registration methods CoFiNet [19], GeoTransformer [20]. TCKDD achieves state-of-the-art RTE and the highest registration recall of 100%, which demonstrates the effectiveness and robustness. Within TCKDD, we compare random sampling (rand) with two saliency-based keypoint selection strategies: sorting (sort) and probabilistic sampling (prob). As shown in Table III, TCKDD maintains 100% of RR even with only 1000 randomly sampled points, indicating the robustness of its descriptors in establishing sparse yet reliable correspondences. The sorted keypoints notably reduce RTE compared to random sampling, especially with fewer sampled points for registration, thus underscoring TCKDD’s ability to detect keypoints with high matchability and repeatability.

In this subsection, we design experiments to demonstrate that 1) TCKDD effectively improves odometry accuracy against its baseline A-LOAM; 2) KDD-LOAM significantly reduces cumulative error in scan-to-scan registration and outperforms classic LiDAR odometry or SLAM systems; 3) KDD-LOAM achieves more memory-efficient mapping while maintaining performance comparable to KISS-ICP. All the algorithms are evaluated with the mean relative pose error (RPE) over trajectories of 100 to 800m (relative translation error in % / relative rotational error in °/100m) [11]. Sequence 08 is excluded from evaluation due to significant errors in its ground-truth localization results.

We first demonstrate the effectiveness of TCKDD by replacing the scan-to-scan registration step of A-LOAM with TCKDD-based RANSAC. As shown in Table IV, TCKDD-based scan-to-scan registration significantly outperforms A-LOAM based on hand-crafted keypoints, providing a much more reliable and accurate pose guess for the subsequent scan-to-map registration. Furthermore, we integrate TCKDD-based scan-to-scan registration with the subsequent mapping step of A-LOAM. Apart from sequence 01 collected from a featureless environment, TCKDD-aided A-LOAM outperforms A-LOAM by a large margin, especially in sequences 00, 02, and 10 with 0.14%, 3.67% and 0.14% improvements of average RTEs, respectively. These results indicate that TCKDD-based odometry outperforms hand-crafted features and effectively complements existing mapping approaches.

Finally, we evaluate KDD-LOAM against state-of-the-art LiDAR odometry methods on the KITTI dataset [11], including LOAM [12], F-LOAM [24], SuMa [1], SuMa++ [32], and KISS-ICP [26]. As shown in Table V, KDD-LOAM consistently outperforms classic odometry systems, LOAM, F-LOAM, and SLAM systems SuMa, SuMa++ across nearly all the scenes. This underscores the effectiveness and robustness of KDD-LOAM. Compared with KISS-ICP using a similar voxel hash map and an ICP-based scan-to-map registration step with adaptive thresholds, KDD-LOAM achieves lower RREs in all sequences while performing on par with KISS-ICP in RTEs, showcasing its potential to achieve lower global localization drifts. The average RPEs are reported as Avg, while Avg^† stands for the results without the featureless sequence 01. Except for sequence 01, KDD-LOAM even achieves a lower average RPE than KISS-ICP with a more memory-efficient map representation. We compare the average memory usage for local maps of some long sequences in Table VI, which indicates that KDD-LOAM consumes 16.6% less memory for local mapping than KISS-ICP.