R-PointHop: A Green, Accurate, and Unsupervised Point Cloud Registration Method

Classical registration methods such as the iterative closest point (ICP) method and its variants (e.g., Point-to-plane ICP [9], Generalized-ICP [10], Go-ICP [31], etc.) have been used in point cloud registration for a long while. For every point in one point cloud, ICP first finds its closest point in the other point cloud. Then, point correspondences are used to estimate the transformation that minimizes the mean squared error between the 3D coordinates of matched points. Since ICP is local by nature, it works well only when the optimal transformation is close to the initial alignment. Go-ICP uses a Branch-n-Bound (BnB) module to search for a globally optimal solution. Various modifications of ICP are summarized and compared in [32]. The Fast Global Registration method [33] conducts global registration of partially overlapping surfaces without an initial alignment. It uses FPFH [34] feature. Meanwhile, Teaser [35, 36] uses truncated least squares to handle scale, rotation and translation. The above-mentioned methods are model-free. They use handcrafted features and solve an optimization problem.

SIFT [6] and SURF [7] are well known 2D keypoint descriptors. Similarly, some local geometric properties (e.g., eigen decomposition, surface normals, signatures, curvatures, histograms, and angles) can be used to describe points in 3D point clouds. SHOT [12] is a 3D descriptor based on the histogram of point normals in a local support region. Spin-images [13] is a local surface representation comprising of oriented points and their images. FPFH [34] combines 3D coordinates and surface normals of $k$ nearest neighbors of a point. USC [37] is a modification of SHOT. The initial idea of R-PointHop was inspired by these unsupervised local descriptors. However, two additional ideas are added to make the local geometrical descriptors more powerful. First, the target descriptor is learned from training samples rather than being defined by a set of pre-determined rules. Second, it has multi-scale representation capability. To meet the first criterion, we introduce principal component analysis (PCA) for feature extraction, which is data driven. To meet the second criterion, features in R-PointHop are learned in a multi-hop manner, where the neighborhood size grows as the number of hops increases. This allows the derivation of multi-scale descriptors centered at a point so that the short-, mid- and long-range neighborhood information can be captured simultaneously.

Several deep learning methods have been proposed for point cloud classification, segmentation and registration tasks in recent years. PointNet [16], PointNet++ [17] and DGCNN [18] are well known networks in this field, and most learning-based registration methods use them as the backbone. Deep Closest Point (DCP) [23] exploits point features learned from DGCNN and uses a transformer to learn contextual information between features of two point clouds to be registered. A differentiable SVD module is designed to predict the rotation in an end-to-end manner. PR-Net [38] extends DCP for registration of partial point clouds through an action-critic closest point module. PointNetLK [25] uses the globally pooled features learned by PointNet and employs the Lucas-Kanade (LK) algorithm [39] to conduct registration in an iterative manner. In contrast with DCP, PointNetLK does not demand explicit point correspondences and instead uses an iterative LK algorithm. Both DCP and PointNetLK use ground truth rotation matrices and translation vectors to train end-to-end networks. Another approach is to optimize an end-to-end network that finds 3D correspondences, where supervision is provided in terms of valid and invalid correspondence pairs. CORSAIR [40] combines global shape embedding with local point-wise features to simultaneously retrieve and register point cloud objects. 3DMatch [20] uses point correspondences available from RGBD reconstruction datasets to train a siamese 3D CNN. PPFNet [21] finds a local point pair feature embedding, which is followed by PointNet to learn point features for correspondence. DeepMapping [41] uses a deep network to register multiple point clouds to a global reference frame. 3DSmoothNet [42] uses Gaussian smoothing to voxelize points in the neighborhood, followed by a Siamese deep network to learn local point descriptors. 3DFeat-Net [43] uses weak supervision to learn correspondences from GPS/INS tagged point clouds. UnsupervisedR&R [44] in an unsupervised method that uses differentiable alignment and rendering. These methods are usually coupled with random sample consensus (RANSAC) [45] for robust geometric registration. Deep learning methods can yield 3D point descriptors as a byproduct. Yet, they are mainly optimized for a single task, such as, classification, segmentation or registration. Furthermore, they tend not to expand the point neighborhood successively.

The successive subspace learning (SSL) paradigm was introduced for point cloud classification (called PointHop) in [14] and for image classification (called PixelHop) in [46], respectively. The idea was originated from the Saab (successive approximation with adjusted bias) transform, which is a variant of principal component analysis (PCA), in [47]. The Saab transform adds a bias term to the PCA transform to address the sign confusion problem when multiple PCA stages are in cascade.

PointHop uses the statistics of 3D points to learn point cloud features in an unsupervised one-pass manner. This procedure is summarized as follows. First, the attributes of a local point are constructed based on the distribution of points in its local neighborhood. All point attributes from the training data are collected and their covariance matrix is analyzed to define the Saab transform at the first PointHop unit. This process is repeated, which leads to multiple PointHop units. The corresponding receptive field grows as the hop number increases. Later, the channel-wise Saab (c/w Saab) transform was introduced in PointHop++ [15]. The c/w Saab transform is more effective than the Saab transform with regard to computational complexity and storage complexity (i.e., model size). Features at different hop units of PointHop (or PointHop++) are pooled to obtain the global feature vector and fed to a classifier for the classification task. Furthermore, UFF [48] extended this framework for point cloud part-segmentation. PointHop and PointHop++ consist of two modules: 1) unsupervised feature extraction and 2) supervised learning for classification. The proposed R-PointHop method leverages the first module for the registration task.

Another closely related work is the salient points analysis (SPA) method [26]. It is an unsupervised point cloud registration method. SPA selects a set of salient points based on the PCA in local neighborhood of points and uses PointHop++ to learn point features and build correspondence among salient points for transformation estimation. However, SPA ignores the long-range neighborhood information in the salient point selection process. An inconsistent choice of salient points may lead to incorrect registration. Also, selected salient points may not provide a clue on which points to match when there is only a partial overlap between the two point clouds. In R-PointHop, we address these shortcomings by using both short- and long-range features to decide proper correspondences.

Rotation-invariant features do not change when the point cloud undergoes any external rotation. These features are desirable for robust 3D correspondence. They are also useful for other tasks (e.g., classification and segmentation) in presence of different viewpoints. Several earlier methods used the local reference frame (LRF) to design 3D descriptors that are invariant under rotation. The LRF idea lies in the adoption of properties (e.g., distances, angles, principal components, etc.) that are preserved under rigid transformations. Comparison of several LRF designs is given in [49]. Modern learning-based methods handle rotation invariance in different ways. A naive approach is to augment the training data by rotating point clouds by an arbitrary amount. Although it helps a model learn from samples with different rotations during training, it does not guarantee rotation invariance explicitly. Other methods bring point clouds to a canonical frame before further processing. There exist separate networks for pre-alignment. The spatial transformer network (STN) [50] can align images to a canonical form. The T-Net module in PointNet is another example that predicts a transformation to align a point cloud before feature learning. IT-Net [51] aligns point clouds to a canonical form using an iterative network. The plane of symmetry in objects is detected in [52]. It gives three axes which represent the 3D object in canonical form. Another approach is to design a convolution operator that is invariant under rotation [53], [54]. The PPF-FoldNet [22] learns rotation-invariant features for point correspondence.

PointHop and PointHop++ both use pre-aligned point clouds to learn features which are not rotation-invariant. SPA [26] fails in registration when the rotation angles are larger, because it is derived from PointHop++ and does not take rotation-invariance into consideration. Similarly, PointHop and PointHop++ do not perform well in the classification task if an object is not pre-aligned. Here, we solve this alignment problem by learning rotation-invariant features in an unsupervised manner. We will show in Sec. IV that R-PointHop outperforms SPA by a large margin. It also makes PointHop and PointHop++ more robust with regard to point cloud classification becuase it can pre-align 3D point clouds to a canonical form.

In the feature extraction process, a $D$-dimensional feature vector is learned for every point in a hierarchical manner. The feature learning function, $g(\cdot)$, takes input points of dimension $D_{0}$ and outputs points with feature dimension $D$. Here, $D_{0}$ represents 3D point coordinates along with optional point properties like the surface normal, color, etc. Stage $h$ in the hierarchical feature learning process (or $h$-th hop) is a function $g_{h}(\cdot)$ that takes the point feature of the previous hop of dimension $D_{h-1}$ and outputs feature of dimension $D_{h}$.

To find feature $f_{i,h}$ of the $i$-th point in the $h$-th hop, the input to $g_{h}(\cdot)$ includes point coordinates $x_{i}$, features of the $i$-th point from the previous hop $f_{i,h-1}$, coordinates of $K$ neighboring points $x_{j}$ in hop $h$, features of neighboring points $f_{j,h-1}$ from previous hop, and a reference frame $F$. Thus, $f_{i,h}$ is given by

There are several choices of $g_{h}(\cdot)$, which can be determined based on whether the goal is to learn a local or global feature. For R-PointHop, we choose $g_{h}(\cdot)$ such that

where $LRF(x_{i},x_{j})$ is the local reference frame centered at $x_{i}$ (see Sec. III-A1 below). This choice of $g_{h}(\cdot)$ encodes only the local patch information and loses the global shape structure. In contrast, the PointHop and SPA learning functions are given by

where $XYZ$ denotes that points are always expressed in the original frame of reference. Although this learning function captures the global shape structure as the spatial locations of the neighborhood patches $x_{j}$ are preserved, it limits the registration performance in presence of a large rotation angle.

Instead, R-PointHop keeps the local position information with LRF. This is desired for registration, since matching points (or patches), which could be spatially far apart, are still close in the feature space now. In contrast, the global position information is vital for the classification task since we are interested in how different local patches connect to other patches that form the overall shape. Thus, the use of the global coordinates in the classification problem is justified.

The trained R-PointHop model is used to extract features from the target and the source point clouds. A feature distance matrix is calculated whose $ij^{th}$ element is the $l_{2}$ distance between the feature of the $i^{th}$ point in the target and the $j^{th}$ point in the source. The minimum value along the $i^{th}$ row gives the point in the source which is closest to the $i^{th}$ point in the target in the feature space. These pairs of points nearest in the feature space are used as an initial set of correspondences. Next, we select a subset of good correspondences. To do so, the correspondences are first ordered in the increasing $l_{2}$ distance between features of matching points. Top $M_{1}$ correspondences are selected using this criterion. We use the ratio test to further select a smaller set of $M_{2}$ correspondences. That is, the distance to the second nearest neighbor is found as the second minima along the row in the distance matrix. The ratio between the distance to the first neighbor and that to the second neighbor is calculated. A smaller ratio indicates a higher confidence of match. Top $M_{2}$ correspondences are selected using the ratio test. These points are used to find the rotation and translation. Instead of choosing $M_{1}$ and $M_{2}$ points explicitly, we can alternatively set two thresholds $t_{1}$ and $t_{2}$, where $t_{1}$ is for the minimum $l_{2}$ distance between matching features and $t_{2}$ for the minimum ratio. These hyper-parameters are selected empirically in our experiments. It is worthwhile to comment that SPA [26] presented an analogous method to select a subset of correspondences. The main difference between R-PointHop and SPA lies in the fact that SPA uses local PCA only to find salient points in the point cloud. It ignores the rich multi-hop spectral information. In contrast, R-PointHop uses multi-hop features to select a high-quality subset of correspondences.

The ordered pairs of corresponding points $({\bf f}_{i},{\bf g}_{i})$ are used to estimate the optimal rotation $R^{*}$ and translation $t^{*}$ that minimizes the error function as given in Eq. (1). A closed-form solution to this optimization problem was given in [55]. It can be solved numerically using the singular value decomposition (SVD) of the data covariance matrix. The procedure is summarized below.

1. 1.

   Find the mean point coordinates from the correspondences by

   $$\bar{{\bf f}}=\frac{1}{N}\sum\limits_{i=0}^{N-1}{\bf f}_{i},\quad\bar{{\bf g}}=\frac{1}{N}\sum\limits_{i=0}^{N-1}{\bf g}_{i}.$$

   (9)

   Then, compute the covariance matrix

   $$Cov(F,G)=\sum\limits_{i=0}^{N-1}({\bf f}_{i}-\bar{{\bf f}})({\bf g}_{i}-\bar{\bf g})^{T}.$$

   (10)

2. 2.

   Conduct SVD on the covariance matrix

   $$Cov(F,G)=USV^{T},$$

   (11)

   where $U$ is the matrix of left singular vectors, $S$ is the diagonal matrix containing singular values and $V$ is the matrix of right singular vectors. In this case, $U,\ S$, and $V$ are $3\times 3$ matrices.

3. 3.

   The optimal rotation matrix $R^{*}$ is given by

   $$R^{*}=VU^{T}.$$

   (12)

   The optimal translation vector $t^{*}$ can be found using $R^{*}$ and the means $\bar{x}$ and $\bar{y}$:

   $$t^{*}=-R^{*}\bar{{\bf f}}+\bar{{\bf g}},$$

   (13)

   $R^{*}$ and $t^{*}$ are then used to align the source with the target.

Finally, the aligned source point cloud $(G^{\prime})$ is given by

where $R^{*T}$ is the transpose of $R^{*}$ which applies the inverse transformation. Unlike SPA which iteratively aligns the source to target, R-PointHop is not iterative and point cloud registration is completed in one run.

We trained and evaluated R-PointHop on indoor point cloud scans from the 3DMatch dataset [20]. This dataset is an ensemble of several RGB-D reconstruction datasets such as 7-Scenes [56] and SUN3D [57]. The dataset comprises of various indoor scenes such as bedroom, kitchen, office, lab, and hotel. There are 62 scenes in total, which are split into 54 training scenes and 8 testing scenes. Each scene is further divided into various partial overlapping point clouds consisting of 200-700K points.

During training and evaluations, 2,048 points are randomly sampled from each point cloud scan. By inspecting several examples visually, randomly selected points roughly span the entire set and, hence, computationally intensive sampling schemes such as FPS can be avoided. 256 neighboring points are used to determine the LRF. We append point coordinates with the surface normal and geometric features (e.g., linearity, planarity, sphericity, eigen entropy, etc. [58]) obtained from eigenvalues of local PCA as point attributes. Since local PCA is already performed in the LRF computation, their eigenvalues are readily available. Furthermore, we use RANSAC [45] to estimate the transformation. Some successful registration results are shown in Fig. 5.

We compare R-PointHop with 3DMatch [20] and PPFNet [21] since they are among early supervised deep learning methods developed for indoor scene registration. Furthermore, several model-free methods such as SHOT [12], Spin Images [13] and FPFH [34] are also included for performance benchmarking. All methods are evaluated based on 2048 sampled points for fair comparison. By following the evaluation method given by [59], we report the average recall and precision on the test set. The results are summarized in Table I. As shown in the table, R-PointHop offers the highest recall and precision. It outperforms model-free methods by a significant margin. Its performance is slightly superior to that of PPFNet.

Next, we trained R-PointHop on the ModelNet40 dataset [27]. It is a synthetic dataset consisting of $40$ categories of CAD models of common objects such as car, chair, table, airplane, and person. It comprises 12,308 point cloud models in total, which are split into 9840 training models and 2468 testing models. Every point cloud model consists of 2,048 points. The point clouds are normalized to fit within a unit sphere. For the task of 3D registration, we follow the same experimental setup as DCP [23] and PR-Net [38] for fairness.

The following set of parameters are chosen as the default of R-PointHop for object registration.

• •

  Number of initial points: 1,024 points (randomly sampled from the original 2,048 points)

• •

  Point Attributes: point coordinates only³³3Although the surface normal and geometric features were included for indoor registration, they are removed in the context of object registration.

• •

  Neighborhood size for finding LRF: 64 nearest neighbors

• •

  Number of points in each hop: $1024,\ 768,\ 512,\ 384$

• •

  Neighborhood size in each hop: $64,\ 32,\ 48,\ 48$

• •

  Energy threshold: 0.001

• •

  Number of top correspondences selected: 256

• •

  Number of correspondences selected after the ratio test: 128

We compare R-PointHop with the following six methods:

• •

  three model-free methods
  ICP [8], Go-ICP [31] and FGR [33].

• •

  two supervised-learning-based methods
  PointNetLK [25] and DCP [23].

• •

  one unsupervised-learning-based method
  SPA [26].

For ICP, Go-ICP and FGR we use the open-source implementation in Open3D library [60].

In Secs. IV-B1-IV-B5, we apply a random rotation to the target point cloud about its three coordinate axes. Each rotation angle is uniformly sampled in $[0^{\circ},45^{\circ}]$. Then, a random uniform translation in $[-0.5,0.5]$ is applied along the three axes to get the source point cloud. For training, only the target point clouds are used. We report the Mean Square Error (MSE), the Root Mean Square Error (RMSE) and the Mean Absolute Error (MAE) between the ground truth and the predicted rotation angles and the predicted translation vector. In Sec. IV-B5, we align real world point clouds from the Stanford 3D scanning repository [28], [29], [30]. In Sec. IV-B6, we show that R-PointHop can be used for global registration as well as an initialization for ICP. In Sec. IV-B7, we explain the use of R-PointHop as a general 3D point descriptor.

The effects of the model parameters, ratio test and use of RANSAC on the object registration performance are discussed in this subsection. We report the mean absolute errors using different input point numbers and neighborhood sizes of LRF in the first section of Table VII. The error values are comparable for different numbers of input points and LRF neighborhood sizes. The performance slightly drops when 512 points are used. Hence, we fix 2048 points and set the neighborhood size of LRF to 128 in following experiments. Next, we consider various degree of partial overlaps and the effect of adding noise of three levels in the second and the third sections of Table VII, respectively. For partial registration, the error increases as the maximum overlapping region decreases. We see consistent improvement with the ratio test and RANSAC. Similarly, the performance improves after inclusion of the ratio test and RANSAC for registration with noise.

To gain further insights into the difference between simple object point clouds and complex indoor point clouds, we remove the surface normal and geometric features and use only point coordinates for indoor registration. This leads to a sharp decrease in performance, to an average recall and precision of 0.39 and 0.19, respectively. Clearly, the use of point coordinates is not sufficient for the registration of complex indoor point clouds. We also reduce the LRF neighborhood size for indoor point clouds and see whether 64 or 128 neighbors could give similar performance as observed in object registration. Again, there is some performance degradation, and the best results are achieved with 256 neighbors. This is attributed to the fact that the more the number of points used to find the LRF, the more stable the local PCA against small perturbations and noise. In other words, the optimal point attributes and the hyper-parameter settings are different for registering object and indoor scene point clouds.

One shortcoming of deep learning methods is that they tend to have a large model size, which make them difficult to deploy on mobile devices. Moreover, recent studies indicate that training deep learning models has a large carbon footprint. Along with the environmental impact, expensive GPU resources are needed to successfully train these networks in reasonable time. The need to search for an environmental friendly green solution to different AI tasks, or green AI [61], is on the rise. Although the use of efficiency (training time, model size etc.) as an evaluation criterion along with the usual performance measures was emphasized in [61], no specific green models were presented.

R-PointHop offers a green solution in terms of a smaller model size and training time as compared with deep-learning-based methods. We trained PointNetLK and DCP methods using the open source codes provided with the default parameters set by authors. We compare the training complexity below.

• •

  DCP took about 27.7 hours to train using eight NVIDIA Quadro M6000 GPUs.

• •

  PointNetLK took approximately 40 minutes to train one epoch using one GPU while the default training setting is 200 epochs. Thus, the total training time was 133.33 hours.

• •

  R-PointHop took only 40 minutes to train all model parameters using an Intel(R) Xeon(R) CPU E5-2620 v3 at 2.40GHz.

The inference time of all methods was comparable. However, since ICP, Go-ICP, SPA, and PointNetLK are iterative methods, their inference time is a function of the iteration number. We observe that the required iteration number varies from model to model.

The model size of R-PointHop is only 200kB compared to 630kB for PointNetLK and 21.3MB of DCP. The use of transformer makes the model size of DCP significantly larger. Although the model free methods are most favorable in terms of model sizes and training time, their registration performance is much worse. Thus, R-PointHop offers a good balance when all factors are considered.

To determine whether the performance gain using supervised deep learning is due to large unlabeled data, data labeling, or both, we split experiments on ModelNet40 into two parts (i.e., tests on seen and unseen object classes) as a case study. Some supervised learning methods performed poorer on unseen classes (see Tables II, III, and IV), which indicates that they learn object categories indirectly, even though their supervision uses ground truth rotation/translation values without class labels. This behavior is not surprising since the two benchmark methods, PointNetLK and Deep Closest Point (DCP). are derived from PointNet and DGCNN, respectively, which were designed for point cloud classification. In contrast, our feature extraction is rooted in PointHop, which is unsupervised and task-agnostic. Our model does not know the downstream task. Hence, it can generalize well to unseen classes. To show this point furthermore, we use the R-PointHop model learned from ModelNet40 and evaluate it on the Stanford bunny model. Its performance gap is smaller than those of supervised learning methods. These experiments indicate that the performance gain of supervised learning methods is somehow limited to similar instances of point clouds that the models have already seen and their generalization capability to unseen classes is weaker.

In general, we see that R-PointHop works extremely well for the object registration case and also matches the performance with PPFNet for indoor registration. However, there exist some recent exemplary networks (e.g., [42]) that have a higher recall on the 3DMatch dataset. The eight octant partitioning operation in the feature construction step fails to encode better local structure information for point clouds from this dataset. Since R-PointHop is based on successive aggregation of local neighborhood information, an initial set of attributes that captures better local neighborhood structure can help improve the performance. One such choice could be the FPFH descriptor.

For ModelNet40, we see that the performance of R-PointHop degrades when the amount of overlap reduces or the amount of noise increases. One reason is the stability of LRF. It is observed that a larger neighborhood number tends to compensate for noise and surface variations in our experiments on the 3DMatch dataset. However, when the number of points in a dataset is small, we cannot opt for more points in finding LRF. Although RANSAC offers a more robust solution, a fine alignment step may be necessary. That is, some ICP iterations may achieve a tighter alignment.