A Review and A Robust Framework of Data-Efficient 3D Scene Parsing with Traditional/Learned 3D Descriptors

Extracting a discriminative local descriptor is very significant for downstream tasks of 3D scene understanding. In the past few years, various 3D descriptors have been proposed. The 3D local feature description is essentially extracting a feature vector around the query point to describe 3D local geometry. The 3D descriptor can be further divided into the histogram-based and signature-based approaches. The histogram-based approaches encode the local geometric variations and put them into the histogram. The typical histogram-based approaches include PFH and FPFH, and the typical signature-based approaches include SHOT. The differences between FPFH and PFH lie in following aspects. Firstly, FPFH merely has partial connected neighbours, while PFH has fully connected neighbours. And the range of neighbourhood is also different. Secondly, in PFH, each edge is counted only once, while in FPFH, a portion of edges are counted twice. Finally, for $N$ points each having $k$ points in the neighbourhood, the computational complexity of PFH is $O(Nk^{2})$. And the FPFH has much less computational complexity, which is $O(Nk)$. According to our experiments in Section 4, the PFH demonstrates better 3D scene understanding performance compared with FPFH, although PFH is relatively computational intensive. The concept of local reference frame has been proposed by SHOT to build a canonical pose of the local neighborhood. By this kind of design, it can achieve the rotational robustness and 6D-pose independence. Based on or similar to the above mentioned descriptors, many signature-based approaches such as the Heat Kernel Signature (HKS), and the Wave Kernel Signature (WKS), the Scale-Invariant Heat Kernel Signatures (SIKS) have been proposed, which are based on HKS. We have also analysed and compared the traditional visual similarity-based descriptor and the voxel cloud connectivity-based descriptor. The voxel connectivity-based approach can encourage generating segmented regions without crossing object boundaries by means of seeding methods based in 3D space and the flow-constrained local iterative clustering using color and geometrical features. The details of them and their advantages and drawbacks are illustrated in Subsection 3.1. Extensive evaluations and comparisons of these descriptors for various tasks of 3D scene understanding are shown in experimental results in Section 4. The detailed performance of these descriptors for the tasks of oversegmentation, semantic segmentation, and instance segmentation are all evaluated and compared in a detailed way. Moreover, we have proposed the adapted traditional and simple contrastive-learning based learnt 3D descriptors can be integrated with our method to achieve weakly supervised 3D scene understanding with SOTAs performance.

Recently, various learning-based approaches have been proposed to tackle scene understanding in both 2D vision and 3D vision yuzhi2020legacy ; liu2019deep . Deep network-based approaches are widely adopted for point clouds understanding liu2020fg . The fully supervised approaches can be roughly categorized into voxel-based liu2022weakly ; liuws3d , projection-based, and point-based methods liu2022fg ; liu2021fg ; liu2023fac . Many recent works proposed to pre-train networks on source datasets with auxiliary tasks such as the low-level point cloud geometric registration xie2020pointcontrast , the local structure prediction, the completion task of the occluded point clouds wang2021unsupervised , and the high-level supervised point cloud semantic segmentation eckart2021self , with effective learning strategies such as contrastive learning and generative models. Then, they fine-tuned the weights of the trained networks for the target 3D understanding tasks to boost performance on the target dataset. However, all the above methods require accessibility to high-quality fully annotated training data, which are hard to obtain for large-scale 3D scenes. It should be noted that 2D image and point cloud can be reciprocal in both scene understanding and generation. Recent approaches propose using distilled information from 2D image segmentation to assist 3D scene understanding. Also, it has recently be studied that the image can bridge the big semantic gap between the modalities of text and 3D shapes. Also, the LiDAR-based approaches are of significance to many industrial applications such as UAV/robotics inspections liu2022robustmm ; liu2022industrialTIE ; liu2022semi and robotic enhanced large-scale localization in the diverse complex environments liu2022light ; liu2022weaklabel3d ; liu2022robustcyb ; liu2022robust ; liu2022integratedtrack ; liu2023dlc ; liu2022enhanced ; liu2022enhancedarxiv ; liu2022lightarxiv , and large-scale robotic scene parsing liu2021fg ; liu2022fg ; liu2020fg , as well as robotic control as well as manipulation applications liu2017avoiding ; liu2023lidar ; liu2022integrateduav ; liu2022integratednoise ; liu2022datasetsicca ; liu2023learning , etc. Differently, we make the first attempt in traditional and learnt 3D descriptor guided weakly supervised point cloud segmentation.

The weakly supervised approaches for point cloud understanding are effective manners to reduce high annotation burdens liu2022semi . Many preliminary attempts have been tried including labeling a small portion of points xu2020weakly ; liu2021one ; li2022hybridcr ; hou2021exploring or semantic classes wei2020multi . Current approaches for weakly supervised 3D scene understanding can be divided into three main categories: consistency learning xu2020weakly ; shi2021label , pseudo label-based self-training liu2021one ; cheng2021sspc , and contrastive pre-training hou2021exploring ; xie2020pointcontrast . However, current weakly supervised point cloud understanding approaches are far from mature and have their own limitations. The graph-based 3D WSL was proposed to transform point clouds to images for obtaining semantic map, but image-level labels are required for training. Sub-cloud annotations wei2020multi require the extra labour to separate sub-clouds and to label points within the sub-clouds. Directly extending current art methods with weak labels for training will result in a great decline in performance liu2022fg if label percentage drops to a certain value, which is less than 1‰. Self-training techniques have been utilized liu2021one to design a two-stage training scheme to produce pseudo labels from weak labels with the 3D scenes, but it is only tested for the semantic segmentation task with limited performance. Xu et al. xu2020weakly adopts semi-supervised training strategies combining training with coarse-grained scene class level information and with partial points using on tenth labels, but their test datasets are limited and it is tough to uniformly choose points to label. The network is elaborately made to approximate the gradient during the learning process, where the auxiliary 3D spatial constraints and color-level evenness were also considered in the network optimizations. However, the approach was restricted to the object part segmentation, and it is difficult to annotate points in a well-proportioned and homogeneous way as required. The unsupervised pre-training hou2021exploring shows great capacity in unleashing the potential of weak labels to serve for complicated tasks, such as instance segmentation. But merely utilizing pre-training can not make full utilization of the weak labels, which results in dis-satisfactory performance. The concurrent work also explores the weakly-supervised video anomaly detection by magnitude contrastive learning MGFN and the weakly supervised semantic segmentation with image-level supervision qi2016augmented . However, our studied modality which is 3D point cloud is different in the modality and has essentially different properties with images or videos.

Recent studies have produced many elaborately designed networks for 3D semantic/instance segmentation jiang2020pointgroup and object detection qi2019deep . However, they all rely on full supervision. Recently, TWIST chu2022twist also employs self-training-based approach to conduct effective semi-supervised learning. They innovatively proposed novel proposal re-correction module to filter out the low-quality proposals and enhance the pseudo label quality. In addition, many frameworks focus only on a single task, or two similar tasks pham2019jsis3d ; wen2020cf , and the relationships mining between those interconnected or complementary tasks, such as correlations between 3D instance segmentation and object detection, and the relationship between the 3D low-level geometry and high-level semantics, are rarely explored.

In this Subsection, we first illustrate our adapted PFH-based descriptor. Then we illustrate other traditional descriptors, and discussed their advantages and disadvantages. Next, we have designed a simple contrastive learning-based descriptor that can achieve SOTAs performance in oversegmentation and downstream tasks. For the other learning-based descriptor, we have also detailed the advantages and disadvantages of them and done comprehensive comparisons in our experiments.

As our weakly supervised learning framework relies on the self-training, we detail the process of self-training for the tasks of semantic segmentation and instance segmentation respectively in this Subsection. For the task of both semantic segmentation and instance segmentation, we adopt the backbone of SparseConv graham20183d .

Our proposed region merging strategies for segmentation is composed of two parts. The first part is the region-level similarity prediction strategy, and the second part is the learning-based region merging. The two parts are detailed as follows:
The Region-Level Similarity Prediction Strategy We have proposed a learning-based region merging method based on the affinity provided by both the traditional and learning-based 3D descriptors. We first use the PFH-based oversegmentation to obtain regions. The oversegmentation procedure is provided in the Appendix. After the oversegmentation, we can obtain the initial geometrically separated regions. Some randomly selected oversegmentation results are shown in the second column of Figure 7. It is apparent that our adapted PFH-based oversegmentation results can automatically divide the whole point clouds scene into geometrically well-separated regions, which are indicated by different colors. After the point clouds oversegmentation, we also obtain the original pseudo labels by expanding each labeled point to all the points in the region that includes the labeled point. For the regions containing more than one label point, we directly use the semantic of the maximum number of points as the inital pseudo label for the region. If two classes have the same number of labeled points, we have the priority for assigning the class that has larger number of points in the training set. Most importantly, in this Subsection, we propose an end-to-end approach to incorporate the traditional or learnt 3D local descriptors mentioned above for learning-based region merging.

To be more specific, we compute the normal vector of every individual region based on the widely adopted PCA analysis of local neighbouring points. Denote the average normal vector of a certain region as $\textbf{n}=(n_{x},n_{y},n_{z})$, the region-level 3D descriptor-based feature vector as $\textbf{f}_{des}\in\mathbb{R}^{F\times 1}$. $F$ is the dimension of the feature vector and it depends on the category of 3D descriptors mentioned above. The cosine angle between the normals of two adjacent regions $R_{i}$ and $R_{j}$ is denoted as $A_{n}$, and cosine angle between two 3D descriptors-based feature vectors of the two adjacent points/regions is denoted as $A_{des}$, the Affinity $A(R_{i},R_{j})$ is given as:

where the parameters $\lambda_{n}$ and $\lambda_{des}\in(0,1]$, and we set $\lambda_{n}=\lambda_{des}=1$ in all our experiments. In the process of region merging, regions with the similarity larger than $cos\theta_{ts}$ ($cos\theta_{ts}\in(0,1)$) are merged iteratively. The threshold $\theta_{th}$ is very important, which determines whether or not a supervoxel should be merged in the next cluster-level region merging. In all our experiments, we have set the $\theta_{th}$ to $60^{\circ}$. The detailed process of region merging is given in Algorithm 1, and is summarized as follows:

Firstly, regions are ranked according to curvatures. The regions with labeled point and regions that have top 2‰ minimum curvature among all points are regarded as seed regions. The fast Octree-based K-nearest neighbor (KNN) search of seed regions is adopted in each iteration for acceleration. Improved based on it, we propose the following three criteria for a faster KNN region query. 1. If an octant is not overlapped with the query ball, we skip it. 2. If the query ball is inside an octant, we stop searching. 3. If the query ball contains the octant, we just compare the query with all regions, so going into children of that octant is not required. We greatly improve the query speed by 18.2 times for a scene of about $3\times 10^{6}$ point for example, and it also substantially speeds up the region-level average normal and curvature calculations. As shown in Algorithm 1, we summarize the detailed algorithms for PFH-based over-segmentation as the following five steps:

1. 1.

   Select regions with labeled point the regions that have minimum curvatures as the initial seed regions.

2. 2.

   Utilizing fast KNN search, we obtain the neighbouring regions of the initial seed regions $r_{seed}$, and calculate the Affinity $A(r_{seed},r_{j})$ of the query center region $r_{seed}$ and the neighbouring regions $r_{j}$.

3. 3.

   If the local feature affinity is large enough (Condition 1), we assign the neighbouring region $r_{j}$ with the same label as $r_{seed}$.

4. 4.

   If Condition 1 is satisfied, denote the curvature of $r_{j}$ and $r_{seed}$ as $r_{j}$ and $r_{seed}$, and denote the the difference of $r_{j}$ and $r_{seed}$ as $\Delta r=\|r_{j}-r_{seed}\|$. If $\Delta r\leq\zeta$ (Condition 2), we set $r_{j}$ as the seed region. That means we set the regions $r_{j}$ as seed regions, only if the difference in curvature is small enough. Otherwise, we only assign $r_{j}$ the same class label as $r_{seed}$.

5. 5.

   If Condition 1 is not satisfied, we assign the neighbouring region $r_{j}$ with a different label from $r_{seed}$, and also assign $r_{j}$ as the seed region.

Finally, the regions with large similarities are merged with the training of the network. The algorithm is summarized in Algorithm 1. The loop will terminate if any of the following convergence conditions is satisfied, which are proposed as:

1. 1.

   All regions have been assigned with labels;

2. 2.

   There are no seed regions that can be added;

3. 3.

   Regions will not expand between two successive steps.

These three conditions are designed to assign as many regions with pseudo labels as possible, which also facilitates the following self-training and region-based neural network processing.

As shown in Figure 6, the output of backbone network gives the prediction of semantic segmentation with $\textbf{P}_{out}\in\mathbb{R}^{N_{i}\times C_{Seg}}$, where $C_{Seg}$ denotes the number of semantic categories and $N_{i}$ is the number of input points. For the limited annotation case, which means there are only a few annotated points (i.e. 0.2%) in a scene with approximately $2\times 10^{6}$ points, we propose the region-level similarity prediction strategy by simply adding a max pooling operation after our backbone network to offer the region-level predictions as shown in Figure 6. To be more specific, we have adopted $1\times 1$ convolution at the last layer of backbone network to obtain a feature $\textbf{P}_{out}\in\mathbb{R}^{N_{i}\times C_{Seg}}$, then we adopt max-pooling for each region to obtain the region-level prediction $\textbf{P}_{clu}\in\mathbb{R}^{N_{clu}\times C_{Seg}}$, where $N_{clu}$ is the initial number of regions obtained from over-segmentation. For instance segmentation, we obtain the region-level prediction $\textbf{P}_{Ins}\in\mathbb{R}^{N_{clu}\times C_{Ins}}$ for each instance based on point clustering method proposed in PointGroup, where the $C_{Ins}$ is the number of instance categories. Note that normalized scores in prediction of the merged clusters is added in each training iteration based on the similarity scores in both geometry and semantics among them. The calculation of the similarity scores in geometry/semantics is detailed in the following paragraph.

where $M_{\mbox{{\tiny{{color,i,j}}}}},M_{\mbox{{\tiny{{scale,i,j}}}}},M_{\mbox{{\tiny{{des,i,j}}}}},M_{\mbox{{\tiny{{seg,i,j}}}}}\in[0,1]$. The $M_{\mbox{{\tiny{{color,i,j}}}}}$, $M_{\mbox{{\tiny{{scale,i,j}}}}}$, and $M_{\mbox{{\tiny{{des,i,j}}}}}$ are the scores that are the normalized average intrinsic color, dimension, and 3D local descriptor-based similarities between the point cluster $i$ and the point cluster $j$, respectively. The average 3D local descriptor-based similarity $M_{\mbox{{\tiny{{des,i,j}}}}}$ is essentially the similarity of the region-level feature vector between the region $i$ and region $j$. The $M_{\mbox{{\tiny{{des,i,j}}}}}$ is calculate in the same way as the PFH-based affinity $A(R_{i},R_{j})$ calculation in Subsection 3.3. While the semantic similarity $M_{\mbox{{\tiny{{seg,i,j}}}}}$ between the $i_{th}$ and $j_{th}$ region is evaluated based on the output similarity of the two clusters:

where $\textbf{p}_{clu,i}$ and $\textbf{p}_{clu,j}$ are the corresponding predictions in $\textbf{P}_{clu}$ for neighbouring cluster $i$ and cluster $j$. We have designed the weight balancing strategy to avoid the noisy and low-quality pseudo labels at the beginning of training. The balancing weights $y_{1},y_{2},y_{3}\in[0,1]$ are set to values declining from a high value to a small value, i.e. $y_{1}=y_{2}=y_{3}=1-\frac{m_{i}}{{{N_{Total}}}}$, while $y_{4}\in\{0,1\}$ is set to the value $\frac{m_{i}}{N_{Total}}$. Where $m_{i}$ represents the current iteration in training, and $N_{Total}$ is total number of training iterations. This design means we firstly trust more on similarity of the local geometric properties and gradually trust more on the updated semantics relations in cluster-level predictions. We replace the Condition 1 in Algorithm 1 with the Condition 3: $S_{i,j}\geq 1.25$ & $\textbf{p}_{clu,i}\geq\gamma$ & $\textbf{p}_{clu,j}\geq\gamma$. And we substitute Condition 2 with Condition 4: $S_{i,j}\geq 1.5$, respectively. $\gamma$ is a confidence threshold to ensure that merely highly confident network predictions can be utilized for the network optimizations. Also, the K nearest neighbour in Algorithm 1 is conducted at cluster level. This design, combined with cluster-level similarity prediction strategy, utilizes weak labels as the guidance to increase the quality of generated pseudo label in training iteratively for both semantic and instance segmentation tasks. And the updated pseudo labels are shown in the third column of Figure 7 for ScanNet instance segmentation. From our further experiments in ablation studies, the traditional or learnt 3D descriptors play a great importance in capturing the local geometric properties, thus enhancing the 3D scene understanding performance.
Data Augmentation Submodule This submodule is inspired by a simple intuition that the network prediction should be consistent under diverse transformations including flipping, rotation, and even down-sampling. The details of data augmentation is provided in the Appendix. Based on the backbone network, for the original point clouds input $\textbf{P}_{in}$ and augmented point clouds input $\textbf{P}^{aug}_{in}$, we firstly obtain the final network semantic/instance predictions $\textbf{P}_{out},\;\textbf{P}^{aug}_{out}\in\mathbb{R}^{N_{i}\times C_{Seg}}$ respectively. For the task of instance segmentation, the outputs are $\textbf{P}_{Ins},\;\textbf{P}^{aug}_{Ins}\in\mathbb{R}^{N_{i}\times C_{Ins}}$ respectively. The KL divergence is universally adopted to evaluate the difference between two probabilistic distributions. In our work, we utilize the JS divergence instead because of its symmetry property, which means it remains constant when two distributions are very distant to each other. Also, the distributional loss for regression such as the JS divergence is easier to optimize with improved gradient and reduces the overfitting problem compared with the mean square error (MSE)-based loss. It is also demonstrated by experiments that the JS divergence outperforms traditional mean square error-based loss in downstream scene understanding tasks. The final JS Divergence Loss for semantic segmentation is formulated as:

where $N_{com}$ is the number of random sampled common intersectional points between $\textbf{P}_{out}$ and $\textbf{P}^{aug}_{out}$. And the same goes for instance segmentation. We keep the number of random sampled points to 1000 in all our experiments for the efficiency consideration. And $\sigma$ is the Softmax function with normalization to produce probabilistic scores for each class. After applying the data augmentation constraints, we aim at ensuring that the distribution of the probabilistic scores of segmentation will remain consistent between the $\textbf{P}_{out}$ and $\textbf{P}^{aug}_{out}$ after various data transformations. To be more specific, the transformation invariance can be achieved.
Pseudo Segmentation Submodule Finally, the network is also guided by the generated pseudo label in both semantic and instance segmentation tasks, which can be formulated as: $L^{sem}_{\mbox{{\tiny{{WSL}}}}}=\frac{1}{N_{i}}\sum_{i=1}^{N_{i}}\textit{{CE}}(\textbf{P}_{out},\textbf{P}_{gt})\mathds{1}(\textbf{p}_{i})$. Where $\textbf{P}_{out}$ is the segmentation output prediction, and $\textbf{P}_{gt}$ is the ground truth supervision provided by the generated pseudo labels in each training iteration. $\mathds{1}(\textbf{p}_{i})\in\{0,1\}$ indicates whether the point has been given a pseudo label in the current training iteration. The final optimization takes losses from all above-mentioned submodules into account, formulated as $L_{Seg}=L^{js}_{Aug}+L^{sem}_{\mbox{{\tiny{WSL}}}}$. The self-training is used for network learning with pseudo labels, and the network is optimized in an end-to-end manner for semantic/instance segmentation.

The object detection network is designed based on widely adopted VoteNet. Based on VoteNet, we propose Dice loss to guarantee tighter aggregations of points within the same cluster, and strict geometric separations of points in diverse clusters. Note that for the object detection, our method operates in an unsupervised manner for doing instance segmentation, followed by our regression submodule to realize object detection. Note that the same as the semantic/instance segmentation branch, the pseudo segmentation submodule is utilized to perform instance segmentation, and the data augmentation submodule is utilized to ensure the transformation invariance. Different from segmentation branch, $\textbf{P}_{out}\in N_{i}\times(C_{det}+1)$, where $(C_{det}+1)$ is the number of object classes plus one for backgrounds in detection. Similarly, in order to obtain the object-level prediction, We apply max-pooling to $\textbf{P}_{out}$ to obtain $\textbf{P}_{det}\in N_{\mbox{{\tiny{{R}}}}}\times(C_{det}+1)$, where $N_{\mbox{{\tiny{{R}}}}}$ is the number of objects which are given true labels rather than pseudo labels. Then we add $1\times 1$ convolution and max pooling after $\textbf{P}_{det}$ to produce $\textbf{P}_{cls}\in\mathbb{R}^{N_{cls}}$, and it predicts the presence of object or not with a scene:

The $L_{\textit{{CE}}}$ is the cross-entropy loss, and $C_{cls}$ is the number of object classes within the scene. In this way, object presence within a scene can serve as the supervision for similarity predictions among regions benefiting from the self-supervision provided by scene object classes.
Regression Submodule Object detection can take advantage of the supervision from instance segmentation because object proposals can be directly obtained from the results of the instance segmentation. The axis tightly aligned bounding box of each instance provided by the results of instance segmentation is selected as the initialization of pseudo ground truth bounding boxes for object detection. At the same time, the Dice loss li2019dice can be utilized to evaluate intersections between the predicted regions and ground truth regions for regression purpose. Note that other submodules are the same as the semantic/instance segmentation branch. Denote the Dice loss as $L_{Dice}$ and the same losses as segmentation branch as $L_{Seg,2}$, the total optimization function for detection is formulated as $L_{Det}=L_{Seg,2}+L_{Dice}+L_{cls}$. Our network is optimized in an end-to-end manner on a single 1080Ti GPU for three scene understanding tasks.

We have also compared the quality of pseudo labels provided by diverse SOTAs approaches as shown in Figure 10. We unified the backbone to SparseConv graham20183d and for a fair comparison. It is demonstrated that the key success our proposed approach lies in that can produce high-quality pseudo labels, which results in the high performance in 3D understanding tasks.

The existing works GPC jiang2021guided , OTOC liu2021one focuses on contrastive learning. The network directly learns the instance discrimination capacity using the positive and negative samples, and pull the positive samples together while pushing the negative samples away in embedding space. According to our experimental results, the constraints in contrastive learning jiang2021guided ; liu2021one is too hard, because it minimizes the distance between the positive samples and maximize the distance between negative samples. While in real circumstances, the same semantic category might have some distance, while the different semantic category can be not that far away. As shown in Figure 10, according to our experiments, the contrastive learning-based approaches GPC and OTOC can generate low-quality pseudo label in self-training if the initial pseudo label quality is not good enough.

While in our proposed self-training pipeline, only the similar regions with high confidence propagate labels to each other, which guarantee the quality of pseudo labels in the first few training iterations. Also, we have the following design to avoid low-quality initial labels in the self-training. In the first few training iterations, the quality of labels is low, we choose to firstly believe more in the traditional or learnt local 3D descriptor-based features $M_{\mbox{{\tiny{{color,i,j}}}}},M_{\mbox{{\tiny{{scale,i,j}}}}},$ and $M_{\mbox{{\tiny{{des,i,j}}}}}$ to do the region merging. And the quality of the pseudo label becomes higher with the increasing of training iterations, then we trust more on the semantic similarity $M_{\mbox{{\tiny{{seg,i,j}}}}}$ to update the pseudo labels. According to our ablation studies, this strategy has successfully avoided the low-quality initial labels and improved the final performances.

Finally, we have shown the quality of the pseudo labels provided by our proposed region merging based approach compared with the SOTA methods OTOC liu2021one , GPC jiang2021guided , and PSD. The GPC works for the limited reconstruction case, we have extended the method to make it suitable for the limited annotation case. As shown in Figure 10, our proposed method has outperformed the state-of-the-art approaches OTOC liu2021one , GPC jiang2021guided , and PSD by a large margin for both the pseudo label generation and the offset predictions. Therefore, our proposed approach can provide more high-quality and noise-free pseudo labels for the downstream 3D scene understanding tasks. The high-quality and noise-free pseudo labels are certainly beneficial for the scene understanding downstream tasks for the fact that more correctly labeled points are used for the self-training. It demonstrates the superior effectiveness of our proposed region merging design.

A main component of our proposed network framework is the 3D descriptor which capture the geometries of the local structure. In this Subsection, we have given a very comprehensive comparisons of the performance of different local descriptors not only on the task of over-segmentation, but also on the task of final semantic segmentation, instance segmentation, and object detection. First of all, we compare different methods on the task of over-segmentation. To test the rotational robustness of various 3D local descriptors, we have done detailed experiments for the original scene and the rotated scene respectively. We adopt random rotation of $0^{\circ}$ to $180^{\circ}$, and then evaluate the quality of over-segmentation, semantic segmentation, and instance segmentation in terms of rotational robustness. The experimental results of instance segmentation are shown in Table 2. The left of ’/’ shows the performances for the original scene, and the right of ’/’ shows the performances for the rotated scene. It can be seen that the above illustrated 3D descriptors can be integrated seamlessly to our proposed weakly supervised learning framework to fulfill the tasks of 3D scene understanding. According to our experimental results, the visual similarity based 3D descriptor chen2003visual has poor performance in the tasks of oversegmentation, semantic segmentation, and instance segmentation. It can be explained by the fact that the visual similarities of the 3D structures from diverse angles can be not that similar, and it results in poor local geometrical description capacity and rotation robustness. Also, the signature based approaches including HKS sun2009concise and WKS aubry2011wave are also not very robust to rotation because they do not explicitly consider rotational robustness in their formulations. The HKS (Heat Kernel Signiture) sun2009concise ; gebal2009shape utilizes the heat diffusion process to capture the extreme surface change. The SIKS bronstein2010scale designs a scale-invariant heat kernel descriptor based on the diffusion scale space analyses. The performances of SIKS bronstein2010scale for the tasks of oversegmentation, semantic segmentation and instance segmentation are marginally better than HKS sun2009concise for its effective scale invariant heat kernel signature designs. The SIKS bronstein2010scale is a scale-invariant version of HKS that can maintain invariance under a wide range of transformations the shape undergoes, therefore, the semantic segmentation and instance segmentation performances of SIKS bronstein2010scale are maintained even if randomly rotated. WKS aubry2011wave embeds and separates information from diverse Laplacian eigen frequencies by varying the energy of the quantum mechanical particle. The WKS aubry2011wave is invariant to isometries and very robust to small non-isometric deformations compared with HKS. Also, for the fact that WKS aubry2011wave permits access even to very high frequency information, it results in marginally better local description capacity and provides more accurate local matching and registration than the HKS. Our experimental results also demonstrates the effectiveness of WKS compared with HKS. The results of oversegmentation and 3D scene segmentation of WKS is comparable to SIKS as shown in Table 2.

Furthermore, according to our experimental results shown in Table 2, our proposed simple contrastive learning-based 3D descriptor has outperformed the typical learning-based descriptors such as Point-SIFT and S-SPG in the tasks of semantic segmentation, instance segmentation, and oversegmentation. The success of the constrative learning-based local descriptor can be ascribed to the fact that contrastive learning has the capacity of capturing very discriminative feature representations of the local 3D geometry in an unsupervised manner. Compared with other learning-based 3D descriptors such as Point-SIFT, and S-SPG mentioned in Section 3, the discrimination of the positive and negative samples in the contrastive learning can be realized in a clearer way, thus resulting in a slightly better result. We have also compared our contrastive learning-based local 3D descriptor with a recently proposed 3D descriptor Predator, which is specially designed for registering point cloud scans with a low overlap. It can be demonstrated that our proposed simple contrastive learning-based 3D descriptor can achieve a slightly inferior and sometimes comparable performance compared with the Predator, which demonstrates the effectiveness of our 3D descriptor design. Also, the rotational robustness of our proposed contrastive learning-based 3D descriptor can sometimes be even better than the Predator huang2021predator , which can be explained by the fact that the contrastive learning has inherently guaranteed the rotational robustness because the selection of the positive and the negative samples is agnostic to viewing angles. In summary, it can be demonstrated from Table 2 that our proposed framework can be integrated seamlessly with various of traditional or learnt 3D descriptors to achieve 3D scene understanding. Also, our proposed adapted PFH-based 3D descriptor and contrastive learning-based 3D descriptor have satisfactory performance for the oversegmentation and instance segmentation tasks in 3D scene understanding. It is validated by experiments that our adapted PFH-based 3D descriptor and proposed contrastive learning-based 3D descriptor also have good rotational robustness. The performance will not drop much even if the random rotation of $0^{\circ}$ to $180^{\circ}$ is applied.

Last but not the least, we have also shown the experimental results just after the learning-based region merging of various approaches. The results for indoor ScanNet are shown in Figure 8. and the results for outdoor SemanticKITTI are illustrated in Figure 9. For the SemanticKITTI, we have a very simple but effective trick to tackle low-density outdoor LiDAR points: discarding all regions containing less than $N_{ths}$ points, i.e. not using these regions in all our network modules ($N_{ths}=100$ empirically for SemanticKITTI). Thus, as shown in Figure 9, isolated regions with fewer points than $N_{ths}$ are successfully discarded for reliable and robust region-level predictions. As shown in Fig. 9, compared to original PFH-based 3D local descriptors, our adapted PFH-based method and contrastive learning-based method better distinguish similar semantic classes such as road and sidewalk, revealing that our proposed methods can have a better local feature description capacity. It is demonstrated qualitatively that both our proposed adapted PFH-based and contrastive learning-based local description approaches can provide more homogeneous and consistent region merging results compared with the traditional PFH-based approach rusu2008aligning and the voxel connectivity-based approach papon2013voxel . It implies that our proposed adapted PFH-based and our proposed contrastive learning-based local description approaches can extract more discriminative feature representations of local 3D structures compared with previous approaches rusu2008aligning ; papon2013voxel . According to our experimental results in Table 2, our proposed adapted PFH-based 3D local descriptor provides better performance in oversegmentation, and instance segmentation compared with various traditional faeture descriptors including HKS sun2009concise , SIKS bronstein2010scale , FPFH rusu2009fast , and WKS aubry2011wave . Our proposed adapted PFH-based 3D feature descriptor provides comparable performance with the learning-based approaches Point-SIFT jiang2018pointsift , and S-SPG landrieu2019point , which demonstrates its strong 3D local geometry description capacity and its effectiveness in 3D scene understanding. Also, the performance of rotational robustness of it is superior compared with other traditional and learnt 3D descriptors.

On the other hand, our proposed contrastive learning-based 3D descriptor attains comparable performance compared with the Predator huang2021predator . It demonstrates that our proposed contrastive learning-based approach is a good choice for describing local geometry. In summary, it can be demonstrated in Table 2 that our proposed learning-based region-merging approach can be integrated seamlessly with various traditional or learnt 3D descriptors to achieve effective 3D scene understanding. In the following experiments, we choose to use our proposed adapted PFH-based 3D local descriptor to conduct region merging for its high efficiency and for the fact that it does not need additional training data and can operate in an unsupervised manner.

We have also done experiments of object detection on KITTI and Waymo benchmarks with two annotated bounding box per ten scenes to keep fairness in comparisons, which means there are merely approximately 1% labeled bounding boxes. Also, we have tested the circumstances of one box case, which denotes merely one randomly selected bounding box is annotated within ten point cloud scenes. It can be demonstrated that our framework outperforms other weakly supervised counterparts by a large margin. As shown in the quantitative experimental results in Table 5, our proposed approach can also achieve superior performance for weakly supervised object detection compared with current weakly supervised learning approaches for the task of object detection.

In this Subsection, we conduct more Studies on the results of transfer learning. As in demonstrated in our experimental results in Table 7, the feature similarities of 3D local descriptor $M_{\mbox{{\tiny{{des}}}}}$ plays a significant role in improving the generalization ability across different domains. For example, when transferring the model trained on S3DIS to ScanNet for the task of instance segmentation with 0.2% label, the performance merely drops by 3.4% and 4.5% for S3DIS $\rightarrow$ ScanNet and ScanNet $\rightarrow$ S3DIS, respectively, demonstrating that our proposed model has good transfer learning capacity. And noticeably, for S3DIS $\rightarrow$ ScanNet, the performance drops ($\downarrow 12.9$) when removing the similarity of our adapted PFH-based 3D local descriptor $M_{\mbox{{\tiny{{des}}}}}$ is much larger than the drop ($\downarrow 8.4$) when removing the similarity in semantic predictions $M_{\mbox{{\tiny{{Seg}}}}}$. It demonstrates that the our proposed adapted PFH-based 3D local feature descriptor-based similarity is of greater importance to the model generalization capacity compared with high-level semantic for the indoor case. Finally, when substituting the adapted PFH-based $M_{\mbox{{\tiny{{des}}}}}$ with predator-based $M_{\mbox{{\tiny{{des}}}}}$, the perofrmance will drop a little. It demonstrates the generalization capacity of our adapted PFH-based descriptor compared with learning-based ones.

In summary, according to our experimental results in Table 7, the $M_{\mbox{{\tiny{{des}}}}}$ is of great significance to the model generalization capacity for the tasks of semantic/instance segmentation and object detection. It is demonstrated that the generalization capacity of our adapted PFH-based 3D descriptor is better compared with learning-based descriptor Predator. It is also validated that the similarity in 3D local descriptor $M_{\mbox{{\tiny{{des}}}}}$ is even more robust and significant to the overall performance than the learnt high-level semantic features similarities $M_{\mbox{{\tiny{{Seg}}}}}$. Removing $M_{\mbox{{\tiny{{des}}}}}$ results in a significant performance drop in transfer learning.