Deep Positional and Relational Feature Learning for Rotation-Invariant Point Cloud Analysis

Point cloud is a basic representation for 3D shape, and point coordinates are taken as raw features by most point-based networks to carry out tasks on aligned shapes. PointNet [8] is the first effort that takes point coordinates as raw features and embeds positional information of points to deep features followed by max-pooling to be a global shape descriptor. Afterwards, a series of works attempt to improve feature embedding by designing novel deep networks on point clouds. PointNet++ [1] builds hierarchical architecture with PointNet as local encoder. More works such as SpiderCNN [9], PointCNN [2], KCNet [21], PointConv [22], RS-CNN [23] learn point cloud features with various local convolutions starting from point coordinates or additional geometric features. Though these works have achieved state-of-the-art performance for various shape analysis tasks, they are sensitive to shape rotation. In our work, we build a novel network with building blocks being able to achieve rotation-invariance, but with positional raw features as inputs. Our network can explore and encode positional information of 3D shapes while precluding the disadvantage of rotation-sensitiveness.

There are several ways to improve rotation-robustness. Data augmentation by randomly rotating shapes in training set is widely applied in training 3D networks as in [8, 9]. Furthermore, transformer network [24] was generalized and utilized in 3D recognition [8, 25, 26]. The transformer implicitly learns alignments among shapes either in 3D space or feature space which may reduce the impact of shape rotations. As shown in [18], these techniques commonly improved robustness but can not achieve rigorous invariance to rotations.

Rotation-invariant geometric features can be taken as network inputs. Traditionally, these hand-crafted features are widely utilized to represent 3D shapes in [27, 28, 29, 30, 31], and further introduced as inputs of deep networks [18, 20, 32]. Most of them utilize distances and angles [20, 32] and sine / cosin values of the angles [18] as raw point-wise features. They encode relations among points and achieve invariance to arbitrary shape rotations. However, the geometric features may lose discriminative clues that are contained in point positions / orientations.

Another way to achieve rotation-robustness is to learn rotation-equivariant features. [33] built filters on spherical harmonics and learned locally equivariant features to 3D rotations and translations. Group convolution [34] is a generalized convolution to achieve rotation-equivariant feature learning based on rotation-group. This idea is extended to 3D domain [35] with various discretized rotation-groups. Spherical CNN [17] and $a^{3}$SCNN [19] defined group convolution on sphere which can be taken as an approximation to infinite group SO(3). They also require the input of the networks to be rotation-equivariant if regarding features as functions over shapes. Commonly, these methods lead to rotation-robustness by global aggregation such as average-pooling or max-pooling, but rigorous rotation-invariance is not ensured due to the discretized rotation groups.

Instead of requiring rotation-invariant geometric features as inputs, our network can directly learn positional features from point coordinates as inputs while achieving rigorous rotation-invariance based on PFE-block. It embeds the input shape into a rotation-invariant pose space, from which we derive a discriminative pose by pose selector to be sent to positional feature extractor, and this operation is rotation-invariant and justified to be effective in experiments.

PR-invNet takes geometric features similar to [18] as additional raw input features complementary to point coordinates¹¹1Even without geometric features, our network would be still rotation-invariant.. As shown in Fig. 3, $c$ and $o$ are the centers of the global shape and a local patch respectively. To compute geometric features of point $o$, we first consider its neighboring points. The feature of each of its neighboring point $q$ is computed as $[|\overrightarrow{cq}|,|\overrightarrow{oq}|,cos(\alpha),cos(\beta)]$, with $[\cdot]$ being the concatenation operation. Then we concatenate features of all neighboring points together with the distance of $|\overrightarrow{co}|$ as the geometric feature of point $o$, i.e.,

where $N(o)$ denotes set of neighboring points of $o$. For simplicity, we denote it as $\{g_{i}\}_{i=1}^{N}$ with $N$ as number of points. It can be noted that $g_{i}$ is rotation-invariant, please refer to supplemental material for proof. In our work, for each given point, the above geometric features are extracted and concatenated over a multi-scale neighborhoods (three neighborhoods determined by Euclidean distance with 8, 16, 32 points), and 8 neighboring points are uniformly sampled at each scale. Then the geometric feature of each point is in length of $3\times 4\times 8+1=97$.

The PFE-block aims to perform rotation-invariant feature embedding by incorporating positional information encoded in point coordinates. Since point coordinates are sensitive to shape rotations, one solution is to take all rotated versions (i.e., poses) of a shape into consideration, and fuse extracted features from them by global pooling. However, this is inefficient since there are infinite number of rotations in SO(3) to ensure rigorous rotation-invariance. We propose an idea that first maps input shape to a rotation-invariant pose space by pose expander, then selects a representative pose by pose selector. Point coordinates of selected pose are utilized to extract positional features by feature extractor.

The relational feature embedding block (RFE-block) is specifically designed to explore and learn point features by modeling interactions of neighboring points. For each point $i$, we aggregate the features of its local neighborhood with $K$ points including point $i$ by convolution. Assuming the neighboring points are indexed by $k$ sorted by increasing Euclidean distance to point $i$, and corresponding point features are $\{f_{k}\in\mathcal{R}^{d}\}_{k=1}^{K}$, the updated feature $f^{\prime}_{i}\in\mathcal{R}^{d^{\prime}}$ for point $i$ is

where $f_{kj}$ is the $j$-th feature channel of the $k$-th neighboring point, $f^{\prime}_{il}$ is the $l$-th channel of the updated feature $f_{i}^{\prime}$. $W_{kjl}$ is defined as

where $C_{k}^{p},C_{k}^{g},C_{k}^{f}\in\mathcal{R}^{K}$ are $K$-d vectors denoting correlations of centered point coordinates, geometric features, point features respectively between $k$-th point and all $K$ points in local neighborhood, followed by $l_{2}$-normalization. For example, for centered point coordinates, i.e., coordinates subtracted by that of center pixel $i$, $C_{k}^{p}=\text{Normalize}([p_{k}^{\top}p_{1},p_{k}^{\top}p_{2},\cdots,p_{k}^{\top}p_{K}])$, where we use $p$ to represent the centered point coordinates. $C_{k}^{g},C_{k}^{f}$ are similarly defined as inner products of corresponding features. $\Phi(\cdot)$ is modeled as MLP, embedding relationships of neighboring points to convolution weight with learnable parameters $\Theta_{kjl}$. It is easy to verify that weights in Eqn. (7) are rotation-invariant because the inputs of $\Phi$ are rotation-invariant defined by inner products.

PR-invNet is designed by concatenating rotation-invariant PFE-block and RFE-block. As shown in Fig. 2, given a shape, we first initialize the point-wise features by initial feature embedding, followed by three stages concatenating PFE-block and RFE-block to extract positional and relational features. For classification, we use an additional shape coarsening operation at each stage to construct a hierarchy of coarsened points, implemented by farthest point sampling with sampling rate of 1/2. The intermediate features generated by PFE-blocks and RFE-blocks of different stages are concatenated and aggregated by a MLP set as two layers of FC+BN+ReLU+Dropout, followed by a FC to output confidence for shape categorization or segmentation. The number of hidden units of the first two FCs are 512, 256, and dropout probability is 0.5. The number of neighboring points in Eqns. (5-6) is 16. For an input shape of PR-invNet, we conduct PCA to normalize shape pose only one time, and all PFE-blocks in different stages derive rot-space by Eqn.(4) starting from point cloud of this normalized pose or its coarsened point clouds. More details of pose selector $\Psi$, feature extractor $\Gamma$, convolution filters $\Phi$, $\Theta$ in RFE-block are introduced in supplementary material.

We mainly compare with point-based methods and rotation-robust methods.

ModelNet40. As done in Spherical CNN [17], we compare in the following three modes. (1) Both training and test on data augmented by azimuthal rotation (z/z); (2) Training with azimuthal rotation and test with arbitrary rotation (z/SO(3)); (3) Both training and test with arbitrary rotation augmented data (SO(3)/SO(3)). The results are presented in Table 1. The traditional point-based methods, including PointNet [8], PointNet++ [1], PointCNN [2], DGCNN [10], perform best in z/z mode, but decline sharply when testing on data augmented with arbitrary rotations (z/SO(3)). Though using arbitrarily augmented training data in SO(3)/SO(3) mode, their performances still drop with large gaps. For these rotation-robust methods including Spherical CNN [17], $a^{3}$SCNN [19], RIConvNet [20], ClusterNet [18] and RRI²²2RRI is an essential part of ClusterNet, and the codes are provided by the authors.-PointNet++, they are more robust in different modes. Considering that Spherical CNN and $a^{3}$SCNN rely on discretized angles in the sphere, they are not rigorously rotation-invariant and performance drops in modes (2)-(3). ClusterNet, RIConvNet and RRI-PointNet++ are rotation-invariant relying on input geometric features. Our PR-invNet is also rotation-invariant, and it achieves highest performance in modes (2-3) and second highest performance in mode (1) than all the rotation-robust methods, demonstrating its superiority for arbitrarily rotated shape classification.

ShapeNetCore55. We compare with point-based methods including PointNet [8], PointNet++ [1], DGCNN [10], RS-CNN [23], SpiderCNN [9], and also rotation-robust Spherical CNN [17] and rotation-invariant RIConvNet [20], RRI-PointNet++. The results are presented in Table 2. As shown in Table 2, we achieve best performance in both aligned and perturbed datasets, compared with Spherical CNN, RIConvNet and RRI-PointNet++. Among all methods, we achieve highest accuracy on the perturbed dataset.

We next apply PR-invNet to point cloud segmentation to predict the part label of each point. We use the same experimental settings as RIConvNet [20], and mean per-class IoU (mIoU, $\%$) are presented in Table 3. Compared with all rotation-sensitive point-based methods and rotation-robust methods, our PR-invNet outperforms them significantly in both SO(3)/SO(3) and z/SO(3) modes, i.e., more than $2.7\%$ and $4.1\%$ mIoU improvements. We also report the per-class IoU for above settings in z/SO(3), SO(3)/SO(3) modes respectively in the supplemental material, and our PR-invNet achieves better performance than the rotation-robust RIConvNet on all categories. In Fig. 6, we show the segmentation results in z/SO(3) mode of several objects as well as the corresponding ground truth labels, and our predictions labels are reasonable and close to ground truth.

In this section, we conduct ablation study on ModelNet40 to justify the effects of our network design, and also test the robustness of PR-invNet to noises.

Effect of proposed blocks. To evaluate the effectiveness of geometric feature extraction, PFE-block, and RFE-block, we conduct experiments that utilize networks without above blocks respectively, i.e., PR-invNet-noGeo, PR-invNet-noPFE, PR-invNet-noRFE, on ModelNet40, and present the experiment results in Table 4.2. Compared with PR-invNet, the networks of PR-invNet-noGeo, PR-invNet-noPFE, PR-invNet-noRFE achieve lower performance, demonstrating the need for each component. Note that all networks of above variants are also rigorously rotation-invariant, and they also achieve better performance than most of the rotation-sensitive and rotation-robust methods whose results are presented in Table 1 in the SO(3)/SO(3) and z/SO(3) modes.

Effect of stage number. To evaluate the effect of number of stages in PR-invNet, we compare performance of PR-invNet-$i$ with $i=0,1,2,\cdots,5$, and $i$ denotes number of stages excluding initial feature embedding stage. As shown in Fig. 4.2, network with 3 stages achieve the highest accuracy. Deeper architectures have larger capacity but marginally decreased performances. This phenomenon was also observed in other graph CNNs [41], and training of deeper networks on point clouds deserves to be more investigated in future work.

Design of global shape feature. As shown in Fig. 2, learned features from all PFE-blocks and RFE-blocks are aggregated by MLP for classification. We evaluate PR-invNet-PFE and PR-invNet-RFE that only utilize features from PFE-blocks and RFE-blocks respectively, achieving $88.5\%$, $87.4\%$ classification accuracies on ModelNet40. Result using only features of the last RFE-block is $88.3\%$. This demonstrates that concatenating all features of PFE-block and RFE-block is more effective.

Effect of PCA-normalization and rot-space. In PR-invNet, we take PCA to normalize shape as network input. We also conduct experiments with networks without PCA-normalization and only with PCA (without using sym-space and rot-space), i.e., PR-invNet-noPCA, PR-invNet-PCA. Their classification accuracies in ModelNet40 with z/z mode are $90.0\%$ and $89.1\%$ respectively, but drop to $44.3\%,74.8\%$ in z/SO(3) mode, which are not rotation-invariant. In SO(3)/SO(3), our pose selector over rot-space upon PCA($89.2\%$) outperforms pure PCA for alignment($88.3\%$). Removing both PCA and pose selector achieves $87.3\%$, showing our pose selection is more effective than PCA.

Effect of rotation group $G_{R}$. In pose expander of PFE-block, we use the rotation group $G_{R}$ to construct rot-space, which is taken as alternating group $A_{5}$ of a regular dodecahedron (or icosahedron). Other alternating (symmetry) group can also be used, such as identity group ($I_{R}$), or $A_{4}$, $S_{4}$ [36] respectively corresponding to regular tetrahedron, cube (or octahedron). The classification accuracies using these groups are in Fig. 10. Compared with other groups, $A_{5}$ has the maximum number of elements, resulting in largest rot-space and achieves the highest accuracy. Operating on this largest space, the pose selector has more probability to select a better pose for shape recognition.

Effect of pose selector. In PFE-block, we use a pose selector to select the representative shape pose in rot-space. However, we can also extract features from all shapes from rot-space and then perform max-pooling or average-pooling over them to aggregate features. We compare our pose selector with these aggregation methods and present the classification accuracy and computational cost in Table 6. Since it is infeasible to extract features from all shapes in rot-space containing 120 poses, we compare these variants on the sym-space containing eight shape poses, i.e., rot-space with identity rotation group. We experiment with batch size 12, and as shown in Table 6, our pose selection method achieves higher performance while needing less GPU memory and computational time.

Effect of head number in $\Psi$. For pose selector $\Psi$, we design it as a multi-head neural network, which has 200 heads. We conduct experiments to demonstrate the effect of head numbers. As shown in Fig. 10, the $\Psi$ with head number as 200 achieves higher performance. Note that $\Psi$ with multi-heads all achieve better accuracies than that with one head.

Design of RFE-block. In RFE-block, we design convolution filters based on three terms, i.e., the relations of point position, geometric feature, and point feature in Eqn. (7). In Table 6, we show our networks using only one of these terms, resulting in networks of PR-invNet-P, PR-invNet-G, PR-invNet-F respectively. We also present result of PR-invNet-g only using geometric feature in Eqn. (7). Their classification accuracies are all inferior compared with full version. We also conduct experiment that without normalization in Eqn. (7), achieving classification accuracy as $88.6\%$.

Robustness to noise. We train PR-invNet on ModelNet40 training dataset and test it on test data with various levels of noise under the z/SO(3) mode. We add Gaussian noise with different standard deviation (Std) on each point (coordinates within a unit ball) independently. The overall classification accuracies are in Fig. 10. PR-invNet keeps robustness under noise with Std of $0.01$.

Visualization for pose selector. In PFE-block, we design multi-head score network $\Psi$ to select a pose from rot-space $H_{SR}(P)$ based on scoring each shape pose by its maximum score belonging to multiple heads. Therefore the selected pose from rot-space corresponds to a certain head. Here we illustrate selected shape poses organized by corresponding heads highlighted in different colors in Fig. 11. We observe that shapes corresponding to each head have similar poses. The multi-head score network may enable to learn to align the shapes to clusters of poses using heads in multi-head score network.