Deformable PV-RCNN: Improving 3D Object Detection with Learned Deformations

3D object detection from point clouds is critical for autonomous driving and robotics. We build on the success of PV-RCNN [7], a state-of-the-art 3D object detector.

Motivation Part of PV-RCNN’s success is due to the randomly sampled keypoints which capture multi-scale features for proposal refinement while retaining fine-grained localization information. However, random sampling is not effective over potentially ambiguous scenes. For example, ‘pedestrians’ and ‘traffic poles’ can be very hard to distinguish in point clouds. In this case, we wish to align the keypoints towards the most discriminative areas, so that principal features for a pedestrian can be highlighted. Similarly, the scales for cars, pedestrians and cyclists are very different. While multi-scale feature aggregation is advantageous for image features, the non-uniform density of point clouds makes it hard to detect them with a single model. We wish to adaptively aggregate and focus on their most salient features at various scales. Lastly to handle clutter and avoid false positives, for example, to avoid detecting all seated individuals as cyclists, we need to pick up on unevenly distributed contextual information.

Contribution We construct Deformable PV-RCNN, a 3D detector that handles sparsity of LIDAR points, is adaptive to non-uniform point cloud density especially at long distances and can address clutter in real-world traffic scenes. We show that we can outperform PV-RCNN across different categories and especially at long-distances on the KITTI 3D object detection dataset.

Our 3D detection pipeline is presented in Fig. 1. It consists of an Adaptive Deformation module (Fig. 2) and a Context Fusion module (Fig. 3).

Adaptive Deformation The $n$ sampled keypoints (shown in yellow in Fig. 1) have a 3D position $v_{i}$ and a feature vector $f_{i}$ corresponding to either of Conv3 or Conv4 layers. Our module computes updated features $f^{\prime}_{i}$ as follows:
$f^{\prime}_{i}=\frac{1}{n}$ReLU$(\sum_{j\in\mathcal{N}(i)}{W_{\textit{offset}}(f_{i}-f_{j})\cdot(v_{i}-v_{j})})$, where $\mathcal{N}_{i}$ gives the $i$-th keypoint’s neighbors in the point-cloud and $W_{\textit{offset}}$ is a learned weight matrix. We then obtain the new deformed keypoint positions as $v^{\prime}_{i}=v_{i}+$tanh$(W_{\textit{align}}[f^{\prime}_{i}])$, where $W_{\textit{align}}$ is a learned weight matrix. This is similar to [4], [6]. We then proceed to compute the features for the deformed keypoints using PointNet++ similar to the PV-RCNN pipeline.

Context Fusion This module uses context gating to dynamically select representative and discriminative features from local evidence, highlighting object features and suppressing clutter. Given a keypoint feature $f_{i}$, the modulating feature is obtained as $g=\sigma(W_{\textit{gate}}f_{i}+b_{\textit{gate}})$ and the context-gated feature is computed as $f^{g}_{i}=g\odot W_{\textit{fc}}f_{i}$, where $W_{\textit{gate}}$, $b_{\textit{gate}}$, $W_{\textit{fc}}$ are learned from data [2].

The 3D object detection benchmark of KITTI [3] contains 7481 training samples. Following [1], we divide them into 3712 training samples and 3769 validation samples. We report our results on the validation (val) split of KITTI.

Comparison with state-of-the-art methods Table 1 shows the performance of Deformable PV-RCNN on the KITTI val split. We run the officially released checkpoints for SECOND, Part-A2 and PV-RCNN.¹¹1https://github.com/open-mmlab/OpenPCDet Our method outperforms the state-of-the-art on the cyclist class. Compared to PV-RCNN, it performs better on pedestrian and cyclist class by 3.5 % and 4%, respectively, and on par for the car class.

Ablation Studies Table 2 validates the effectiveness of learning deformable offsets and context gating by comparing with a retrained PV-RCNN baseline. We find that deformation prediction contributes to the performance increase over all classes, but especially for the pedestrian class, which is probably due to being able to focus on representative features like arms and legs. Context fusion also increases performance over all classes, indicating that contextual information is important for refinement.

Comparison with PV-RCNN Table 3 shows that our model outperforms the baseline at different distances for the pedestrian and cyclist class. We also find a 2% increase in AP for the car class at long distances where the scans are sparse and context information becomes important. Fig. 4 shows that similar AP performance can be obtained with fewer keypoints as compared to the baseline across all classes, which is probably due to their deformability to cover representative locations. Fig. 5 shows qualitative 3D detection results.

We present Deformable PV-RCNN, a 3D object detector for detecting 3D objects from raw point cloud. Our proposed deformation prediction method adaptively aligns the keypoints encoding the scene towards the most discriminative and representative features. The proposed context gating network adaptively highlights relevant context features thereby favouring more accurate proposal refinement. Our experiments show that Deformable PV-RCNN outperforms PV-RCNN in various challenging cases on the 3D detection benchmark of KITTI dataset, especially benefiting smaller objects and complex scenes more.