Range-Aware Attention Network for LiDAR-based 3D Object Detection with Auxiliary Point Density Level Estimation

The fixed-shaped anchor regression approach [14][13] [28] has been proposed such that intermediate features can be extracted. Girschick et al. [14] perform independent forward propagation on each region proposal. Region proposal task and object detection task are compounded in [13], which avoids heavy convolution operation overhead and preserves the precision throughout the network. To decrease the computation further, Ren et al. [28] introduce a separate network to predict region proposals. This removes the time-consuming selective search algorithm and lets the network learn the region proposals. These works employ the two-stage detection scheme. However, the region-proposal-joint process (e.g. RPN) is still slow and memory intensive, and thus not compatible with real-time detection. To reduce computation as well as memory footprint during both training and inference phases, one stage detectors are exploited in many works [22][27][20], which can train object classification and bounding box regression directly, without pre-generated region proposals during training, and maintain a lower memory footprint at the same time. Redmon et al. [27] redefine object detection as a single regression problem, which employs an end-to-end neural network to do a single forward propagation to detect objects. Liu et al. [22] develop a multi-resolution-anchor technique to detect objects of a mixture of a scale as well as learn the offset to a certain extent than learning the anchor. Lin et al. [20] present a focal loss to tackle the dense and small object detection problem while dealing with class imbalance and inconsistencies. Zhou and Tuzel [50] and Lang et al. [18] propose neural networks for point clouds, which open up new possibilities for 3D detection tasks. It should be noted that, although the aforementioned studies are real-time detection works, they actually involve the computation of all anchors’ regression corresponding for different anchor scales.

To address the computation overhead and hyperparameter redundancy introduced by anchor regression, and effectively process point cloud encodings, anchor-free object detection has been exploited in many works [33][45][15][52][19][11][48]. Anchor-free object detection can be classified into two broad categories, namely center-based and key point-based approaches.

The main architecture of our proposed Range-Aware Attention Network (RAANet) is presented in Fig. 2. We incorporate ideas from CenterNet [47][44] to build an anchor-free detector, and also introduce two novel modules: the Range-Aware Attention Convolutional Layer (RAAConv) and the Auxiliary Density Level Estimation Module (ADLE).

RAANet takes the 3D LiDAR points as input and generates a set of 3D oriented bounding boxes as the output. Inspired by VoxelNet [50], we implement a feature extractor with 3D sparse convolutional network on the voxelized point clouds. This resulting features are reshaped and formed into BEV feature maps, which can be regarded as a multi-channel 2D image. The Region Proposal Network (RPN) takes this BEV feature maps as input and employs multiple down-sample and up-sample blocks to generate a high-dimensional feature map. In addition to the detection heads in main task, we propose an auxiliary task for point density level estimation to achieve better detection performance. Meanwhile, the RAAConv layers are utilized in all convolutional modules. The details of the RAAConv layer are shown in the bottom half of Fig. 2. Center probability heatmap is generated from the detector head, and maximums in local neighborhoods are treated as the centers of ground-truth bounding boxes (bbox). The regression of bbox attributes, including offsets, dimensions and orientations, are computed from the other heads in the main detection task. Parallel to the main task, the proposed ADLE, as an auxiliary task, estimates the point density level for each bounding box. In general, ADLE classifies each bounding box into different density levels, based on the number of points in each bbox. ADLE is a portable block for the whole network, i.e. it is employed only during training and is not used during inference for higher computational efficiency.

For the training phase, the loss components, namely the heatmap classification loss, bbox regression loss and density-level auxiliary loss are denoted as $\mathit{l}_{hm}$, $\mathit{l}_{box}$ and $\mathit{l}_{aux}$, respectively. The total loss function of RAANet is formulated as

where $\lambda_{box}$ and $\lambda_{aux}$ are scalar weights for balancing the multi-task loss. $l_{hm}$ and $l_{aux}$ are designed as penalty-reduced focal losses [21][47], and $l_{box}$ is designed as the smooth L1 loss.

We propose the Range-Aware Attentional Convolutional Layer (RAAConv), which, by leveraging the specially designed attention blocks, becomes sensitive to range and position information. This is the major difference between the proposed RAAConv and a traditional convolutional layer. RAAConv is employed in the aforementioned RPN and detection heads. As shown in Fig. 2, RAAConv first utilizes two sets of convolutional kernels to extract an intermediate feature map for each branch. Then, the position encodings are generated and embedded into intermediate feature maps. Two range-aware heatmaps are calculated by a series of convolution and pooling operations using the intermediate feature maps and range encodings. More specifically, given an input feature map $I\in\mathbb{R}^{H_{in}\times W_{in}\times C_{in}}$, two separate intermediate feature maps, $\mathbf{F_{a}},\mathbf{F_{b}}\in\mathbb{R}^{H_{out}\times W_{out}\times\frac{C_{out}}{2}}$, are generated by

where $\zeta(\cdot)$ represents the convolution operation and $\mathbf{w_{F}^{i}}$ denotes the corresponding convolution kernel.

Meanwhile, the position encoding map $\mathbf{\xi}$ $\in$ $\mathbb{R}^{H_{out}\times W_{out}\times 2}$ and range encoding map $\mathbf{\rho}$ $\in\mathbb{R}^{H_{out}\times W_{out}\times 1}$ are generated from the intermediate feature map $\mathbf{F_{a}}$. $\mathbf{\xi}$ and $\mathbf{\rho}$ are calculated from shape information and do not depend on the specific values inside the tensor $F_{a}$. There are two components of $\mathbf{\xi}$: row encoding $\mathit{r}$ and column encoding $\mathit{c}$. $\mathbf{\xi}$ and $\mathbf{\rho}$ are generated as follows:

Values of $\mathit{r,c}$ are bounded in $[0,1]$. Values of $\mathit{\rho}$ are bounded in $[-1,1]$.

Since the proposed network is designed by using two different convolution kernels, to extract dense and sparse features separately, we append $\mathbf{\xi}$ to intermediate feature map $\mathbf{F_{a}}$ channel-wise, and reversely append $1-\mathbf{\xi}$ to $\mathbf{F_{b}}$. The generated features are denoted by $\mathbf{F_{a}^{\prime}}$ and $\mathbf{F_{b}^{\prime}}$. Maximum pooling, mean pooling and $1\times 1$ conv2D are applied on $\mathbf{F_{a}^{\prime}}$ and $\mathbf{F_{b}^{\prime}}$ separately to obtain the spatial embeddings $\mathbf{F_{a}^{\xi}}$ and $\mathbf{F_{b}^{\xi}}$. Then, similar to positional appending, the range encoding map $\mathbf{\rho}$ is appended to $\mathbf{F_{a}^{\xi}}$, and $-\mathbf{\rho}$ is appended to $\mathbf{F_{b}^{\xi}}$. The appended features are processed by a $3\times 3$ conv2D layer followed by sigmoid activation to obtain the range-aware attention heatmaps $\mathbf{f_{a}}$ and $\mathbf{f_{b}}$. The heatmaps $\mathbf{f_{a}}$ and $\mathbf{f_{b}}$ are then multiplied by learnable scalars $\gamma_{a}$ and $\gamma_{b}$, respectively. $\gamma_{a}$ and $\gamma_{b}$ are initialized as $1.0$ and their values are gradually learned during training [46]. The output feature maps $\mathbf{F_{a}^{\prime\prime}}$ and $\mathbf{F_{b}^{\prime\prime}}$ are calculated from $\mathbf{F_{a}^{\prime}},\mathbf{f_{a}},\gamma_{a}$, and $\mathbf{F_{b}^{\prime}},\mathbf{f_{b}},\gamma_{b}$ using a residual connection, which is defined as:

where $\otimes$ is the operation performing channel-wise stacking and element-wise multiplication. The final output for RAAConv is the channel-wise concatenation of $\mathbf{F_{a}^{\prime\prime}}$ and $\mathbf{F_{b}^{\prime\prime}}$, which has a size of $H_{out}\times W_{out}\times C_{out}$.

It is worth noting that the proposed RAAConv can be readily plugged in any convolutional network for LiDAR BEV detection.

The Auxiliary Density Level Estimation Module (ADLE) is an additional classification head that is parallel to the main task heads. The ADLE module indicates the point density level for each bbox. In general, we design the ADLE module to estimate the number of LiDAR points inside a detected bbox. We have empirically found out that this estimation is straightforward yet beneficial for the detector. Moreover, we divide the number of points into three density bins and let the ADLE perform a classification task, instead of a value regression task, which helps training to converge better. More specifically, the three bins are labeled as sparse (label $1$), adequate (label $2$) and dense (label $3$). We have statistically analyzed the distribution of the number of points and employed two thresholds $T_{0}^{i},T_{1}^{i}$ to determine the density level for each object class $i$. The three density levels are defined as:

where $n$ denotes the number of points, and $D_{i}$ represents the density level for the $i$-th class. As shown in Fig. 3, the unoccluded instances near the ego-vehicle have high point density levels and faraway instances have low density levels. However, when an instance is occluded, the density level will be lower than the unoccluded case at the same distance. Since we incorporate position and range information into RAAConv, ADLE is able to further help RAANet extract occlusion information by supervising the point density levels.

Anisotropic Gaussian Mask represents the centerness probability for each position in an annotated bbox. The centerness heatmap is generated as the ground truth for the classification head. In this work, we propose an Anisotropic Gaussian Mask, shown in Fig. 3, to generate a 2D Gaussian distribution, to be used for the centerness probabilities of a given oriented bbox. More specifically, the Gaussian distribution $\textit{N}(\mu,\Sigma)$ is designed in an anisotropic way, i.e. the $\sigma$ values for the 2 dimensions, which are the diagonals of $\Sigma$, and denoted as $\sigma_{0},\sigma_{1}$, are different, and their values are determined from the length and width of the bbox, respectively. The procedure for generating the center heatmap, denoted by $\mathbf{\mathfrak{M}}$, can be formulated as:

where,

and $(C_{H},C_{W})$ and $(H,W)$ denote the center location and dimensions of a bbox, respectively. $s$ indicates the ground truth class for the bbox. $\mathbbm{1}(\cdot)$ is the indicator function and $\mathbf{\mathfrak{B}}$ denotes the set of locations that are inside the bbox. $d_{cls}$ is a decay factor determining the sharpness of generated heatmap. The value of $d_{cls}$ depends on the class of a given bbox.

Following Eq. (1), the center heatmap loss $\mathit{L}_{hm}$ is defined as:

where $h$ and $\hat{h}$ are the ground-truth and predicted heatmap values, respectively. $\epsilon$ is a small value for numerical stability. $\alpha$ and $\beta$ are the parameters for controlling the penalty-reduced focal loss. For all the experiments, we follow the setting in [32], i.e. $\epsilon=1e-3$, $\alpha=2$ and $\beta=4$.

The bounding box head is responsible for the location, dimension and orientation regression, which are represented by the following parameters: $\mathbf{\Gamma}=[\delta_{x},\delta_{y},\delta_{z},l,w,h,\theta]$, where $\delta_{x},\delta_{y},\delta_{z}$ are the box center offsets with respect to the corresponding voxel centers; and $l,w,h$ and $\theta$ denote the length, width, height and yaw of a bounding box. We apply smoothed L1 (SL1) losses to regress $sin(\theta)$ and $cos(\theta)$ for the $\theta$ attribute, and directly regress actual values for other attributes:

where, $\hat{\cdot}$ denotes predicted values corresponding to original symbols. The total loss for box regression is:

where $N$ is the number of box samples and $\mathbbm{1}_{i}$ equals 1 if the ground truth heatmap value of the $i$th feature map pixel is larger than a threshold $\tau$, which is set to 0.2 in the experiments.

We perform the evaluations on the nuScenes [3] and KITTI [12] datasets, which are publicly available, large-scale and challenging datasets for autonomous driving. The full nuScenes dataset includes approximately 390K LiDAR sweeps and 1.4M object bounding boxes in 40K key frames. The dataset contains 1000 scenes in total, of which 700 are separated for training, 150 for validation and 150 for testing. There are a total of 10 classes: $\{$car, truck, bus, trailer, construction vehicle, traffic cone, barrier, motorcycle, bicycle, pedestrian$\}$. The scenes of 20 sec. length are manually selected to cover a diverse and interesting set of driving maneuvers, traffic situations and unexpected behaviors. The dataset was captured with a 32-channel spinning LiDAR sensor with 360 degree horizontal and -30 to +10 degree vertical FOV. The sensor captures points, which are within 70 meters with $\pm 2$ cm accuracy and returns up to 1.39M points per second.

We also use the KITTI dataset to benchmark performance. The 3D object detection dataset consists of 7481 training and 7518 testing samples for the point clouds, including a total of 80256 labeled objects. Following the experimental setting used in [18], we divide the official training samples into 3712 training samples and 3769 validation samples. Meanwhile, considering the test submission requirement, we create 784 samples as a minival validation set, and use the remaining 6697 samples to train. Since the KITTI benchmark only provides the ground truth annotations of objects that are visible in camera images, we followed the standard literature practice on KITTI of only using LiDAR points that project into the image, and trained one network for cars to benchmark our model performance.

RAANet is implemented in PyTorch framework [24] with an anchor-free object detection architecture. The weights $\lambda_{box}$ and $\lambda_{aux}$ in loss function (Eq. 1) are set to $0.25$ and $0.2$, respectively. The decay factor $d$ in Sec. III-D is set to $3$ for large size classes (car, truck, bus, trailer, construction_vehicle), to $6$ for small object classes (traffic_cone, barrier, motorcycle, bicycle, pedestrian). As for the parameters in Sec. III-E, we set $\epsilon$, $\alpha$, $\beta$ and $\tau$ to $0.001$, $2.0$, $4.0$ and $0.2$, respectively.

For nuScenes, all classes are trained end-to-end. The resolution for voxelization is set to $[0.075\text{m},0.075\text{m},\text{0.2m}]$ and the voxelization region is $[-54\text{m},54\text{m}]\times[-54\text{m},54\text{m}]\times[-5\text{m},3\text{m}]$. The model is trained by 20 epochs. Initial learning rate is set to $1e-3$ and then is modified by one-cycle adjuster [31]. Adam optimizer [16] with decay parameters $0.9,0.99$ is applied. The models are trained by 8 Nvidia V100 GPUs, and the batch size is set to 4 for each GPU. More details are provided in the configuration files of our Github repository [1].

For KITTI experiments, the resolution for voxelization is set to $[0.16\text{m},0.16\text{m},\text{0.5m}]$ and the voxelization region is $[0\text{m},69.12\text{m}]\times[-39.69\text{m},39.68\text{m}]\times[-3.5\text{m},1.5\text{m}]$. The model is trained by 80 epochs. Initial learning rate is set to $2e-4$ and is decayed by 0.8 for every 15 epochs. Adam optimizer with decay parameters $0.9,0.99$ is applied.

Two versions of the RAANet are designed in this work: (i) the full version RAANet applies RAAConv layers throughout all modules; (ii) RAANet-lite version applies RAAConv layers only at the detection heads. Both designs outperform SOTA baselines. Compared to the RAANet, RAANet-lite has faster inference speed, with a small drop in performance in terms of the mAP and NDS metric.

RAANet is trained end-to-end from scratch. Several data augmentation strategies are implemented in the training phase. They are random flipping along the $x$-axis, random scaling for the point clouds, and global rotation around the $z$-axis with a random angle sampled from $[-{\pi}/4,\pi/4]$.

During inference, we reuse the checkpoint with the best mean average precision (mAP) value and evaluate on the official lidar-only nuScenes and KITTI evaluation metrics. The detection score threshold is set to $0.1$ and IoU threshold is set to $0.2$ for non-maximal suppression.

The inference speed is tested on the nuScenes. The reason is that the nuScenes data is much closer to the real autonomous driving scenarios, whereas KITTI dataset only stores the front views of the scenes. The latency for the full version of RAANet and RAANet-lite are measured as 62 ms and 45 ms per frame, respectively. Both versions satisfy the real-time 3D detection criteria.

In this section, we provide the results of the ablation studies for different components and critical parameters of our approach, more specifically the attention module, decay factor $d$ (for the anisotropic Gaussian mask) and $\lambda_{aux}$ (weight for the density level estimation loss). The models are evaluated on the nuScenes validation set, since it has much more samples than the KITTI. For computational efficiency, all the experiments in this section are performed on the ‘Car’ and ‘Pedestrian’ classes to represent large and small objects, respectively. The baseline is the CenterPoint [44] with Isotropic Gaussian Mask and traditional convolutions.

The structures of the RAAConv layers employed in the ablation studies are shown in Fig. 4. In order to evaluate the effectiveness of our proposed RAAConv layer, we test each sub-component inside the RAAConv. Fig. 4 shows three main variants of RAAConv: (a) Dual branch module built with the original convolution, where each branch has half the number of output channels; (b) Single branch module built with range-aware attention, yet without the learnable scalar $\gamma$; (c) Dual branch module built with all sub-components. The reverse encoding indicates different positional encodings on the two attention branches, in order to extract dense and sparse features separately. The last sub-figure (Fig. 4(d)) shows the structure of the RAA block that is used for generating attention heatmaps.

Anisotropic Gaussian Mask. We evaluate the contribution of the Anisotropic Gaussian Mask by varying the decay factor $d$. The ablation study results presented in Fig. 5(a) show that, with anisotropic Gaussian mask, the performance improvement (in terms of AP) for the ‘Car’ class is more compared to the ‘Pedestrian’ class. One of the probable reasons is that the annotated bboxes for the pedestrian class have similar widths and lengths. This leads the anisotropic Gaussian mask to fall back to the original isotropic Gaussian mask. Additionally, we set $d=3$ for large object and $d=6$ for the small objects in our main experiments. In general, IoU-based evaluation is more sensitive on small objects than large objects, and sharper Gaussian masks help achieve higher accuracy values for small objects.

Auxiliary Density Level Estimation Module. The AP results obtained with various weights of the loss for the auxiliary component ($\lambda_{aux}$), are shown in Fig. 5(b). The loss weight values are chosen between $0$ and $1$. Based on these results, We set $\lambda_{aux}=0.2$ for the main experiments.

Range-Aware Attentional Convolution. Table IV shows the performance of the baseline as well as several types of the Range-Aware Attention Network. For RAAConv, there are three sub-components: dual branch structure, range-aware attention heatmap and learnable scalar $\gamma$ (Fig. 2). Each of these components contributes to the improvement of detection performance. We observe that the combination of dual-branch structure and range-aware attention provides 0.9% gain on the small object. With the learnable scalar $\gamma$, the dual-branch attentions can be adaptive to the variation of the training data resulting in further improvement.

The proposed RAANet is further evaluated by a different backbone from PointPillar [18]. More specifically, we replace the original 3D voxel-based backbone with a 2D pillar-based backbone. The pillar size is set to $(0.2,0.2)$. The perception range is set to $[-51.2,51.2]\times[-51.2,51.2]\times[-5,3]$. The baseline is trained with the CenterPoint-PointPillar version. As shown in Table V, our proposed RAANet still provides a competitive performance with each component contributing to the performance improvement.

Pairs of example outputs, obtained by the baseline and the proposed RAANet are provided in Fig. 6 for qualitative comparison. The sample data is selected from the nuScenes validation dataset. In a pair, images on the left are the results obtained from the baseline, and the images on the right are the results from the proposed RAANet. It can be observed that RAANet increases true positives and suppresses false positives at far or occluded regions.

Fig. 7 shows more visualization examples on all-class detection of the RAANet as well as the attention heatmaps generated from RAAConv. For each sub-figure, the image on the left is the all-class detection result obtained from our proposed RAANet, and the two images on the right are the attention maps from both convolutional branches in the RAAConv layer. It can be observed that both attention modules represent the positional heatmaps based on the BEV. In general, the attention heatmap from branch ‘a’, highlights the regions with dense point clouds or the ego-vehicle vicinity. The attention heatmap from branch ‘b’, on the other hand, pays attention to the regions with sparse point clouds or are farther from the ego vehicle. The combination of both branches effectively extracts powerful BEV features and generates superior 3D object detection results.

Fig. 8 shows some hard examples that were misjudged by our proposed method. In Fig. 8(a), although our RAANet successfully generates the detection results for all ground truths, some of the detection headings are not regressed accurately enough due to the long distance and sparse points inside the objects. In Fig. 8(b), some false positives appear on severely occluded regions.