Minkowski Tracker: A Sparse Spatio-Temporal R-CNN
for Joint Object Detection and Tracking

Most modern 3D object detectors for outdoor autonomous driving scenarios follow a similar structure. First, a point cloud encoder encodes 3D point clouds into a BEV feature. Second, tracking heads infer 3D object detections on the extracted BEV features.

First, for the point cloud encoder, VoxelNet (Zhou and Tuzel 2018) proposes to voxelize point clouds using a 3D voxel feature encoder based on PointNet (Qi et al. 2017) followed by a 3D CNN. PointPillar (Lang et al. 2019) proposes to encode 3D point clouds into a 2D pillar feature map followed by a 2D CNN. SECOND (Yan, Mao, and Li 2018) makes VoxelNet efficient by using sparse convolution (Graham 2014) instead of a dense 3D CNN. The point cloud encoder of Minkowski Tracker is a 4D sparse convolutional encoder. The 4D encoder improves object detection by reasoning over a long temporal horizon, being robust to temporal changes such as occlusion, truncation, and transformation. Additionally, the 4D encoder outputs a spatio-temporal BEV feature map to enable our tracking framework.

Second, the tracking heads infer 3D object detections on the BEV feature map. Point R-CNN (Shi, Wang, and Li 2019) and Voxel R-CNN (Deng et al. 2021) proposed two-stage region proposal networks similar to Faster R-CNN (Ren et al. 2015). CenterPoint (Yin, Zhou, and Krahenbuhl 2021) proposes a two-stage anchor-free keypoint detector network. 3DSSD (Yang et al. 2020) proposes a anchor-free single stage 3D detection network. Our proposed method is agnostic to the choice of object detection heads. We use CenterPoint as our baseline detector network for a fair comparison against previous works in MOT.

Region-based CNN (R-CNN) is a neural network architecture that infers scene knowledge based on region-of-interest (ROI) features. The initial goal of R-CNN was object detection (Girshick 2015; Ren et al. 2015), extracting ROI features based on region proposals to predict and refine object detection parameters. As the technique matured, the ROI features found a new use for predicting additional information about the detected objects. Mask R-CNN (He et al. 2017) proposes to solve instance segmentation as the second stage of object detection by making an object mask prediction from bilinear-interpolated ROI features. Mesh R-CNN (Gkioxari, Malik, and Johnson 2019) proposes reconstructing a corresponding 3D mesh from the detection’s ROI features. The proposed Minkowski Tracker similarly extends object detection networks to multi-object tracking using a spatio-temporal ROI feature. Our proposed TrackAlign samples collective ROI features of detections and tracks to predict the detected object’s match likelihood to current tracks.

There are numerous track association algorithms of detection-to-track trackers. The most simple yet powerful association algorithm is the global nearest neighbor (GNN) match (Rana et al. 2014), which associates a detection with its closest track on the Bird’s-Eye View (BEV) global Euclidean coordinate space. CenterPoint (Yin, Zhou, and Krahenbuhl 2021) slightly improves this algorithm by backprojecting a detection based on its predicted velocity into the timestep of the track and then performs GNN Matching there. Some use Kalman filters (Kalman 1960) to find a posterior estimate of a tracked object instance. A simple motion model predicts a tracked object’s next state which is then matched to a detection based on the Mahalanobis distance (Chiu et al. 2020) or IoU/GIoU (Rezatofighi et al. 2019; Weng et al. 2020b). Others use random finite sets (RFS) (Pang, Morris, and Radha 2021; Liu et al. 2022), factor graphs (Pöschmann, Pfeifer, and Protzel 2020; Liang and Meyer 2022), and graph neural networks (Zaech et al. 2022; Brasó and Leal-Taixé 2020; Weng et al. 2020c) for track association and management. Unlike previous works, our network directly predicts associations between the detections and tracks, learning an implicit motion model.

In addition to the association algorithms, (Chiu et al. 2020; Weng et al. 2020c; Baser et al. 2019) propose to fuse the aforementioned motion models with appearance features to improve track management quality. Additionally, (Wang et al. 2021; Wu et al. 2021) propose maintaining invisible tracks to prevent ID switches during occlusions. The contribution of Minkowski Tracker is complementary to these efforts, and it may benefit from multi-modality input and advanced track management algorithms.

Minkowski Tracker R-CNN takes 4D spatio-temporal sparse point cloud in Minkowski timespace as an input. $\mathcal{P}=\{(x,y,z,t,r)_{i}\}$ where $(x,y,z)$ is 3D location in Euclidean space, $t$ is time in a relative temporal index, and $r$ is reflectance measurement. The output of the encoder is a spatio-temporal Bird’s-eye-view (BEV) feature map $\mathcal{F}=\{\mathcal{F}^{1},\mathcal{F}^{2},\ldots,\mathcal{F}^{T}\}$.

In short, our encoder is a 4D-equivalent of SECOND (Yan, Mao, and Li 2018) with some adjustments on the temporal axis to maintain its temporal shape. First, the 4D voxelizer quantizes the points in 3D Euclidean space while retaining the temporal index. Let the 3D location of the point clouds range within $W$, $H$, $D$ along the $X$, $Y$, $Z$ axis respectively. The voxelizer groups points along the 3D Euclidean voxel grid of size $v_{W}$, $v_{H}$, $v_{D}$ for each $t$. Then, the voxel feature encoding (VFE) layer (Zhou and Tuzel 2018) extracts voxel features for each voxel group. Note that the voxelizer does not apply any grouping nor VFE along the temporal axis to retain the temporal dimension of the voxel tensor. The resulting 5D sparse tensor is of size $(C_{in}\times T\times\lfloor\frac{D}{v_{D}}\rfloor\times\lfloor\frac{H}{v_{H}}\rfloor\times\lfloor\frac{W}{v_{W}}\rfloor)$ where $C_{in}$ is the size of the input feature channel.

Then, the 4D sparse convolutional middle encoder network processes the voxelized 5D sparse tensor to generate dense spatio-temporal BEV images $\mathcal{F}$. Our middle encoder follows the same architecture as that of SECOND (Yan, Mao, and Li 2018) with the following modifications. First, our middle encoder uses 4D sparse convolution instead of 3D sparse convolution to enable spatio-temporal reasoning. Second, the network never strides along the temporal axis to preserve the temporal shape $T$. Given the encoder network spatial stride factor of st, the resulting dense 4D spatio-temporal BEV tensor is of size $\mathcal{F}=(C_{out}\times T\times\lfloor\lfloor\frac{H}{v_{H}}\rfloor\frac{1}{\text{st}}\rfloor\times\lfloor\lfloor\frac{W}{v_{W}}\rfloor\frac{1}{\text{st}}\rfloor)$ where $C_{out}$ is the output channel size and the $Z$ axis is squashed down by the network design. Finally, we split $\mathcal{F}$ along the temporal axis to get a 2D BEV feature images at each specific $t$, $\mathcal{F}=\{\mathcal{F}^{1},\mathcal{F}^{2},\ldots,\mathcal{F}^{T}\}$. As a result, our 4D sparse encoder fully propagates information across the entire 4D spatio-temporal space, explicitly reasoning on the 3D spatial domain and its temporal changes.

The first stage of Minkowski Tracker R-CNN, including the encoder above, is a 3D object detector. It takes a 2D BEV feature image $\mathcal{F}^{1}$ at the current frame $(t=1)$ as an input and outputs 3D bounding boxes $\mathcal{B}=\{(u,v,d,w,l,h,\alpha,vel,c,s_{\text{det}})\}$ of a center location $(u,v,d)$, 3D size $(w,l,h)$, rotation $\alpha$, velocity $vel$, semantic class $c$, and detection confidence score $s_{\text{det}}$ per box.

Minkowski Tracker R-CNN is agnostic to the choice of the detection algorithm. Without loss of generality, we chose CenterPoint (Yin, Zhou, and Krahenbuhl 2021) as our 3D detection model for fair comparison against most previous works. The performance of the tracker is heavily affected by that of the detector. Minkowski Tracker R-CNN is a modular network that can incorporate any research advance in the object detector to improve its general performance. For simplicity, we define any loss functions used to train the 3D object detector as $L_{\text{det}}$.

Although Minkowski Tracker R-CNN shares the identical detector head to CenterPoint (Yin, Zhou, and Krahenbuhl 2021), there are two crucial differences that set our detection results apart. First, the explicit temporal reasoning of the 4D encoder is robust to temporal changes such as occlusion, truncation, and transformation. Second, the multi-task learning of object detection and tracking enhances the quality of each other. In our experiment, we will quantitatively ablate the two to demonstrate how Minkowski Tracker R-CNN improves detection quality without any changes in the detector head.

The second stage of our network is a 3D online tracker. The input to our tracker is:

• •

  3D object bounding boxes from the detector $\mathcal{B}=\{\mathcal{B}_{1},\mathcal{B}_{2},\ldots,\mathcal{B}_{N}\}$ where $N$ is the number of detected objects.

• •

  Current tracks $\mathcal{T}=\{\mathcal{T}_{1},\mathcal{T}_{2},\ldots,\mathcal{T}_{M}\}$ where $M$ is the number of tracks and each track $\mathcal{T}_{i}=\{\mathcal{T}_{i}^{2},\mathcal{T}_{i}^{3},\ldots,\mathcal{T}_{i}^{T}\}$ is a set of 3D bounding boxes of the same instance moving across the previous frame $t=2$ to the last frame $t=T$. Note that part of the track could be missing due to occlusion or missing detections (e.g. $\mathcal{T}_{i}=\{\mathcal{T}_{i}^{2},\mathcal{T}_{i}^{4},\mathcal{T}_{i}^{5}\}$).

• •

  The spatio-temporal BEV image features $\mathcal{F}=\{\mathcal{F}^{1},\mathcal{F}^{2},\ldots,\mathcal{F}^{T}\}$ from the 4D encoder.

The tracker outputs a matrix $S=(N\times M)$ where each element is a match probability between the detections and tracks.

A common way to associate detections to tracks is based on a hand-designed filter that often uses simple linear motion models. The filters predict the next state of the track, and trackers measure the distance of the predicted states to the detections to build a cost matrix. However, simple motion models may not be sufficient to represent complicated real-world actions such as a curvy road as visualized in Figure 2. In contrast, a neural network can directly observe the car’s motion along the road and associate its movement. Therefore, we propose to implicitly model motions within the neural network by directly predicting the match probability of detections to current tracks.

Furthermore, object detection and tracking are related problems, predicting the object instance and location over time. Research in multi-task learning reveals that solving related problems jointly with a shared network enhances each other (Zhang and Yang 2021). However, most previous approaches to detection-to-track decouple detection and tracking as independent pipelines. To this end, we propose solving online tracking as the second stage of a detector by aligning spatio-temporal ROI features with our novel TrackAlign. Our experiment will quantitatively verify that matching tracks based on the learned detection-to-track matching score improves tracking quality.

We visualize TrackAlign in Figure 3. TrackAlign samples ROI features $\mathbb{F}=(N\times M)$ of a combination of $\mathcal{B}$ and $\mathcal{T}$. Without loss of generality, we describe how TrackAlign extracts the $i$th row and $j$th column of $\mathbb{F}$, $\mathbb{F}_{i\times j}$ from the $i$-th bounding box $\mathcal{B}_{i}$ and $j$-th track $\mathcal{T}_{j}$.

First, rotated ROIAlign (Ren et al. 2015) extracts the temporal ROI feature $\mathbb{F}_{i\times j}^{t}$ of each timestep $t$ from a 2D BEV feature $\mathcal{F}^{t}$ and a 2D rotated bounding box $\mathbb{B}^{t}$. At the time of inference, $\mathbb{B}^{1}=\texttt{BBox3Dto2D}(\mathcal{B}_{i})$ and $\mathbb{B}^{t}=\texttt{BBox3Dto2D}(\mathcal{T}_{j}^{t})\forall t\in[2,T]$. BBox3Dto2D is a function that converts 3D bounding box to a 2D BEV bounding box by removing its $Z$ axis elements, $d$ and $h$. In case tracks are missing at certain timestamps $t_{\text{miss}}$, we skip extracting the feature $\mathbb{F}_{i\times j}^{t_{\text{miss}}}=\emptyset$. Finally, we MaxPool all extracted ROI features across the temporal axis $\mathbb{F}_{i\times j}=\texttt{MaxPool}(\mathbb{F}_{i\times j}^{1},\mathbb{F}_{i\times j}^{2},\ldots,\mathbb{F}_{i\times j}^{T})$.

There are three benefits of TrackAlign worth mentioning: 1. The aggregated feature $\mathbb{F}$ is aware of the heading of an instance since we use rotated ROIAlign. 2. The bilinear interpolation of ROIAlign allows sub-pixel reasoning. From a practical standpoint, the size of a pixel of the BEV image used in our network is 60cm which is bigger than the average human shoulder-to-shoulder distance, 40cm. 3. TrackAlign is agnostic to the length of the temporal window $T$ and missing detections in tracks, owing to the symmetric function MaxPool.

Given the ROI features of a combination of detections and tracks $\mathbb{F}=(N\times M)$, the DetectionToTrackClassifier predicts a cost matrix of a match probability between detections and tracks $S=(N\times M)$. DetectionToTrackClassifier is simply a couple of layers of element-wise multilayer perceptrons (MLP) on $\mathbb{F}$ to predict match probability logits for logistic regression.

During training, we are given ground-truth tracks $\mathcal{T}_{\text{GT}}=\{\{\mathcal{T}^{t}_{GTi}\}\forall t\in[1,T]\}\forall i\in[1,K]$, where $K$ is the number of ground-truth tracks. Using $\mathcal{T}_{\text{GT}}$, we generate a set of positive and negative matching tracks with a combination of tracks at the current frame ($t=1$) and previous frames ($t\in[2,T]$).

$\mathcal{T}^{1}_{GTi}$ resembles detection $\mathcal{B}_{i}$ and $\{\mathcal{T}^{2}_{GTj},\mathcal{T}^{3}_{GTj},\ldots,\mathcal{T}^{T}_{GTj}\}$ resembles track $\mathcal{T}_{j}$ during evaluation. Then, we consider the matching tracks $(i=j)$ as positive and all else $(i\neq j)$ as negative matching tracks.

However, two issues arise with this approach. First, the positive-to-negative ratio $1:(K-1)$ is heavily biased. Second, most of the negative samples are too easy, e.g. matching a pedestrian on one end of the map to the car on the other end of the map. To mitigate this issue, we have two fixes. First, we extract hard negative samples using a heuristic filter $f$. We define heuristic filter $f(\mathcal{T}^{1}_{GTi},\{\mathcal{T}^{2}_{GTj},\mathcal{T}^{3}_{GTj},\ldots,\mathcal{T}^{T}_{GTj}\})$ as follows: 1. $c(\mathcal{T}_{GTi})=c(\mathcal{T}_{GTj})$ where $c$ is a semantic class of an object 2. $dist(\mathcal{T}^{1}_{GTi},\mathcal{T}^{1}_{GTj})<G$, where $dist$ is center-to-center 2D Euclidean BEV distance and $G$ is a hyperparameter gating distance. Any tracks that do not satisfy $f$ are filtered out to make the negative samples of the training data more meaningful. Second, we use binary focal loss (Lin et al. 2017), where $\hat{p}$ is the predicted match probability, $\alpha$ is a post-filter positive-to-negative ratio, and $\gamma=2$.

The final training loss of Minkowski Tracker R-CNN is as follows, where $\lambda_{\text{track}}$ is a loss balancing term.

Minkowski Tracker updates track $\mathcal{T}$ given the output of Minkowski Tracker R-CNN detection bounding boxes $\mathcal{B}$ and detection-to-track match probability $\mathcal{S}$. Prior to assigning detections to tracks, the tracker filters out obvious mismatches using the aforementioned heuristic filter $f(\mathcal{B}_{i},\mathcal{T}_{j})$.

Additionally, in order to enforce geometric relations to the matching scheme, we also define distance ratio $\mathcal{D}=(N\times M)$ as $\mathcal{D}_{i\times j}=dist(\mathcal{B}_{i},\mathcal{T}^{2}_{j}+{vel}_{j}^{2})/G$. Since any tracks with $dist(\cdot,\cdot)>G$ are filtered out, the distance ratio is within [0, 1], the smaller the better. Then, we Hungarian match detections $\mathcal{B}$ to tracks $\mathcal{T}$ to minimize the cost matrix $\lambda_{D}\mathcal{D}-(1-\lambda_{D})\mathcal{S}$, where $\lambda_{D}$ is a balancing hyperparameter. Any unmatched detections are initialized as a new track. Any tracks unmatched for longer than $T$ are considered as dead tracks.

Lastly, we estimate the confidence for each track. Track confidence scores determine the reliability of a track which is crucial for its applications. Thus, the tracking evaluation metric AMOTA (Weng et al. 2020a) takes track confidence into account. The most commonly defined track confidence score is an average of detection confidence scores $s_{\text{det}}$ in a track. This is a reasonable choice since a track is composed of detections. However, it lacks information on the confidence of associations among the detections. Therefore, we complement the missing information by defining track score as a track average of $(1-\lambda_{S})s_{\text{det}i}+\lambda_{S}\mathcal{S}_{i\times j}$, a linear combination of a detection score and its detection-to-track match probability score. In our experiment, we will quantitatively confirm that our score incorporating the detection-to-track match probability improves AMOTA.

The Nuscenes dataset (Caesar et al. 2020) is a large-scale autonomous driving dataset. It comprises 1000 driving scenes of 20 seconds length. Each scene is labeled with 3D bounding boxes of 7 object classes at 2Hz to facilitate multi-object tracking algorithms. The dataset provides multiple modalities of sensors, while we only use lidar point clouds as input to our system. Following the tracking challenge protocol, our temporal window at $t$ is roughly 0.5 seconds.

Following the Nuscenes tracking challenge, we use the following metrics to evaluate our tracker: AMOTA (Weng et al. 2020a), MOTA (Bernardin and Stiefelhagen 2008), RECALL, FP (False Positive), FN (False Negative), IDS (track ID switches), and FRAG (track FRAGmentation). MOTA has long been a standard evaluation metric for object tracking, measuring the common track association errors such as FP, FN, and IDS. AMOTA takes track confidence scores into account, being an integral of MOTA over recall scores. AMOTA is the primary metric of the Nuscenes tracking challenge.

CenterPoint (Yin, Zhou, and Krahenbuhl 2021) is our primary baseline 3D object detection and online tracking model. Centerpoint built a simple yet powerful 3D online tracking baseline by matching tracks with the closest object detection backprojecting the predicted velocity to the track. Our method is agnostic to the choice of a 3D object detector. We chose CenterPoint since most of the previous works in the Nuscenes tracking challenge take the detection result of CenterPoint as an input to the tracker.

We tested our method on the test set of the Nuscenes tracking challenge. As shown in Tabel 1, our method achieved the state-of-the-art AMOTA of all published submissions based only on lidar point clouds. Minkowski Tracker has notably high recall and lowest false negative among all methods. This result indicates that our method is superb at recovering ground-truth tracks with limited observations, owing to the 4D encoder’s temporal reasoning and multi-task training of detection and tracking. Minkowski Tracker also has the lowest track fragmentation and low ID switches demonstrating that the network’s implicit motion model successfully connects the tracks. Overall, our proposed framework quantitatively outperformed previous works in various metrics.

It is also worth noting that the performance of multi-object tracking is bound to that of object detection. Minkowski Tracker uses CenterPoint as a object detection head, with other CenterPoint-based trackers marked as ^∗ in Table 1. Our network is modular, and the latest development of 3D object detection can easily be incorporated by updating the detection head.

In Figure 4, we visualized a qualitative result of tracks on the validation set of the Nuscenes dataset, comparing CenterPoint with Minkowski Tracker. First, CenterPoint suffers from detection errors such as false positives and false negatives. In contrast, our method predicts persistent detections benefiting from the temporal reasoning of the 4D encoder. This result aligns with the quantitative result that our method achieved the best recall among all methods. Furthermore, CenterPoint suffers from ID switches, whereas our method robustly tracks the target, demonstrating that our implicit motion model successfully assigns detections to tracks. This result explains why our method achieved the lowest track fragmentation among all methods and less than half ID switches compared to CenterPoint.

We demonstrate in large-scale experiments that the overall performance gain of our method is due to four factors: 1. The 4D encoder improves detection by adding temporal reasoning. 2. Multi-task learning of object detection and online tracking enhances each other. 3. The network learns the implicit motion model to improve track assignment 4. The track confidence score incorporating the detection-to-track score is informative. In this section, we thoroughly analyze the performance gain of each component.

First, the 4D encoder of Minkowski Tracker R-CNN improves the object detection performance, subsequently improving the tracking performance. In Table 2 and 3, we compare the detection and tracking performance of CenterPoint with the original 3D encoder and our 4D encoder. Minkowski Tracker R-CNN outperforms CenterPoint in object detection owing to the extended horizon of temporal reasoning through the 4D sparse encoder. Subsequently, the tracking result of the 4D detector global nearest neighbor match outperforms CenterPoint.

Second, the multi-task learning of detection and tracking jointly enhances the performance of each other. In Table 2 and 3, we compare the detection and tracking performance of the 4D encoder CenterPoint, trained with and without the tracking loss ($\lambda_{\text{track}}=\{1,0\}$). A detector and tracker trained with the tracking loss (track) outperforms the bare detector. The improved detection from the 4D encoder and multi-task learning explains the quantitative result that our method has the highest recall and lowest false negative among all methods.

Third, the cost matrix incorporating the detection-to-track match probability score improves the quality of the track assignment. As shown in Table 3, the advanced track assignment with the learned score, denoted as cost, gives a persistent quantitative improvement in the quality of tracks. This result indicates that our network learns an implicit motion model that improves track assignment, which explains the lowest track fragmentation and low ID switches of the quantitative results.

Fourth, the detection-to-track score prediction by Minkowski Tracker R-CNN improves the quality of the track confidence. As shown in Table 3, the track confidence incorporating the matching score, denoted as conf, gives a persistent quantitative improvement in the AMOTA metric, which is a tracking metric that takes track confidence score into account.

In Figure 5, we measure the performance of our model with varying hyperparameters. First, the network predicted matching score $\mathcal{S}$ and distance ratio $\mathcal{D}$ are both effective measures for track assignment. Nevertheless, a linear combination of $\mathcal{D}$ and $\mathcal{S}$ gives a noticeable improvement in tracking performance, hinting that there are complimentary information between the two.

Second, the combination of the network predicted matching score $\mathcal{S}$ and the detection confidence $s_{\text{det}}$ results in more robust track confidence. The fact that a track is composed of detections and their temporal associations explains why track confidence should be composed of both detection confidence $s_{\text{det}}$ and association score $\mathcal{S}$.

In Figure 6, we analyze and compare the runtime of Minkowski Tracker with CenterPoint (Yin, Zhou, and Krahenbuhl 2021) and ImmortalTracker (Wang et al. 2021). The runtime is measured on a single-core AMD Ryzen 7 1800X CPU with a single TITAN RTX GPU. The primary concern of the runtime of our method is that we use a 4D encoder, which requires at least $\times T$ more computation compared to its 3D counterpart. Not surprisingly, the network runtime of our method is roughly $T=3$ times slower than CenterPoint.

However, Minkowski Tracker does not require complicated hand-designed filters to achieve high performance. CenterPoint relatively does not perform well in tracking since it uses a bare minimal motion model. ImmortalTracker, based on CenterPoint, requires a much higher runtime in the tracker to compute 3D Kalman Filter with linear motion model and 3D GIoU (Rezatofighi et al. 2019) for a reliable matching. In contrast, Minkowski Tracker relies on the network’s implicit motion model, taking a simple linear assignment of the detection-to-track score, which does not require much runtime in the tracker. As a result, Minkowski Tracker is $\times 6$ faster than ImmortalTracker while achieving the state-of-the-art tracking performance.