DetFlowTrack: 3D Multi-object Tracking based on Simultaneous Optimization of Object Detection and Scene Flow Estimation

Instead of independent optimization of object detection and scene flow estimation, we proposed a 3d multi-object tracking framework that simultaneously optimizes object detection and scene flow estimation. The proposed framework takes point cloud sequences as inputs, and outputs objects’ trajectories.

As shown in figure1, point-wise data association between adjacent frames are achieved by scene flow head to avoid dependence on motion model parameters. To simplify the network structure, we extract both object detection feature and scene flow feature in the feature extraction module. During the training stage, the loss of object detection and scene flow will be weighted summed. The total loss will be used to optimize the overall network. Considering that the detection results contain semantic information of points, we input the detection results into the scene flow module as a guide to avoid wrong data association of points belonging to different objects. To achieve multi-object tracking, boxes between frames are matched according to the result of object detection and scene flow in the box association module. The trajectory generation module takes box matches as input and updates trajectory.

To extract the point-wise detection feature and flow feature of the point cloud simultaneously, we utilize the backbone of PV-RCNN[12]. As shown in figure 2, voxel feature at multiple resolutions is extracted by voxel feature extraction. After that, the overhead feature map is obtained from the last layer of the feature voxel by vertical projection. We downsample the point clouds by farthest point sampling and key points are extracted. In the key point feature extraction stage, multiple resolutions voxel features are summarized into the feature embeddings of a small set of key points. Two parallel multi-layer perceptions take the key point feature embeddings as input and output detection feature and scene flow feature simultaneously.

For accurate detection results, we apply a two-stage detection head as PV-RCNN[12]. In the first stage, 3D box proposals are generated from the overhead feature map. In the proposal refine stage, the 3D box proposals are refined by detection features of points inside. For 3D box proposal $i$, the outputs of detection head include box result $\emph{b}_{i}=(px_{i},py_{i},pz_{i},w_{i},l_{i},h_{i},\theta_{i},c_{i})$ and score result $\emph{s}_{i}$.

The scene flow head contains the cost layer and scene flow calculation layer. The cost layer realizes feature aggregation between two point clouds, and outputs cost. The scene flow calculation layer takes cost volume as input, outputs scene flow result for each point in frame t-1.

Box association realizes inter-frame motion estimation of objects with the inter-frame motion estimation of points from scene flow results. Firstly, the inter-frame transformation of trajectory each box is predicted by its internal points’ scene flow results. And then, the predicitive trajectory box is obtained by transforming the previous frame trajectory box with predicted transformation. The matching cost matrix between the predictive trajectory boxes and the detection boxes is calculated by the boxes’s intersection of union. Finally, the Hungarian algorithm[14] is applied with the matching cost matrix and outputs match between the predictive trajectory boxes and the detection boxes.

The key to the box association is to achieve inter-frame boxes’ motion estimation according to inter-frame points’ motion results from scene flow. The process of box association is the reverse of the the process of scene flow groundtruth label calculation. Corresponding to the scene flow label calculation, we solve the boxes’ inter-frame transformation based on point cloud registration. Specifically, for each trajectory box $b_{i}^{t-1}$, its internal points are extracted and form in-box point cloud $c_{i}^{t-1}$. For each point in $c_{i}^{t-1}$, translate it with its scene flow and form the predictive in-box point cloud $\tilde{c}_{i}^{t-1}$. Then, the transformation $T_{i}^{t-1,t}$ between $c_{i}^{t-1}$ and $\tilde{c}_{i}^{t-1}$ is calculated based on svd method. The transformation $T_{i}^{t-1,t}$ is applied to trajectory box $b_{i}^{t-1}$ and obtain the predictive trajectory box $\tilde{b}_{i}^{t-1}$.

The matching result from box association module is input into the trajectory generation module to update the trajectory. The matching result is represented by three lists: match list $m^{t-1,t}$, detection box unmatch list $um^{t-1,t}_{(d)}$ and trajectory box unmatch list $um2^{t-1,t}_{(t)}$. The match list $m^{t-1,t}$ consists of matching pairs of detection boxes and trajectory boxes. The unmatch list $um^{t-1,t}_{(d)}$ and $um^{t-1,t}_{(t)}$ consists of unmatch boxes of detection boxes and presictive trajectory boxes separately. In order to cope with occlusion and error detection, track boxes are added and deleted according to the times of matching and losing.

The loss function consists of two parts: detection loss and scene flow loss as shown in equation 8.

In PV-RCNN[12], the detection loss can be split into the region proposal loss $L_{rpn}$, the keypoint segmentation loss $L_{seg}$ and the proposal refinement loss $L_{rcnn}$. In our work, we fix the parameters of RPN. As a result, our detection loss is calculated as equation 9.

For scene flow loss calculation, we calculate in-box mask $M^{(\mathrm{inbox})}_{i}$ for each point according to the MOT ground truth label. Scene flow loss is calculated by root mean square error (RMSE) as equation 10. Only points belonging to objects are considered.

where $d_{i}$ and $\hat{d}_{i}$ are scene flow estiamtion result and scene flow label for point i. $N_{(\mathrm{inbox})}$ is the number of points inside object boxes.

We train and evaluate our DetFlowTrack on KITTI MOT dataset[15]. KITTI MOT dataset consists of 21 training sequences and 29 testing sequences. Each sequence provides a sequence of point clouds collected by lidar and a sequence of images collected by color monocular cameras. We split the training sequences into two parts: Seq-0000 to Seq-0015 for training and Seq-0016 to Seq-0020 for validation.

To improve sample diversity and avoid overfitting, we intorduced a data augmentation approach for both object detection and scene flow estimation. First, we generate a object database with the KITTI Object dataset[15] by intercepting the labels and the in-box point cloud of all the groundtruth targets. Then during training, several groundtruth objects are randomly selected from the object database and are put on to the same random location of two adjacent frames’ scene. Using this approach, the number of objects and the diversity of the objects’ location is greatly increased. After that, global flip, rotation with random angle and scaling with random ratio. To increase the diversity of objects’ motion between frames, random traslation is applied to each objects independently.

We adopt HOTA[16] (Higher Order Tracking Accuracy) as the evaluation metric. The HOTA metric can split into DetA (Detection Accuracy Acore), AssA (Association Accuracy Score) and LocA (Localization Accuracy Score).

We fix the parameters of RPN in PV-RCNN backbone and train detection network and scene flow estimation network at the same time during training. We use an Adam optimizer with one cycle learning rate[17]. The learning rate changes a cosine function with 0.01 upper bound.

We train on an Intel 3.40GHz Gold 6128 CPU and 4 GeForce GTX 1080Ti GPU with batch size=4. Our DetFlowTrack is implemented on pytorch with CUDA 10.0.

An ablation study is designed to verify the validity of the proposed data augmentation module and the detection results’ guidance in scene flow module. The results are show in table I. In the catagory of both car and pedestrian, the proposed data augmentation module and the detection results’ guidance in scene flow module are effective to improve the accuracy of MOT.

We tested the trained model on KITTI MOT test dataset and submitted the test results to KITTI website for accuracy evaluation. The proposed algorithm is named FlowDetTrack and the results are compared with the latest online MOT algorithm. Table II shows the accuracy evaluation results. The HOTA of the proposed method in this paper far exceeds that of the state-of-the-art methods with point cloud input.

The robust in the presence of extrem motion with rotation is test on hard data set. The hard data set was obtained by adding random translations in the range of 0-0.5 m and random rotations around the Z-axis in the range of 0-20 degrees to each target on the KITTI MOT validation set. The results of car category and pedestrian category are show in table III. When compared with the state-of-the-art 3D MOT method 3DMOT, the proposed method can still maintain high HOTA on the hard data sets.

Figure 4 is the visualization of the MOT results. Trajectory boxes are projected to the image. The box with the same tracking id is painted with the same color.

Figure 5 is the visualization result of the feature aggregation in the cost layer of scene flow module. When only the distance in geometric space is considered, those points that are close to the aggregation center but do not belong to the object will be given a large aggregation weight as indicated by the red circle in subimage (1). With the proposed detection-guidance feature aggregation methods, the aggregation weights of those background points will be small enough to be ignored.

Figure 6 shows the comparative visualization results of the scene flow calculation methods when the target has rotational motion. We translate the points in frame $t-1$ with the calculated scene flow ground truth label and the result is shown as green points. With box-translation-based method as in PointTrackNet[11], the green points as indicated by the red circle do not properly match the points in frame t. While, with the proposed box-transformation-based method, the green points can match the points in frame t nicely. This proves that the proposed box-transformation-based scene flow label calculation method can obtain a more accurate true value of scene flow in the case of target rotation.