MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries

For autonomous cars, it is critical to track surrounding objects while estimating their position, orientation, size, and velocity. Due to recent advances on 3D detection [14, 49, 41, 43, 28], modern 3D MOT follows tracking-by-detection paradigm. These methods detect objects in the current frame and then associate them with previous tracklets. Weng et al. [38] benchmark a simple yet effective association methods. They predict the location of previous tracklets through Kalman filtering, then associate current detections using 3D IoU. Beyond IoU, several works used L2 distance [43] and generalized 3D IoU[21] to associate 3D box with pure location cues. Many works use more advanced association by adding learned motion and appearance features [2, 8, 9] or using graph neural networks [45, 39, 5]. Several works study how to improve life cycle management [3, 21] by utilizing cues from detection scores. QD3DT current SOTA (State-of-The-Art) camera-based tracking algorithms learn an appearance matching feature through dense contrastive learning They use an LSTM-based motion model to learn motion features and predict current locations. Finally, it combines visual features, motion cues, and depth-ordering for the association. Though with strong RGB appearance cues, performance of camera-based 3D MOT [27, 2, 47, 8, 11, 32] has been lagged behind compared to LiDAR-based. On the nuScenes 3D MOT challenge’s public leaderboard, STOA camera-based methods achieve 21.7% AMOTA while STOA LiDAR-based methods reach 67.9% AMOTA. The problem of tracking through multiple distinct viewpoints also draws attention [30].

3D object detection have seen great advances in recent years. A stream of algorithms build upon 2D detection framework [42, 28, 48, 34]. To resolve the fundamental ambiguity of instance depth and scales, categorical canonical shapes[1, 19], geometric relation graphs[33] and pretrained monocular depth [18, 22] are used. Another stream of methods works with representations on 3D space or Birds-Eye-View. Pseudolidar [35, 44] use pre-trained monocular depth models to lift pixels to 3D point clouds, then perform 3D detection using a LiDAR-based detector. Lift-Splat-Shot [23] makes the lifting process fully differentiable and joint trains the lifting modules with downstream tasks. Later CaDDN[24] and BEVDet[12] used similar represents for 3D detection. DETR3D[36] adopt an inverse projecting process and build query-based multi-camera 3D detectors. Compared to working on perspective image planes directly, one major advantage of working in 3D space is the ease of adopting arbitrary camera rigs and fusing multiple sensor features. Currently, there is no clear advantages on performance[22]. More comparisons are still yet under-explored.

A dominant type of modern detection and tracking approach is to reduce the task of detection to pixel-wise regression and classifications[26, 25, 17, 13, 48, 31], then perform tracking by associating detection boxes. Recently, DETR[7] successfully used query-based set prediction to achieve state-of-the-art detection results. Later TrackFormer[20], MOTR and TransTrack[46, 29] extends this idea to online 2D MOT. Our work builds upon the framework of query-based tracking. We extend the framework to multi-camera 3D MOT with a motion model.

We adopt query-based tracking for our algorithms. Query-based tracking is extended from query-based detection[7], where detect queries, a fixed-size set of embedding, are used to represent 2D object candidates. Track query extends the concept of the detect query to multi-frames, i.e., representing a whole tracklet across frames [46, 20, 37]. Specifically, we initialize a set of newborn queries at the beginning of each frame, then queries update themselves frame-by-frame in an auto-regressive way. A decoder head predicts one object candidate from each track query in each frame, and boxes decoded in different frames from the same track query are directly associated. With proper query life cycle management, query-based tracking can perform joint detection and track in an online fashion.

There are three key ingredients in our query-based multi-camera 3D tracker. (1) A query-based object tracking loss assigns different regression targets for two different types of queries, newborn queries, and old queries. (2) A multi-camera sparse attention uses 3D reference points to sample image features for each query. (3) A motion model estimates object dynamics and updates the query’s reference point across frames. We illustrate the flow of our trackers in Figure 2.

We first explain the concept of label assignment in the context of query-based tracking. Our algorithms maintain a changing set of track queries across frames. At the current frame, we decode one object candidate from each query. Ideally, The decoded object candidates from the same query should represent the same object across frames, thus forming a whole tracklet. To train the query-based tracker, we need to assign one target ground truth object for each query in each frame, and the assigned ground-truth object acts as the regression target for the query. Specifically, label assignment is a mapping function between ground-truth objects and track queries. We typically pad the set of ground truth objects with $\varnothing$ (no object) to the number of predicted object candidates to ensure the mapping is a one-to-one mapping. Suppose we have $N$ decoded object candidates $\{\hat{y}_{1},\ldots\hat{y}_{N}\}$ in current frame, label assignment can be denoted as a mapping $\pi\in\{1,2\ldots,N\}\mapsto\{1,2\ldots,N\}$. Then the training loss can be expressed as a sum of paired box loss:

where $y_{\pi(i)}$ denotes the assigned target ground-truth object, and $\mathcal{L}_{\mathrm{box}}$ could be any bounding box loss. There are two types of queries for each frame, and they have different label assignment strategies. Newborn queries are a set of learned queries. They are input-agnostic and will be added to the set of queries at the beginning of each frame. Newborn quires are responsible for detecting newly appeared objects in the current frame. So we perform bipartite matching between object candidates from newborn queries with newly appeared ground truth objects as DETR[7]. Old queries are active queries from previous frames which successfully detected or tracked objects. Old queries are responsible for tracking previously appeared objects in the current frame. The assignment for old queries is fixed after the first time it successfully detected a ground truth object. It is assigned to track the same object if they are in the current frame; otherwise, $\varnothing$ (no object).

The 3D box loss $\mathcal{L}_{\mathrm{box}}$ in equation 1 is defined as:

We use $L_{1}$ loss for $L_{\mathrm{reg}}$, and $L_{\mathrm{cls}}$ is the focal loss [17], and the 3D object $y_{\pi(i)}$ is parameterized using the class label $c_{\pi(i)}$, and bounding box parameters $b_{\pi(i)}$, details for the parameterization is in equation 6.

Our transformer decoder head takes track queries and attends them with multi-camera image features, and the extracted query featurees would be used to decode object candidates. Our decoder has two types of attention modules: self-attention between queries and cross attention between queries and image features. For memory efficiency, we adopt a reference-point based attention from DETR3D[36] for cross attentions. For notation in this section, only 3D coordinates or their 2D projections are in bold, e.g., 3D coordinates of reference points, $\boldsymbol{c}_{i}$, estimated velocities $\boldsymbol{v}_{i}$.

To deal with disappearing objects in an online fashion, we need to remove inactive queries after each frame. We define the confidence score of each query as the classification score of their predicted box. We use two threshold parameters $\tau_{\mathrm{new}}$ and $\tau_{\mathrm{old}}$ for box scores and a time length, $T$ to control the life management.

During inference, for newborn queries in each frame, if the score is lower than $\tau_{\mathrm{new}}$, we remove it. For old queries, if their scores have been lower than $\tau_{\mathrm{old}}$ for successive $T$ frames, we remove it. We select $\tau_{\mathrm{new}}=0.4$, and $\tau_{\mathrm{old}}=0.35$ and $T=5$ for nuScenes dataset after several trails.

During training, we view queries matched to $\varnothing$ as inactive. For newborn queries in the current frame, if it is matched to $\varnothing$, we remove it. For old queries, we remove it if it has been matched to $\varnothing$ for successive $T$ times. Note that old queries that have been matched to $\varnothing$ but have not been removed continue to update themselves through the transformer decoder.

After filtering out outdated (dead) queries, we update track queries, both their features and 3D reference points. The purpose of updating the 3D reference point is to model object dynamics and compensate for ego-motion. There are two commonly used motion models in 3D tracking, Kalman Filter, e.g., [38, 21], which uses observed position across frames to estimate unknown velocity, and predicted velocity from detectors, e.g., CenterTrack [47, 43]. We use velocity predicted from queries, which updates through frames and can aggregate multi-frame features. We use a small FFN to predict ego frame velocity. The predicted velocity is supervised with ground truth. Denote the ego pose of current frame and next frame as $R_{t},R_{t+1}\in R^{3\times 3},T_{t},T_{t+1}\in R^{3}$. Denote the time gap between these two frames as $\Delta t$. We update the reference point $\boldsymbol{c}_{i}$ of the $i$-th query using the predicted box velocity $\boldsymbol{v}_{i}=(v_{i}^{x},v_{i}^{y},0)\in\mathbb{R}^{3}$ :

To implicitly model multi-frame appearance variations, we update the track query using features from previous frames. Following MOTR[46], we maintain a fixed-size first-in-first-out queue for each of the active queries, named memory bank. After each frame, we apply an attention module for each query and its memory bank. The track query acts as the query for the attention module, and the corresponding memory bank act as a set of keys and values.

We use nuScenes[6] dataset for all of our experiments. It consists of 1000 real-world sequences, 700 sequences for training, 150 for validation, and 150 for the test. Each sequence has roughly 40 annotated keyframes. Keyframes are synchronized frames for each sensor with a sampling rate of 2 FPS. Each frame includes images from six cameras with a full 360-degree field of view. It provides 3D tracking annotations for 7 Object categories.

Average multi-object tracking accuracy (AMOTA) and average multi-object tracking precision (AMOTP) are the major metrics for nuScenes 3D tracking benchmark. AMOTA and AMOTP are computed by integrating MOTA(multi-object tracking accuracy) and MOTP(multi-object tracking precision) values over all recalls:

where $\mathrm{FP}_{r}$, $\mathrm{FN}_{r}$ and $\mathrm{IDS}_{r}$ represents the number of false positives, false negatives, and identity switches computed at the corresponding recall $r$. $\mathrm{GT}$ is the number of ground truth bounding boxes. AMOTA can be formulated as:

where $d_{i,t}$ denotes the 2D birds-eye-view position error of matched track $i$ at time $t$, and $\mathrm{TP}_{r}$ indicates the number of matches computed at the corresponding recall $r$.

We also report tracking metrics from CLEAR[4] and Li et al.[15] such as MOTA, MOTP, IDS. The confidence threshold for these metrics is selected by independently picking the threshold with the highest MOTA for each category.

We compare our method with SOTA methods in Table 1. We outperform current SOTA methods for the camera-based tracker by a large margin. The gain in AMOTA from the current SOTA method QD3DT[11] is over 5.2 points on the validation set and 5.3 points on the test set. Our tracker operates in an end-to-end fashion, with no NMS and no association stages as in QD3DT[11].

We put the comparisons of two of our Kalman filter baselines in Table 2. We outperform the basic version of the Kalman filter. However, when compared with more tailored baselines from SimpleTrack[21], we only have slight gains on metrics like AMOTA, MOTA, MOTP.

The motion model provides one of the primary cues for 3D Multi-object Tracking. The motion model aims to describe the moving patterns of tracklets. To evaluate the motion models of different tracking algorithms, we develop two metrics, Average Tracking Velocity Error (ATVE) and Tracking Velocity Error (TVE), following the idea of AMOTP and MOTP. ATVE can be computed as:

where we traverse over all pairs of matched tracking predictions and ground truth and compute the $L_{2}$ error between the predicted velocity $v_{i}$ and the ground-truth velocity $v_{t}$. Average Tracking Velocity Error is computed by averaging over all recalls $r$, and $\mathrm{TP}_{r}$ represents the number of matches in corresponding recall $r$. Like MOTP, Tracking Velocity Error is the average velocity error computed at the recall with the highest MOTA. We evaluate the evaluation of motion models in Table 3. Compared to the previous state-of-the-art camera tracker QD3DT[11], our velocity is more accurate. Compared to Kalman filtering-based motion models, our algorithm achieves better Tracking velocity Error.

We study two factors in the ablation study. First, we study the effect of dropping our motion model, i.e., do not update the 3D reference points at the end of each frame. We show the ablation results in Table 4. Removing our motion model degrades the performance in all metrics.

Second, we study the effect of the number of training frames. Our methods track objects in an auto-regressive way, and no teacher-forcing is applied. During training, gradients computed in latter frames will still propagate to compute graphs in previous frames. In the ablation study, we perform all the experiments using ResNet-50 backbones. We report the performance of training with 3,4,5 frames in Table 5. Results showed increasing the number of training frames gradually improves the performance.

We provide visualizations of our tracking algorithms in both BEV and camera views for an 8 seconds clip in Figure 3. Near-filed objects on the left/right side of the car are usually truncated by several cameras, which is a substantial challenge for multi-camera 3D tracking. See, the gray and black cars are truncated in the Front-Left camera and Back-Left camera(3-rd/7-th and 4-th/8-th row), and our algorithm handles them correctly.