Fast Online and Relational Tracking

As a first step, the position of tracklets is predicted at the current frame using a motion model. Here, the kalman filter  [19] with a constant velocity motion model is used for prediction. The states of this model are:

in which $u$ and $v$ are the center, $a$ is the aspect ratio and $h$ is the height of a tracklet box and the next four parameters are their corresponding changing rate. The predicted location of tracklets in the current frame is shown as $\mathbf{\hat{T}_{t}}$.

The detections and the prediction of tracklets in the current frame are the inputs of the association part of the proposed algorithm. after association, most of detections are assigned to their corresponding tracklets. These detections are now represented by $\mathbf{\tilde{D}_{t}}$ and the modified tracklets are represented by $\mathbf{\tilde{T}_{t}}$. Consequently, there are a few unmatched detections and tracklets which are represented by $\mathbf{\bar{D}_{t}}$ and $\mathbf{\bar{T}_{t}}$.

For assigning detections to tracklets, an association metric based on the geometric features of bounding boxes are used to reduce the calculation cost. The proposed metric is the so-called normalized Intersection over Union(nIoU). To compute the $nIoU$, we first need to compute the normalized difference between position and size of boxes:

Now, the average of these four terms is subtracted from IoU to get normalized IoU:

in which $A_{D^{i}_{t}}$ and $A_{T^{j}_{t}}$ are the area of $i^{t}h$ detection box and predicted detection box of $j^{t}h$ tracklet in current frame. Also, the $I$ function calculates the intersection of these two bounding boxes. From detection algorithm, a confidence score for each detection is also reported that shows the probability of existence of a track. In Fig. 1 detections along with their corresponding score is visualized. Evidently, as the detection score decreases, the chance of track’s existence decreases. Therefore, our tracking algorithm considers this cue in matching detections with tracklets in two steps. Here, the tracking algorithm tries to match most high score detections and match low score detections with more caution. To this end, two thresholds on detection score are defined as $L_{h}$ and $L_{l}$. First, detections with a score higher than $L_{h}$ are matched with tracklets. In this step, the minimum accepted nIoU is set at a low value to match high score detections whether there is a minimum similarity. The detections with a score between these two thresholds enter the second matching step and are matched with the tracklets which are not matched in the first step. Because the chance of existence of false-positive detections is higher in this step, matching is done with more caution by increasing the minimum accepted similarity. At the end, detections whose score is below than $L_{l}$ are discarded as with high probability they are false positives. The tracklets which are not matched in the second step are reported as unmatched tracklets. The other variables of the two steps are aggregated and reported as the corresponding variable. The pseudo-code of this block is shown in Algorithm  2.

In constant velocity motion model, the assumption is that targets moves with the constant velocity and the camera motion is negligible. But this is a strong assumption which is not always valid. Sometimes the camera motion is so significant that causes challenges in matching. When the camera moves, detections move in the opposite direction and the amount of their movement is proportional to the camera movement. In such cases, the camera motion influences the matching of detections with tracklets and increases the risk of ID switch or mismatching. This effect is even sever for the tracks who are far from the camera and their bounding boxes are small.

In the proposed algorithm, we introduce a novel method for measuring camera motion and removing it’s effect on association. When the camera motion is high, all tracks have displacement relative to their predicted position. So, after matching, the difference between the position of matched pair of detection-tracklet is calculated in each direction. These differences are averaged and reported as the amount of camera motion. This value is subtracted from the position of all tracklet boxes and the matching step is repeated with the modified predicted position of tracklets. In Fig. 3 the effect of the proposed method for two consecutive frames with significant camera motion is shown. In the top figures, the camera motion removal is not applied. As can be seen, some tracklets are not matched or matched incorrectly. At the bottom, the camera motion removal is applied. As it can be seen, the identity of tracklets is preserved correctly.

One of the challenges in multiple object tracking algorithms is where a tracklet is covered partially or completely by other objects or tracklets and is not observed by the camera. In [45] the low score detections are used to find partially seen tracklets between them. As explained in the previous sections, the probability of false positives between these detections is high. So, in our approach for detecting occlusion, a method based on the relation of tracklets is developed. As the tracklet boxes are known, occlusion can be detected by considering their position relative to each other. To quantify the relation of them, the tracklet covered ratio is defined based on the similar concept in  [26] as:

This parameter shows the maximum ratio of a tracklet box that is covered by any of the other tracklets. The difference between this parameter and IoU is that the intersection of two bounding boxes is divided into the area of one bounding box, instead of the union of two bounding boxes. The high amount of this parameter can point to the occurrence of occlusion. To improve the precision of occlusion detection, another parameter is used alongside this as the tracklet’s confidence:

in which $Age_{i}$ is the total number of frames that tracklet is existed, $t_{so_{i}}$ or time since observed is the number of successive frames the tracklet is not matched with any detection and $A^{n}_{i}$ is the normalized area of tracklet box which is calculated by division of area of tracklet box to the average area of all tracklet boxes. This parameter measures the confidence of a reported tracklet by tracking algorithm. When a tracklet is tracked for more frames and is nearer to the camera and seen bigger, the probability of its existence is higher. On the other hand, when a tracklet is not visible for more frames, this confidence is decreased.

For the tracklets that do not match in the matching step, these two above parameters are calculated. If $CR_{i}$ is higher than $L_{cr}$ and $C_{i}$ is higher than $L_{c}$, that tracklet is reported as occluded in $\mathbf{O_{t}}$.

The unmatched tracklets which are not detected as occluded are possible candidates for removal. In some works [5, 39, 13, 45], they are kept for a certain number of frames (L frames). If L is a low number such as 3, the occluded tracks after reappearing will get a new identity giving rise to an ID Switch. If a high number such as 30 is chosen for L, the number of kept tracklets increases that causes a high computation cost. The lost tracklets are kept for occlusion cases to the identity of occluded tracks is preserved after reappearing. The target covered ratio ($CR$) which is used for occlusion detection in the proposed algorithm, is used for keeping lost tracklets, too. While the covered ratio of a tracklet is higher than $L_{cr}$ it is considered as covered. But when the covered ratio of that tracklet comes below this threshold for a few consecutive frames, i.e. 3, it is removed.

Detections that are not matched with any tracklets may point to existence of new tracklets. If their score is higher than $L_{n}$ which is close to 1, the probability that they indicate a real tracks is very high. So, for each detection a tracklet is created initialized with that detection. Other unmatched detections are kept and passed for association in the next frame.

For comparison of result, the widely used CLEAR MOT metrics  [4] which include Multiple Object Tracking Accuracy (MOTA), Multiple Object Tracking Precision (MOTP) are reported. In addition, the HOTA  [23] and IDF1  [31] are reported. The MOTA metric consider FP, FN and IDS effect equally. As the number of FP and FN are much higher than IDS, the focus of this metric is more on detection performance. But in HOTA and IDF1 more attention is paid to association performance by focusing on preservation of track’s identity. Also, other popular metrics are used including Mostly Tracked (MT), Mostly Lost (ML), the number of False Negatives (FN), False Positives (FP), ID-Switches (IDS), and track Fragmentation (FM).

The results of running the proposed algorithm on MOT17 and MOT20 datasets are presented in  1 and  2 respectively. The result of other state-of-the-art algorithms, especially the ones using interaction models are also available in these tables for comparison. All of these algorithms use private detections. The private detections that we used are the same as two other algorithms  [45, 8]. So, the results of our algorithm and these two algorithms are separated by a line at bottom of the tables. In MOT17, our algorithm achieves the best results on MOTA, IDF1, HOTA, and, MT among other algorithms. Also in MOT20, the results of our algorithm are comparable with the state-of-the-art algorithms.

In the proposed algorithm, a method for removing camera motion is presented. MOT17 dataset is composed of 7 training and 7 test sequences. In most sequences, the camera motion is negligible. So, the camera motion removal has no significant effect. among MOT17 training sequences, sequences 5 and 9 have significant camera motion in several frames. The results of the proposed algorithm with and without camera motion removal block are presented in the table 3. As it can be seen, the camera motion removal could notably improve MOTA, IDF1, and ID Switch metrics.

The occlusion detection method in the proposed algorithm calculates the covered ratio and confidence of tracklets that are not matched with any detections and report them as tracklet of that frame by checking some conditions on these parameters. So, the tracklets that are marked as occluded, do not have any corresponding detection. To investigate the effectiveness of this block, the detections in training sequences are matched with ground truths. In this way, all the detections match correctly with the corresponding tracks, and the number of false positives and ID switches reaches zero and the best possible result is achieved. In this way no track will be reported when it has no corresponding detection. The results in this situation with the result of the proposed algorithm on the training set are presented in the table  4. As it can be seen, the proposed algorithm could pass the best achievable MOTA by reporting some unmatched tracklets as occluded.