Box Re-Ranking: Unsupervised False Positive Suppression for
Domain Adaptive Pedestrian Detection

Before introducing our proposed unsupervised false positive suppression framework, three important prerequisites have to emphasize:

• •

  The bounding boxes generated by the pre-trained model are directly applied as ground-truth for box regression training, no matter positive or negative boxes, since the location error is much weaker than classification error in object detection tasks as proved by [1].

• •

  Reliable positive boxes are the seed prerequisite for our proposed framework to drive box re-ranking. Usually, those with extreme high positive confidence predicted from source domain pre-trained model are very likely to be real positive boxes. This condition is usually satisfied especially when the pre-trained model is trained on a very large-scale dataset collected from numerous source domains. Beyond this, deep transfer clustering is adopted for further label denoising and keeping the purity of seed positive boxes.

• •

  In most practical scenarios, false positive are occurred along with false negative. After our proposed box re-ranking, the confidence position among false positive and false negative are swapped, which indirectly suppresses false positive while mining back a corresponding part of false negative.

As mentioned above, we select seed positive boxes with very high confidence (greater than a given threshold $h$, 0.95 by default in this paper), while the remaining ones are directly treated as negative, which can be formulated as:

where $M_{s}$ is a given pre-trained model and $\mathcal{X}$ denotes target training dataset. Exploiting $M_{s}$ on $\mathcal{X}$, we can initially achieve bounding box set $\{(b,c)\}$ where $b$ denotes a detected box attached with corresponding confidence $c$. $\mathcal{Y}$ is the initialized label set including positive part $P$ and negative part $N$ divided from $\{(b,c)\}$ by a very high confidence threshold $h$.

Using the initial pseudo labels $(\mathcal{X},\mathcal{Y})$, the source domain pre-trained model $M_{s}$ is fine-tuned into $M_{t}$ as shown in Eqn.3. The objective loss function is a weighted sum of the confidence loss (conf) and the localization loss (loc). As metioned in 2.1, we directly use initially-generated boxes as ground-truth for box regression training. Usually, if the positive part is pure enough, the false positive will be well-suppressed. But it is quite easy to encounter the degeneration of true positive. To avoid this situation, more positive boxes should be mined back from the negative part elaborately.

Adopting the optimized model $M_{t}$, the pre-generated bounding boxes $\{b,c\}$ are re-predicted to update their confidence. To prevent the catastrophic forgetting of bounding box localization, the ground-truth of localization always keeps unchanged. In this way, we use the initial bounding box $\{(b,c)\}$ to match the re-predicted ones $\{(b,c)\}_{t}$. Let $d_{ij}=\{1,0\}$ be an indicator for matching the $i$-th initial bounding box to the $j$-th re-predicted box. $d_{ij}=1$ only if the IoU of this pair is the greatest one and the IoU should be greater than a given threshold (0.3 by default in this paper). So the confidence update process is:

After confidence update, the bounding box $\{b\}$ are re-ranked from higher to lower scores. The boxes in negative part with similar appearance to positive part will tend to achieve higher positive confidence $N_{top}$. This process is actually similar to PU-Learning [3]. We update the label $\mathcal{Y}$ by removing $N_{top}$ from $N$ and adding $N_{top}$ to $P$:

After updating $\mathcal{Y}$, Eqn.3-5 are repeated recursively until achieving the best performance.

It is too redundant to re-rank the boxes in an instance level, since no matter true positive and false positive usually appear frequently in a single target scene. To further discretize the granularity of box re-ranking, the boxes with similar semantic feature are packed into a cluster via deep transfer clustering:

where $f_{s}(b_{n})$ means we map the center of the $n$-th box to the feature map $f_{s}$ generated by $M_{s}(\cdot)$ and extract the corresponding feature of the box for clustering. $f_{s}(b_{n})\in\mathbb{R}^{d}$ and $d$ is the channel size of $f_{s}$. $C\in\mathbb{R}^{d\times k}$ is the cluster centroid matrix, $k$ is the cluster number and $y_{n}$ is the cluster assignment for box $b_{n}$. $N$ is the box number of the entire dataset.

As shown in Fig.3, we use YOLOv3 [16] as an example since we mainly use YOLOv3 in this paper. We concatenate the last second layer of three FPN [20] heads for feature extraction, and directly map the center of the boxes to the corresponding positions in the feature map. By default, we use $k$-means for boxes clustering. Without specific statement, we over-cluster the boxes into $100$ clusters in this paper. Note that the extracted features include semantic as well as scale and aspect ratio information, since they are used for classification and box regression. Scale and aspect ratio information is beneficial for clustering, since the boxes with similar semantic information usually share similar scale and aspect ratio.

After clustering, the boxes in the same cluster are re-ranked together as a whole via averaging the confidence in a cluster.

Concretely speaking, the boxes whose averaging confidence in the corresponding cluster is greater than $h$ are selected as seed positive boxes. During box re-ranking, the candidate positive boxes are added from the negative part to the positive part cluster by cluster.

An extra benefit of deep transfer clustering is label denoising. A reliable assumption is that the boxes with similar feature should share a similar confidence score consistently. Clustering makes the confidence of each box more robust via voting, avoiding accidently high confidence brought by domain noise and achieving more reliable seed positive boxes. The outlier boxes of each cluster are annotated as ignored labels during optimization.

In an unsupervised setting, no ground-truth labels are available in target data, which means no annotated validation data is available for cherry-picking during box re-ranking. Actually, this is also the main challenge of all unsupervised learning tasks. In this paper, we will introduce a self-supervised evaluation metric box number alignment to be an alternative of annotated validation data for false positive suppression tasks.

A main objective during false positive suppression is that the detection capacity of true positive should not be declined. Hence, we aims at keeping the number of detected boxes over a confidence threshold unchanged before and after box re-ranking. Actually, in practical application, a confidence threshold $o$ is usually pre-set to output the detected boxes (Note that $o$ is usually smaller than $h$. In practical, $o$ is usually set according to the False Positive Per Image on source domain validation set.). So the box number in the unlabelled validation set are countable and can be used as an evaluation metric for unsupervised false positive suppression. It can be formulated as:

where $M_{s}$ and $M_{t\_i}$ denote the pre-rained model in source domain and the $i_{th}$ optimized model in target domain during box re-ranking. $\mathcal{X}^{`}$ is an unlabelled validation set in target domain, and $o$ is a confidence threshold to count the output box number after NMS [24]. $C(\cdot)$ separately represent the function of box counting. In general, with our proposed box number alignment, the more false positive are suppressed, the more false negative are mined back.

The entire pipeline of our proposed unsupervised false positive suppression method is shown in Fig.2 and Algorithm.1. In one sentence, we carry out box re-ranking in the cluster level until meeting the condition of box number alignment to avoid detection degeneration.

We initialize $h=0.95$ and $o=0.4$ to drive seed positive boxes selection and box number alignment in a very conservative way. We carry out deep transfer clustering to the boxes with confidence greater than $0.1$. Detailedly, the boxes in the clusters with average confidence greater than 0.95 are set as seed positive boxes. Note that we only carry out deep transfer clustering once before box re-ranking. In each repetition for confidence re-prediction, we zero out the confidence of initial boxes generated by the pre-trained model, and match them to the re-predicted boxes generated by the optimized model with maximum IoU (at least greater than 0.3) for confidence updating. Without specific statement, in each repetition, the boxes in top-3 clusters with highest confidence in the negative part are mined back to positive part to drive the next repetition until reaching the condition of box number alighnment. The experiments on following cross-domain pedestrian detection datasets share the same implementation configurations. The entire pipeline can be referred to Algorithm.1.

Since cross-domain unsupervised false positive suppression is a new task we propose in this paper, no previous work is appropriate to discuss with our proposed box re-ranking framework quantitatively. To further verify our proposed method, we also compare our proposed method with other state-of-the-arts in two general domain adaptive object detection datasets, including the adaptation from KITTI to Cityscapes [7] and the adaptation from Sim10k [15] to Cityscapes. Following other work [4, 19, 18, 13, 12], these two adaptation datasets use Faster-RCNN [25] for car detection and take Average Precision (AP) as the main evaluation metric. Different from YOLOv3, we use ROI pooling feature of each box for deep transfer clustering in Faster-RCNN. As shown in Table.9 and 10, our proposed method outperforms other competing methods.