Non-iterative optimization of pseudo-labeling thresholds for training object detection models from multiple datasets

In Fig. 2, we illustrate the pseudo-labeling method for learning object detection from multiple datasets with different sets of object classes. Assume that the entire dataset $DS$ consists of $N$ datasets $DS_{1},DS_{2},\dots,DS_{N}$ that have different class sets $C_{1},C_{2},\dots,C_{N}$, respectively. The class sets of dataset $DS$ can be written as $C=C_{1}\cup C_{2}\cup\dots\cup C_{N}$, and a complementary label set of dataset $DS_{i}$ can be written as $\overline{C_{i}}=(C\setminus C_{i})$. First, we train an object detection model $M_{i}$ for a label set $C_{i}$ with a dataset $DS_{i}$ by supervised learning. We train models with $N$ datasets individually, so $N$ object detection models are generated. Second, we generate pseudo labels. To this end, models that cover $\overline{C_{i}}$ are used to get predictions of object locations in the dataset $DS_{i}$. If the raw predictions were used as pseudo labels, they would be too noisy, so we only adopt predictions with the confidence scores higher than a certain threshold. Finally, we train an object detection model $M$ with the original ground-truth labels and the generated pseudo labels.

There is a variant of the pseudo-labeling method where two thresholds, $\tau_{h}$ and $\tau_{l}$, are used to further reduce the noise of pseudo labels [11]. If a prediction has a confidence score higher than $\tau_{h}$, then it is used as a pseudo label, while if the confidence scores of all the classes in a region are below $\tau_{l}$, such an area is treated as pseudo background. Regions that do not satisfy either are ignored in the training of a student model.

The goal of optimizing the thresholds is to find a set of pseudo labels that brings us the best student model. However, it is time-consuming to measure the performance of a student model for each pseudo-labeled dataset generated with a different threshold. Therefore, we propose to use $F_{\beta}$-score of pseudo labels as a surrogate for the student performance. It is defined as the weighted harmonic mean of precision and recall:

If the $\beta$ value is less than 1, $F_{\beta}$ is a precision-weighted metric, while if the $\beta$ value is more than 1, $F_{\beta}$ is a recall-weighted metric. Importantly, because this essentially measures performance of the teacher models, we do not need to train a student model to calculate the $F_{\beta}$-score.

Ideally we would evaluate the $F_{\beta}$-score of the pseudo labels themselves, but this is impossible because there are no ground-truth labels in the dataset that we want to generate pseudo labels for. Therefore, we measure the $F_{\beta}$-score on a pseudo-label validation dataset. We can prepare this dataset either by splitting from a training dataset or separately under the condition that it has annotations of evaluation categories.

To find the optimal thresholds, we use the teacher models $M_{i}$ to generate predictions of object locations in the validation dataset and calculate the $F_{\beta}$-score. For the one-threshold variant of pseudo labeling, we adopt the threshold that brings the maximum $F_{1}$-score as $\tau$. For the two-threshold variant, on the other hand, we take as $\tau_{h}$ the threshold that maximizes the $F_{0.5}$-score, a typical precision-weighted metric, and as $\tau_{l}$ the threshold that maximizes the $F_{2}$-score, a typical recall-weighted metric.

The method discussed above determines thresholds by maximizing the $F_{\beta}$-score of pseudo labels. However, because training of a student model uses both the human-annotated ground-truth labels and the generated pseudo labels, it is preferable to take both of them into account to determine the thresholds.

Figure 3 illustrates the basic idea. For the object class $j$, let $x_{j}$ be the ratio $\frac{g_{j}}{t_{j}}$ of the number $g_{j}$ of labeled objects over the total number $t_{j}$ of labeled and unlabeled ones. We assume that the ground-truth labels are definitely correct, so we set precision = 1 and confidence score = 1. Then we can calculate the precision and recall of all the labels as

Here, $p_{j}(\tau)$ and $r_{j}(\tau)$ are precision and recall of pseudo labels only. By using these expressions, we can calculate the $F_{\beta}$-score of all labels and select thresholds $\tau$, or the pair $\tau_{h}$ and $\tau_{l}$, in the same way as explained in Section 2.2.

Under the current problem setting where no knowledge about unlabeled instances is available, $t_{j}$ is also unknown, because this includes the number of unlabeled objects. In such cases, a crude but simple way to estimate $x_{j}$ is to use the ratios of the number of images in the datasets with annotations of the object class $j$ and the number of images in the entire dataset.

We experiment with two semi-synthetic datasets, COCO-splitting and COCO-VOC, which emulate the problem setting as described in Section 1.

COCO-splitting We split Microsoft COCO Detection 2017 (COCO) [14], a general object detection dataset consisting of 80 object classes, into $N$ subsets ($N=2,5,10$) to make multiple datasets with different class sets, as follows: split 110,000 images from COCO training data into $N$ subsets to make $N$ datasets ($DS_{i}$); assign category ids $i$ (mod $N$) to $C_{i}$; and remove images in $DS_{i}$ that have no objects belonging to a category in $C_{i}$. In this setting, the $N$ splits are expected to have similar characteristics with each other. The remaining 5,000 images of the COCO training split are used as a pseudo-label validation dataset.

COCO-VOC This dataset combines COCO [14] and Pascal VOC [15] (COCO-VOC dataset), mimicking the situation where there is a domain gap between constituent datasets. The VOC dataset consists of 20 object categories, which is a subset of COCO’s 80 classes. We remove annotations of the overlapping classes from COCO, so that the two datasets have mutually exclusive sets of object classes. Similarly to COCO-splitting, we use 5,000 images from COCO’s training split and 500 images from VOC’s trainval split for validation. For testing, we use the same dataset as used in Ref. [11], which is taken from VOC’s test set and COCO’s validation set. The VOC portion of this dataset annotates not only 20 object categories but also an additional 60 categories that are only annotated in the COCO training dataset. This allows us to measure mAP over 80 classes on both of the datasets.

We use M2Det [4] as an object detection model. M2Det is a one-stage object detection model that applies a multi-level feature pyramid network (MLFPN) to a feature map generated from a backbone to perform detection by using multi-scale feature maps. The input image size is $320\times 320$ pixels and a VGG16 network pretrained on ImageNet is used as the backbone. For each experiment, the model is trained for 150 epochs by using NesterovAG with an initial learning rate of 0.01 and a momentum of 0.9.

In the experiment with COCO-splitting, we use the simpler version of the pseudo-labeling method with one threshold to reduce the computational cost. We also found from preliminary experiments that the gain from the second threshold was tiny for this dataset. On the other hand, we adopt the two-threshold variant with COCO-VOC, because preliminary experiments suggested that this setting benefit significantly from the second, lower threshold.

For the grid search algorithm, we need to predetermine the sets of candidate threshold values. When tuning the single threshold $\tau$, we take $[0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1]$ as the pool of the candidates. Note that taking the value $\tau=1$ is equivalent to “w/o PL”, which simply combines datasets without any additional pseudo labels. On the other hand, the thresholds $\tau_{h}$ and $\tau_{l}$ are selected from $[0.2,0.4,0.6,0.8,1.0]$, with the constraint $\tau_{h}\geq\tau_{l}$. This results in 15 pairs of the candidate threshold values for this algorithm.

In the $\rm{Fmax_{DS}}$ method, we need to estimate the ratio of the number of labeled objects to that of all objects for each object class. In the COCO splitting experiment, for simplicity, we use the true number of $x_{j}$, which is known from COCO’s original annotation. On the other hands, with COCO-VOC, we follow the simple procedure explained in Section 2.3.

Figure 4 shows the experimental results for the COCO-splitting datasets. The plot indicates that one of our proposed method, $\rm{Fmax_{DS}}$, performs better than or competitively to the grid search method, without any iterative search for the optimal $\tau$. $\rm{Fmax_{PL}}$ prefers smaller thresholds, which means that it uses noisier pseudo labels to train a student model. This slightly degrades the detection accuracy, as compared with grid search and $\rm{Fmax_{DS}}$, but it is still competitive (Table 1).

Interestingly, Fig. 4 implies that the optimal value of the threshold depends on a dataset. In other words, there is no universal value of the best threshold that leads to the optimal performance for any dataset. This observation corroborates the underlying premise of our research that we need to optimize the threshold to achieve the best results. Figure 4 also indicates that the value found by $\rm{Fmax_{DS}}$ is similar to the optimal value found by grid search.

Table 2 lists the experimental results of the COCO-VOC dataset. Without repeated training of student models, $\rm{Fmax_{DS}}$ achieves the best result with respect to mAP over the whole test set (“C+V”). This result suggests that our proposed method is robust to the domain gap between the constituent datasets.