Delving into the Imbalance of Positive Proposals in Two-stage Object Detection

Shortage of Positive Proposals in the Beginning. In the early training stage, since RPN cannot make accurate foreground/background discrimination, the large number of low-quality RPN generated proposals are fed into the following R-CNN module as training samples. Thus, the proposals can hardly satisfy the “positive proposal definition” (e.g., having an IoU which exceeds 0.5 with any ground-truth instance), resulting in a severe shortage of positive proposals in the early training stage. Although a fixed positive/negative sampling ratio, e.g., 1:3, is often used during training current detectors, a “soft” sampling ratio 1:3 is actually adopted in the practical implementation. For example, assuming batch size to be 512 for R-CNN, a sampling ratio 1:3 requires to sample 128 positive and 384 negative proposals from all the RPN proposals. In the early training stage, the number of all the positive ones $N_{real}^{+}$ provided by RPN is probably lower than 128. In this case, current implementations just use all the $N_{real}^{+}$ positive ones and $512-N_{real}^{+}$ negative ones to form the batch. So the actual positive/negative sampling ratio is smaller than 1:3, which is referred to “soft” sampling ratio 1:3 here. The insufficient positive proposals in the early training stage hurts the model performance, especially on the positive samples.

Magnifying and Annealing R-CNN Gradients. As we have mentioned in Sec. 3.1 – the entire training phase can be viewed as a process of learning what is positive. Fig. 3 Middle shows that the classification accuracy of negative proposals first reaches 98% then starts to drop gradually, while the classification accuracy of positive proposals stably increases. Such a phenomenon indicates that the gradients of positive proposals take over the training process at very early stages (i.e. the average gradient of positive proposals is larger than negative proposals’). We thus propose to simultaneously magnify the gradients from both positive and negative proposals by a factor $\lambda$ because such an operation has a similar effect to only magnifying the weight of positive proposals while keeping the proportion of positive proposals v.s. all proposals unchanged. Gradually decreasing the magnification factor $\lambda$ as the training progresses is introduced to guarantee that the gradients from both positive and negative proposals can always rival each other. We name such a solution “R-CNN Gradient Annealing” (RGA). A formal description of RGA is as follows.

Positive Proposal Imbalance in Testing Phase. As aforementioned, a better per-image accuracy on the testing set can be achieved when the sample ratios on all the classes of the training set are more consistent with that of the testing set. Object detection can be viewed as proposal-level classification. Therefore, the positive/negative proposal ratio during training the R-CNN module is also required to be more consistent with that in the testing phase, for a better testing performance. Although the positive/negative proposal ratios in testing images are not directly decided by the ground-truth instance numbers, they are still highly correlated as shown in Fig. 1 (b). Therefore, we conclude that testing images with diverse ground-truth instance numbers require different positive/negative sampling ratios during training process of the R-CNN module, for the optimal testing results. Based on this finding, we claim that using a single positive/negative sampling ratio in R-CNN module training in the existing works is a sub-optimal solution, when facing the great diversity of ground-truth instance numbers of the testing set (e.g., MS COCO).

Model Structure. To address the above issue, better consistency between the training sampling ratio and ground-truth instance number of each testing image is desired. To achieve this goal, an ideal solution could be: a Faster R-CNN with a set of parallel R-CNN modules trained with different positive/negative proposal sampling ratios; in the testing phase, the model dynamically dispatches each testing image to the best matched R-CNN module for prediction, based on its ground-truth instance number. However, two obstacles impede the effective implementation of this solution. Firstly, the ground-truth instance number is unknown during testing, and the task of the accurate prediction of the ground-truth instance number remains difficult. Secondly, even if the accurate prediction of the ground-truth instance number is possible, the exact matching from each ground-truth instance number to the optimal training sampling ratio is also hard to decide.

Result Ensemble. In the testing phase, multiple R-CNN modules following a shared backbone and RPN allows “Average Ensemble” to be conveniently performed on each testing proposal. “Average Ensemble” effectively combines the decisions of the R-CNN modules biased to different class distributions. Therefore, the combined results are naturally better than that of an R-CNN module trained with a single sampling ratio, when encountering testing images with diverse ground-truth instance numbers.

Gradient Ensemble. Apart from result ensemble in the testing phase, gradient ensemble in the training phase also plays a key role in PRM. Without result ensemble, PRM also enhances the detection performance of each individual R-CNN module compared to training them solely. The performance gain of each individual R-CNN module can only be attributed to the gradient ensemble in the shared backbone. This is similar to the phenomenon that the tasks have mutual benefit to each other in the multi-task learning. Gradients from different tasks are fused together in the shared backbone, strengthening the adaptation of the backbone to different tasks (i.e., testing images of diverse ground-truth instance numbers in our case). We plot the gradients of the weights of Block4.Conv1 in the shared backbone propagated from different R-CNN modules in Fig. 5 Left. It can be seen in Fig. 5 Left that the magnitude of gradients’ vector sum from two R-CNN modules is always smaller than the sum of two gradients’ magnitudes, which means the knowledge provided by two R-CNN modules are not exactly the same. As shown in Fig. 5 Right, a similar phenomenon is observed in Mask R-CNN[9] where the BBox head and the Mask head can be viewed as multi-task heads.

Dataset. COCO[14] is a widely used benchmark in object detection field. In this work, we train all our models on the COCO trainval35k which consists of 118k images, and evaluate our results on the COCO $minival$ which consists of 5k testing images. Our evaluation metric follows the standard COCO style mean Average Precision (AP) at different BBox IoU thresholds.

Implementation Details. To provide a strong baseline, we incorporate FPN[12] and RoI-Align[9] into the naive Faster R-CNN[22]. If not specifically noted, in our paper, the term “Faster R-CNN” represents this modified version and optimized in “jointly training” manner. We train detectors on 8 GPUs (2 images per GPU) with an initial learning rate of 0.02, and decrease it by 0.1 at the 8th and 11th epoch. The magnification factor $\lambda$ is initialized to 7. Following the structure in [12], our R-CNN module only contains 2 shared $fc$ and 2 separate $fc$ layers for classification and regression respectively. In our re-implemented Faster R-CNN[22], the sampling ratio of positive/negative proposals is set to 1:3, which is the same as original papers. For PRM, we use two R-CNN modules with sampling ratios of 1:1 and 1:9.

Main Results. Results on COCO minival are presented in Table 2. As shown in Table 2, incorporating RGA into Faster R-CNN can bring 1.2% improvement on AP. Such a gain reaches to 2.0% after further adopting the structure of PRM (36.1% v.s. 38.1%). RGA and PRM can still improve the baseline by 1.7% (37.3% v.s. 39.0%) when we increase the total training time from 1x to 2x to make sure the models are better optimized. Furthermore, we evaluate our method on a better backbone – ResNeXt-101 and a better two-stage detector – Cascade R-CNN. Results in Table 2 tell that our proposed RGA strategy and PRM can yield consistent improvements across different learning rate schedules, backbones and variants of two-stage detectors. It is worth mentioning that our method can especially improve the performance under AP₇₅ by 2.8% (38.8% v.s. 41.6%) which illustrates the effectiveness of RGA and PRM under more severe evaluation protocol.

R-CNN Gradient Annealing. We study how the annealing of magnification factor $\lambda$ helps improve the final prediction results by setting $\lambda$ as a constant value. Comparing the 1st, 2nd and 5th line in Table 4, we find that although magnifying the gradients in R-CNN module 7 times can bring 0.6% improvement on AP compared to its baseline (36.1%), with the introduction of annealing, the final performance can be further improved by 0.6%, which proves the effectiveness of gradient annealing. Since a new hyper-parameter $\lambda$ is introduced, we test the influence of different $\lambda_{0}$ in Table 4. Results indicate that our method is not sensitive to the value of $\lambda$ within the range of $[5,9]$. It deserves to be mentioned that RGA brings no extra computation cost during testing time. All of these merits make our RGA strategy more applicable.

Sampling Ratios. Sampling ratio is crucial to PRM. To understand this issue better, we investigate the case of two R-CNN modules and run experiments with a set of different sampling ratio pairs. Results are presented in Table 5. It shows that when the two sampling ratios are 1:1 and 1:9, the detector achieves the highest AP – 37.3%. When the two sampling ratios are 1:1 and 1:1, the detector achieves the worst AP which is 36.8%. According to such results, we can conclude that the detection performance is correlated to the gap between the two chosen sampling ratios in a pair. Specifically, the larger gap between two sampling ratios is, the better final result will be achieved. In addition, to prove the performance gain of PRM does not come from more parameters, we train two separate Faster R-CNN models with the sampling ratios of 1:1 and 1:9, whose number of parameters is far larger than a single Faster R-CNN with PRM. The results are presented in the second row in Table 5. Comparing the results in the second and fifth rows, we can see that the performance of Faster R-CNN with PRM exceeds the ensemble of two separate Faster R-CNN models. It means the benefit brought by Gradient Ensemble is larger than the number of parameters, which is one main source of gain of PRM.

Number of R-CNN Modules. The number of R-CNN modules is another hyper-parameter in PRM. Following the conclusion we draw in the previous paragraph, we choose PRM with two R-CNN modules, with the sampling ratios of which are 1:1 and 1:9 as our baseline. After adding the third R-CNN module with the sampling ratio of 1:3, as shown in Table 6, we observe a little performance drop both with and without RGA (-0.2%), which indicates that two R-CNN modules are sufficient. Such phenomenon probably results from that while all R-CNN modules are put on one shared backbone, as the number of R-CNN modules increases, jointly optimizing them may cause over-amplification of the gradients from different losses in the backbone.

The Gain of RGA. The performance gain of RGA has been shown in Sec. 4.1. However, where does such gain come from still remains unverified. The answer is hidden in Fig. 3 which describes two full training processes for a baseline Faster R-CNN and Faster R-CNN with RGA, respectively. Fig. 3 Middle shows that the training accuracy for negative proposals are nearly identical when with and without RGA, which means RGA can not improve model’s ability of identifying negative proposals. However, Fig. 3 Left shows that after applying RGA, the training accuracy for positive proposals is consistently better than baseline. Such results verify our assumption that by magnifying the gradients from both positive and negative proposals, it is positive proposals whose impact are amplified so that the R-CNN module can better identify positive proposals. Consequently, the validation performance is improved.

Comparisons between Different R-CNN Modules. Considering the same optimization of objectives used in different R-CNN modules, it is natural to ask this question: is it possible that those different R-CNN modules actually learn similar parameters to each other? We try to answer this question from the discrepancy between their prediction scores. The detector we use is a well-trained Faster R-CNN with two R-CNN modules trained with the sampling ratios of 1:1 and 1:9. Fig. 6 Left shows the distribution of predicted scores on COCO $minival$ from two R-CNN modules. From Fig. 6, we can learn that the R-CNN module trained with the smaller sampling ratio of positive proposals tends to predict lower scores, and vice versa, which is already stated in Sec 1. Fig. 6 Right visualizes the absolute difference of scores between corresponding outputs. It shows that there are more than 21.9% pairs of prediction scores have the absolute difference larger than 0.1. Such a phenomenon verifies our claim in Sec. 1 that R-CNN modules with different sampling ratios can show different prediction biases, which is the reason why output ensemble can bring improvements. The improvements of PRM on different subset of COCO $minival$ can be seen in Fig. 2.