Generalized Few-Shot Object Detection without Forgetting

Following previous literature[17, 42], we split the categories of a dataset into base classes $\mathcal{C}_{b}$ and novel classes $\mathcal{C}_{n}$, with $\mathcal{D}_{b}$ and $\mathcal{D}_{n}$ denoting the corresponding sub-datasets, respectively. $\mathcal{D}_{b}$ contains abundant annotations for training, while only a few data in $\mathcal{D}_{n}$ is available. Our objective is to learn a detection model $f(\cdot)$ for both $\mathcal{C}_{b}$ and $\mathcal{C}_{n}$ from the few novel class samples without forgetting the learned capability from the abundant base class samples.

Such an objective can be easily achieved by meta training a model to perform an exemplar-based visual search on $\mathcal{D}_{b}$, then directly deploy it without finetuning, as in one-shot detection literature [14, 30]. However, these methods do not perform as good as ordinary detection methods on $\mathcal{D}_{b}$ and require high time and space complexity. On the contrary, transfer learning based methods can efficiently deal with full-way detection and achieve competitive results on $\mathcal{C}_{n}$, as demonstrated in previous works[42, 45]. Thus we propose to tackle the problem of G-FSD in a transfer learning paradigm: first obtain a base model $f^{b}$ by training on $\mathcal{D}_{b}$ and then obtain a novel model $f^{n}$ via finetuning $f^{b}$ on $\mathcal{D}_{n}$ (or a combination of $\mathcal{D}_{b}$’s subset and $\mathcal{D}_{n}$). However, the finetuning stage tends to degrade the base class performance due to the forgetting effect[29, 27] if it is finetuned on $\mathcal{D}_{n}$, or due to the sample limitation on $\mathcal{D}_{b}$ to balance class frequency if finetuned on both classes. Regarding this problem, a question is probably raised: is the degradation unavoidable?

To answer this question without loss of generality, we analyze TFA[42]’s properties as a representative transfer learning model on few-shot detection tasks. TFA is first pretrained on $\mathcal{D}_{b}$ as ordinary R-CNN, then the last layers in classification and box regression heads are tuned on $\mathcal{D}_{n}$. The finetuned novel class heads’ weights are concatenated with base class weights as the initialization for the final finetuning on a combined dataset consists of $\mathcal{D}_{n}$ and $\mathcal{D}_{b}$’s subset, where the number of samples per category is enforced to be identical. A slow and steady learning schedule is also applied during the final finetuning stage. Take 10-shot settings on MS-COCO for example, AP on $\mathcal{C}_{b}$ is better reserved than a pure finetuning baseline (31.8 to 35.0), though AP of the base class detector can achieve as high as 39.2.

Why cosine classifier works? Cosine classifier is commonly adapted in few-shot classification[9, 31] as cosine similarity bridged transfer learning and metric learning approach and generally performs well on base and novel class trade-off. The conclusion remains valid for TFA[42] as the base class performance is generally higher with a cosine classifier. We collect the ROI features from an R-CNN pretrained on $\mathcal{D}_{b}$ of MS-COCO[25] and compute the average pixel-wise L2-norm of $\mathcal{C}_{b}$ and $\mathcal{C}_{n}$. The results are shown in Figure2(a). A massive variation of norms between base classes and unseen novel classes can be easily observed. This may account for the effectiveness of cosine classifiers for being agnostic to feature norms. Also, the norms of unseen classes with closer relationship with seen classes are relatively higher (blue names annotated in Figure2(a)).

Does base detector find novel class salient objects? To a large extent, no. We hypothesize it is due to the features deactivated during training on $\mathcal{C}_{b}$ as indicated by low L2-norm, which will not produce a high confidence score with a dot-product classifier. We visualize the detection results and its feature norms on FPN[23] P3 in Figure2. The local features around the people in the first two images are obviously deactivated, although the object is of great saliency to humans. Features of the bus are somewhat activated in the third image of Figure2(b) (probably due to close relationship with trucks of $\mathcal{C}_{b}$), yet the detector is still able to recognize it as background. More results indicating this property are provided in the supplementary materials without cherry-picking. To quantitatively answer this question, the average recall between $f^{b}$’s RPN proposals, final outputs, and ground truths of unseen class $\mathcal{C}_{n}$ (uAR) and seen classes (AR), is calculated in Table1. The drastic drop of uAR well demonstrates the ability of $f^{b}$ to reject novel class objects. Thus we can utilize this property to reserve base class performance as $f^{b}$ does not introduce many false positives on $\mathcal{C}_{b}$ when encountering novel class instances.

Is RPN class-agnostic? While most transfer learning[42, 45] and meta-learning[47, 46, 18] works treat RPN as class-agnostic and freeze it during finetuning, RPN is not ideally class-agnostic and biased on its seen categories. During training on $\mathcal{D}_{b}$, anchors of novel class instances are categorized into non-object due to lack of annotations, making RPN bias on training samples. We compare AR of all classes of a finetuned RPN on $\mathcal{C}_{b}\cup\mathcal{C}_{n}$ under 10-shot setting with pretrained RPN in Table2, where the apparent improvement validates our answer.

Our proposed model for G-FSD, Retentive R-CNN, consists of Bias-Balanced RPN and Re-detector to utilize the aforementioned properties of the base class detector $f^{b}$. The model architecture is illustrated in Figure3.

Re-detector. Re-detector consists of two detector heads, predicting detections of $\mathcal{C}_{b}$ and $\mathcal{C}_{b}\cup\mathcal{C}_{n}$ from object proposals in parallel, where one stream remains the same weight as in $f^{b}$ to predict objects of $\mathcal{C}_{b}$ (denote as $det^{b}$) and the other holds the finetuned weights to detect objects of both $\mathcal{C}_{n}$ and $\mathcal{C}_{b}$ (denote as $det^{n}$). Detecting both classes can well alleviate the false positives due to inadequate data training, as shown in Section4.3. $det^{b}$ utilize a fully-connected layer for classification and $det^{n}$ use a cosine classifier to balance the variation of features in their norms. Similar to TFA, we finetune merely the last layers of classification and box regression head of $det^{n}$, which is capable of producing competitive results.

As $f^{b}$ is trained from abundant data, we hope that $det^{n}$ can inherit the reliable knowledge of $f^{b}$. Towards this end, we propose an auxiliary consistency loss to regularize $det^{n}$ to score object proposals similar to $det^{b}$ on the base class entries, which takes the form of KL-Divergence as in previous knowledge distillation works[15, 49]. For proposals of $C_{b}$, $det^{n}$ is enforced to predict high confidence, and for proposals not belonging to $\mathcal{C}_{b}$, $det^{n}$ mimics $det^{b}$ with similarly low probabilities. Given the final probabilities $p_{c}^{b}$ and $p_{c}^{n}$ of class $c$ predicted by $det^{b}$ and $det^{n}$, the consistency loss is formalized as:

where $\tilde{p_{i}^{n}}=\frac{p_{i}^{n}}{\sum_{c\in\mathcal{C}_{b}}p_{c}^{n}}$ and the same for $\tilde{p_{i}^{b}}$. This is quite different from TK in LSTD[1] where the KL-Divergence is computed between the highest probabilities of $\mathcal{C}_{b}$ and $\mathcal{C}_{n}$. Note that $p_{i}^{n}$ is the normalized marginal probability distribution over base classes after softmax over all class entries. The total loss of Re-detector during finetuning stage is

where $\mathcal{L}_{cls}$ and $\mathcal{L}_{box}$ takes the same form as Faster R-CNN[35] and is computed on $det^{n}$ only, and $\lambda$ denotes the coefficient for the consistency loss.

Bias-Balanced RPN. R-CNN relies on RPN to generate object proposals as training samples for second stage classification and other subsequent processing. The quality of RPN proposals is especially crucial when the network is trained under low-data scenarios. As shown in Section3.2, a pretrained RPN may fail to catch novel class objects, further aggravating the scarcity of samples, while a finetuned RPN can alleviate this issue, thus providing better samples for second-stage modules to learn. We try to unfreeze different layers of RPN for finetuning and empirically, unfreeze the final layer that predicts objectness is sufficient to produce a noticeable improvement (results given in Setion4.3).

To retain performance on base classes, we propose Bias-Balanced RPN to integrate both pretrained RPN and the finetuned one. It ensembles the objectness prediction heads to raise $\mathcal{C}_{b}$ and $\mathcal{C}_{n}$ proposals properly. Given a feature map of size $H\times W$, base RPN predicts an objectness map $\mathcal{O}^{H\times W}_{b}$ and finetuned RPN predicts $\mathcal{O}^{H\times W}_{n}$, the final output objectness of Bias-Balanced RPN is defined as $\mathcal{O}^{H\times W}=\max(\mathcal{O}^{H\times W}_{b},\mathcal{O}^{H\times W}_{n})$. Note that, during the finetuning stage, only the objectness of finetuned RPN is set unfrozen. Box regression and the convolution layer are shared across base RPN and finetuned RPN, as illustrated in Figure3. Theoretically, the max operation guarantees the RPN not to overlook proposals of previously learned classes catastrophically. With little computational overhead and extra weights, we believe Bias-Balanced RPN can serve as a general component for G-FSD. The full loss function of Retentive R-CNN during the finetuning stage is

where $\mathcal{L}^{n}_{obj}$ is the binary cross-entropy loss on finetuned RPN’s objectness layer.

Training. As a transfer learning based method, Retentive R-CNN is trained in two stages: pretraining on $\mathcal{D}_{b}$ and then finetune on the combined dataset of $\mathcal{D}_{n}$ and $\mathcal{D}_{b}$’s subset. As aforementioned, we only unfreeze three layers: objectness of the finetuned RPN, the last linear layers of classification and box regression of $det^{n}$. Thanks to the capability of retaining base class performance, we can apply a swifter learning schedule for finetuning, e.g., 5000 iterations for 10-shot MS-COCO[25] compared to 160000 iterations of TFA[42].

Inference. Given the object proposals from Bias-Balanced RPN, the corresponding features are fed into the two heads of Re-detector in parallel. The set of predicted boxes of both heads are gathered into one for the final NMS procedure. As $det^{b}$ is somehow more reliable as it learns from abundant data, we add a little bonus (0.1 in our implementation) for the scores predicted from $det^{b}$ if they surpass the pre-NMS threshold, which could encourage the NMS procedure to take $det^{b}$’s output when $det^{b}$ and $det^{n}$ find similar base class results. More details will be described in the supplementary material. As the backbone and feature transformation layers in Bias-Balanced RPN and Re-detector are shared among both detector heads, we can maintain the base class performance with little overhead compared to an ordinary R-CNN.

We evaluate our method on the well-established few-shot detection benchmark[42, 17] based on MS-COCO[25] and Pascal VOC [5], following the same class splits and data splits in previous works[17, 42, 45] for a fair comparison. We report 5,10,30-shot results on MS-COCO and 1,2,3,5,10-shot results on 3 random splits of Pascal VOC. Towards the problem of G-FSD, the overall performance of both classes is our major concern. We reproduce Meta R-CNN[47] and FsDetView[46] using exactly the same samples for finetuning without hyperparameter changing (by running their official code) and denote the reproduced results with a * at the upper right corner. Results for ONCE[16], MetaDet[44] and FSRW[17] are reported from their original paper.

We use an ImageNet pretrained ResNet-101[13] with FPN[23] as the backbone. Pretraining on $\mathcal{D}_{b}$ is the same as in [42], then the finetuning layers are initialized by random. For all experiments, we set learning rate to 0.05 and $\lambda$ to 0.1 to finetune until full convergence.

We compared our results with both transfer learning[42, 45] and meta-learning based methods[16, 47, 17, 46]. Towards maintaining base class performance, one can quickly come up with an R-CNN model with N binary classifiers for detecting a dataset of N classes as binary classifiers are decoupled with each other. We also train such a model (denoted as FRCN-BCE) with binary cross-entropy loss for ROI classification as a strong baseline, using the same hyperparameters as Retentive R-CNN except for initializing the classifiers’ bias as in RetinaNet[24].

Results on MS-COCO[25]. Table3 shows mean average precision over 0.5 to 0.95 IOU thresholds on all, base, and novel classes (AP, bAP, nAP) under different data settings. We outperform previous methods significantly on AP and bAP, as our method does not degrade on base classes at all. Meanwhile, we achieve competitive results on novel classes as well (state-of-the-art for 10-shot and on-par with state-of-the-art for 5- and 30-shot).

Towards the same objective to incrementally detect rare objects, ONCE[16] does not degrade the base class performance as well, yet its performance on both classes is limited. The very competitive TFA[42] models can gradually recover base class performance with samples increasing; however, the gap is still indispensable, e.g., the bAP gap between 30-shot TFA w/cos[42] and the base model is as large as 3.4. As expected, FRCN-BCE can maintain base class performance from its pretrained model intrinsically, but the performance on both base and novel class is lower than an ordinary RCNN by a large margin. Given that Retentive R-CNN only adds little overhead with layers mostly shared, our method is a superior choice for G-FSD. Despite MPSR[45] slightly outperform our method with respect to nAP on 30-shot, the performance drop on base classes is significant, and thus it is not suitable for G-FSD. An even larger performance drop can be observed in Meta R-CNN[47] and FsDetView[46], probably because they predict the whole probability distribution of an ROI from features reweighted by a certain class-attentive vector. The vast performance drop is alleviated to some extent in FSRW[17] (see Table4), which only predicts the probability for the class of the reweighting vector.

Results on Pascal-VOC[5]. Table4 and Table5 show overall and novel class results on VOC benchmark respectively. Results of Meta R-CNN[47] from original paper are also included in Table5 as a reference. Note that the results are not directly comparable because samples used for finetuning are different, which can make a significant impact on the final metrics. We consistently outperform all methods on overall AP across all datasplits thanks to the non-forgetting property. As stated above, MPSR[45] and several other meta-learning methods[47, 46, 17] do not perform well on overall performance as the base class knowledge is forgotten during the finetuning stage.

Notice that performance on novel classes is not our primary concern, though, competitive results are achieved by Retentive R-CNN under most cases on VOC novel classes as shown in Table5. MPSR[45] made most of the best nAP records; however, non-negligible base class performance is sacrificed. Compared to the methods that better preserves base class performance, we outperform the current best TFA[42] in most cases, with approximative results under the rest.

Without loss of generality, we conduct ablation experiments on COCO benchmark under 10-shot scenario. All models are trained with the same hyperparameters unless otherwise stated.

Bias-Balanced RPN. To validate the effectiveness of our design, results on RPN recall and final detection precision for different classes of different RPN designs, including the ensembling strategy for both RPN outputs and the choice of unfrozen layers during finetuning, are evaluated in Table6. Taking max as the ensembling strategy performs best, among other alternatives. Taking geometric average significantly degrades performance because any low objectness will produce a low final score. It can also be observed from the experiment that novel class AP is tightly related to AR of the RPN, while base class AP can remain stable with slightly inferior RPN AR, which validates one of our design philosophies to debias RPN thus improve novel class performance. Unfreezing box regression layers and ensembling does not make much difference. Thus the extra computation overhead is not necessary.

Re-detector. We study various design options in Re-detector, including the form of consistency loss (KL divergence, L1 difference, and negative cosine similarity between the normalized marginal probability distribution on base classes), layers in Re-detector to set unfrozen and classifier choice. As shown in Table7, our current design maximizes the overall performance. Surprisingly, unfreezing more layers even lowers the performance.

In addition, to validate the necessity of finetuning on both base and novel classes, we also implement a Re-detector where $f^{n}$ only detects $\mathcal{C}_{n}$. It produces relatively low results, probably due to severer false positives as diverse objects in base classes are encountered during test time, but unseen during finetuning, and the model is trained to categorize these objects the same as those deactivated background features from only a handful of samples, which is undoubtedly challenging.

Inference time. We report average inference time per image on COCO 2014 test set by adding modules into Faster R-CNN in Table8. The inference time of Meta R-CNN[47] is also provided as a reference for representative meta-learning methods that demand exemplars for inference, which also introduces much lower extra computation than others, to the best of our knowledge. As most weights are set frozen and shared, Retentive R-CNN introduces little overhead during test time to realize few-shot detection without forgetting, especially compared to meta learned models requiring exemplars at test time.

Visualization. We provide exemplary results obtained by Retentive R-CNN and TFA w/cos[42] in Figure4 for comparison under MS-COCO 10-shot setting. The non-forgetting property of our method can be observed from the last four images containing either crowded scenes or less salient instances where TFA[42] tends to ignore some of these objects, e.g., the inconspicuous baseball bat in the third image is ignored, and many well-learned objects are overseen in the fourth image by TFA[42]. We also perform better on novel classes under certain cases, as shown in the first two images.

We further investigate the role of consistency loss by comparing the classification distribution of our method and TFA w/cos[42] and a base detector. Specifically, we show two representative examples for the base class and novel class and visualize the logits of their classifiers for analysis. To make a fair comparison, both our method and TFA w/cos[42] are trained upon this same base detector. It can be easily observed that our method produces much more similar logits distribution as the base model on base classes rather than TFA w/cos[42]. Such property can better reserve base class performance, as shown in Figure5(a), where the base model and ours produce unimodal distribution with one strong peak. When it comes to novel classes, as shown in Figure5(b), base class distribution is suppressed, thus making a more confident response to novel classes.