Proposal Distribution Calibration for
Few-Shot Object Detection

FSL methods can be roughly divided into three categories: 1) the meta-learning-based ones [36, 35, 34, 39] typically design specific models or optimization for fast model adaption; 2) the metric learning-based ones [28, 29, 30, 31, 32, 33, 7] pursue optimizing the embedding manifold for feature discrimination enhancement; 3) the data augmentation-based ones [38, 37] target augmenting novel data to improve the model generalizability.

As FSL shares similar intrinsic challenges with FSOD while relatively simple, seminal FSL works continually inspire the development of FSOD [40]. Concretely, MVT [16] transfers factors of variations across classes via semantic transformations to alleviate the data requirement. DC [17] steps forward by economically migrating distribution statistics from base classes to their most similar novel classes for data sampling. These valuable explorations encourage the in-depth attempts in FSOD to exhaust the base knowledge at a low cost.

General object detection methods can be commonly categorized as one-stage and two-stage detectors, which are continually evolving with the tremendous progress in deep neural architectures [21]. One-stage detectors [18, 19] locate and classify objects simultaneously to achieve better efficiency. Two-stage detectors [20] propose suspicious objectness regions, then attempt to discriminate and rectify each proposal for superior accuracy. However, these achievements are severely data-hungry for generalization, which limits their real-world application.

Few-shot object detection, initially inspired by meta-learning [1, 2], targets expanding the object vocabulary from label-rich scenarios to the open world with limited supervised showcases. TFA [5] advocates to keep the backbone learned during base training frozen and only fine-tune the box regressor and classifier. Such a simple two-step training scheme reports surprisingly superior performance, which encourages and inspires the FSOD community to maximize the usage of base knowledge for fast adaptation.

Concretely, subsequent fine-tuning-based few-shot detectors typically delve into optimizing embedding space via metric learning [8, 7], relation reasoning [27], and feature interaction [11], alleviating the essential objective contradiction across training stages via base knowledge retention [13], detection head decoupling [9, 10], and annotation rectification [26], and augmenting novel instances via object pyramids [6], data hallucination [22], etc. In this work, we intend to bridge the semantic gap in repurposing the class-aware RoI head for novel classes by applying PDC on the class-agnostic RPN with base proposal statistics as a “free lunch.”

Assume the input proposal of the RoI head after base training is denoted by $\bm{p}_{B}\sim{\mathbb{P}}_{B}$ and the input proposal during fine-tuning is denoted by $\bm{p}_{F}\sim{\mathbb{P}}_{F}$, where ${\mathbb{P}}_{B}$ and ${\mathbb{P}}_{F}$ are the corresponding proposal distributions, respectively. They are sampled from different data distributions as $i.i.d.$. Let ${\mathbb{P}}$ denote proposal distribution after sufficient dataset $\cal D$ training and ${\mathbb{P}}_{B}$ is approximately equal to ${\mathbb{P}}$.

We can determine the relationship between the true risk and the empirical risk as follow:

where $l(\bm{p})\in\mathcal{L}$ is the loss function of fine-tuning for the RoI head, and $\mathcal{L}=\mathcal{L}_{Re-Roi}+\mathcal{L}_{Roi}$. $n$ is the number of proposals with probability at least $1-\delta$. ${\rm E}_{\mathcal{D}}(l(\bm{p}_{F}))$ denotes the true risk and $\hat{{\rm E}}_{{\mathcal{D}}_{f}}(l(\bm{p}_{F}))$ denotes the empirical risk for available labeled data.

Following  [23], we can obtain the empirical risk upper bound as follow:

Note that $\sqrt{\frac{\ln 1/\delta}{n}}$ and $2R_{n}(\mathcal{L})$ are constants when the dataset and the model are given. As $\hat{{\rm E}}_{{\cal D}_{f}}(l(\bm{p}_{F}))$ is minimized during model finetuning, Eq. 12 can be approximated as

where MMD [24] is the maximum mean discrepancy term, which is proved as

where $N_{b}$ and $N_{f}$ denote the input proposal numbers of RoI head of during base training and fine-tuning respectively. $\mathcal{H}$ denotes the RKHS(Reproducing Kernel Hilbert Space).

According to Eq. 14, when the mean and variance differences of distributions ${\mathbb{P}}_{B}$ and ${\mathbb{P}}_{F}$ are decreased, the maximum mean discrepancy of distributions ${\mathbb{P}}_{B}$ and ${\mathbb{P}}_{F}$ are reduced at the same time. Combining Eq. 13 and Eq. 14, the empirical risk upper bound is effectively reduced.

Datasets. Pascal VOC 2007 [41], VOC 2012 [42], and MS COCO [43] datasets are commonly used for FSOD evaluation. They are typically split into fully annotated base classes and $K$-shot novel classes in accordance with the settings in FSRW  [1]. Specifically, the Pascal VOC dataset is divided into three splits for cross validation, where 5 classes are chosen as novel classes and the rest 15 classes are base classes. The number of annotated instances $K$ is set to 1, 2, 3, 5, and 10. For MS COCO, twenty categories are chosen as novel classes with $K$ set to 10 and 30, and the remaining 60 categories are treated as base classes.

Implementation Details. We mainly conduct the ablation study and model analysis of PDC based on the baseline detector imTED [12]. ImTED [12] with ViT-S [45] backbone is implemented with Pytorch 1.8.0, mmcv 1.4.0, and mmdetection 2.11.0 on 8 NVIDIA A40 GPUs. We feed each GPU 2 images with data augmentations of random cropping, horizontal flipping, random resizing, and size normalization for base training and finetuning with the AdamW optimizer.

The validity of PDC components with proper hyper-parameters on the Pascal VOC split-1 is justified in Table I. The average PDC improvement with the proposal sampling module over the baseline method is 1.6%. With contrastive learning, the performance gain further rises to 3.0%, demonstrating the significant advancement brought by our PDC.

Proposal Sampling In Table I, we evidence that sampling proposals from distributions similar to that produced by base trained RPN, i.e., ${\mathbb{N}}(\bm{\mu}_{b},\bm{\sigma}_{b}^{2})$ which is defined in Eq. 2, insures superior novel class detection adaptation. Specifically, we denote the optimal uniform distribution of maximum union with ${\mathbb{N}}(\bm{\mu}_{b},\bm{\sigma}_{b}^{2})$ as $\mathbb{U}^{*}$, where $\mathbb{U}^{*}$ = [U(-0.055,0.055), U(-0.036,0.036), U(-0.077,0.077), U(-0.057,0.057)]. As shown in rows 3-7 of Table I, when sampling from uniform distributions with different perturbation intensities, the performance gain progressively improves as the sampling distribution approaches ${\mathbb{N}}(\bm{\mu}_{b},\bm{\sigma}_{b}^{2})$, justifying the efficacy to calibrate proposals for fine-tuning using base training statistics. From rows 5 and 8-10, one can see that with more high-quality positives sampled (less than 50 per instance), the average performance gain increases from 1.0 to 1.5, while drops to 0.9 when too many positives break the balance of positive and negative samples for RoI head repurposing. As adopting ${\mathbb{N}}(\bm{\mu}_{b},\bm{\sigma}_{b}^{2})$ reports slightly higher performance than $\mathbb{U}^{*}$, i.e., 0.1 “avg. $\Delta$” gain, rows 9 and 11, we sample 50 proposals from the Gaussian distribution of ${\mathbb{N}}(\bm{\mu}_{b},\bm{\sigma}_{b}^{2})$ by default.

Contrastive Learning Following FSCE [8], we introduce an MLP head for contrastive learning. From row 1-2 in Table I, one can see that it is not wise to directly impose contrastive loss on the biased RPN proposals. While according to rows 12-16, it is clear that with $\mathcal{L}_{Re\mbox{-}Roi}^{con}$ applied on the proposals sampled from the calibrated distribution, the detection performance improves consistently with different settings of temperature and dimensions of the contrastive MLP head output. With the superior setting of the temperature as 0.2 and the output dimension as 128, we confirm again from results in row 13, 17, and 18 that it is better to absorb discriminative representations by only applying contrastive learning upon high-quality proposals from ${\cal P}_{S}$. These facts enhance the importance of the proposed PDC approach for current FSOD with the two-step training paradigm.

Losses for Repurposing RoI Head As defined in Eq. 7, 8, and 9, our PDC attaches three loss terms to repurpose the RoI head for novel class detection, i.e., $\mathcal{L}_{Re\mbox{-}Roi}^{con}$, $\mathcal{L}_{Re\mbox{-}Roi}^{cls}$, and $\mathcal{L}_{Re\mbox{-}Roi}^{reg}$. From rows 19–22 in Table I, the combination of three losses based on ${\cal P}_{S}$ reports the best performance (+3.0 “avg. $\Delta$” gain), indicating that high-quality proposals with these losses facilitate the classification and localization abilities of the detector.

In this subsection, we quantitatively and qualitatively reveal the specific advantages of our PDC based on imTED [12] with the 5-shot setting, including aspects of the RoI head’s localization and classification capabilities, embedding space, and detection results.

Concretely, we visualize the IoU histograms of bounding boxes predicted by the RoI head of the baseline method, i.e., imTED [12], w or w/o our PDC after fine-tuning in Fig. 3. It is clear that our PDC remarkably improves the localization ability of the RoI head, which predicts more accurate bounding boxes (high IoU with ground truth) whose distribution approaches the base class statistics produced with sufficient supervision. In Fig. 4, we explore the detailed classification precision improvement of the RoI head for localized instances with bounding boxes of different IoUs. PDC achieves better results than the baseline not limited in proposals of high IoU, which validates that fine-tuning with the calibrated proposal distribution facilitates the improved classification ability generalizing to low IoU cases, cf. Sec. III-D. so, from the figure, PDC classification ability improvement is proven.

In Fig. 5, we visualize the RoI feature embeddings of the baseline method and PDC after 5-shot finetuning. It is obvious that our PDC produces superior feature space with better discrimination, especially for novel classes. Fig. 6 showcases some detection results of the baseline and PDC on Pascal VOC and MS COCO datasets. Since the calibrated proposal distribution encourages to recall the novel class suppressed in base training, it is clear that our PDC can significantly reduce missed samples, i.e., green boxes in the figure.