SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object Detector

Open-world object detection (OWOD) is a more practical detection problem in computer vision, facilitating the development of object detection (OD) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] in the real world. Within the OWOD paradigm, the model’s lifespan is pushed by an iterative learning process, as shown in Fig. 2 (a). At each episode, the model trained on the data with known object annotations is expected to detect known objects and all potential unknown objects. Human annotators then label a few of these tagged unknown classes of interest gradually. The model given these newly-added annotations continues incrementally updating its knowledge without retraining from scratch.

In the existing works [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23], the open-world detectors are expected to know about the open world through several datasets [24, 25] with tiny scales. However, the annotations of these datasets are too few to provide adequate object attributes for the model. It is difficult for the model to achieve the ideal goal through these datasets. The large pre-trained vision-language grounding models (VL) [26, 27, 28, 29, 30] have rich knowledge of the open world due to countless millions of parameters, huge open-world datasets, and training costs.

However, their detection could not leave the participation of text prompts. Before detecting, the text prompt of all objects must be pre-listed to be detected, and the object whose text prompt was not listed could not be detected. There are many detection scenarios in which pre-defined language descriptions are unavailable during inference. Therefore, it poses a challenge how to equip the VLM with language-agnostic unknown object detection capability. In addition, their detection speed is also a criticism due to the following question. $i)$ The huge number of parameters and FLOPs. $ii)$ The large pre-trained vision-language grounding model could only infer with several text prompts for the detecting performance, so they must infer many times when the number of prompts is large.

Humans’ ability to recognize objects they have not seen before largely depends on their brains’ knowledge base. Inspired by how humans face the open world, we propose to learn from the large pre-trained vision-language grounding models by knowledge distillation for OWOD. In this paper, we attempt to specialize the VLM model for OWOD tasks by distilling its open-world knowledge into a language-agnostic detector, as shown in Fig.1. We were surprised to observe that the simple knowledge distillation approach for the OWOD algorithm can achieve better performance for unknown object detection than the large open-vocabulary pertaining vision-language model, even with a small amount of data. Unfortunately, knowledge distillation for unknown objects severely affects the learning of the detector with conventional structures for known objects due to the following challenges:

(I) Different from traditional Knowledge Distillation [31, 32, 33, 34, 35], the teacher and student contain different modalities and have different training manners, structures, and inference procedures. Meanwhile, the learning process of OWOD algorithms [12, 13, 36, 15, 14, 37] always has its own specific open-world pseudo-unknown labels. Therefore, the existing distilling objectives do not work and it is difficult to mix the learning of known grounding truth, unknown distilling knowledge and unknown pseudo-knowledge. The performance of detecting known objects is also crucial for OWOD. Through the experiments, we reveal that the direct use of distilled knowledge dramatically affects the model’s learning ability of the original annotations. The model’s performance in detecting known objects is damaged substantially. To alleviate this, we propose the down-weight training loss for the distillation training, which utilizes the distilled labels’ object confidence from the large pre-trained vision-language grounding model and the searched pseudo objectness to reduce the weight of unknown loss in the total loss during training, as shown in Fig.2 (b).

(II) The presence of objects with highly similar features to known classes within the “unknown objects” can greatly affect the process of open-world object identification. In particular, the inclusion of unknown objects will impact the detector’s performance on known objects. This issue impacts not only the identification process but also the localization process for models that use coupled information for both tasks. Therefore, we propose decoupling the detection learning process. However, the decoupled structure completely separates the localization and identification, leading to many potential problems (e.g.mismatch between localization and identification results). Therefore, we propose a cascade structure that decouples the detecting process via two decoders and connects the two decoders via a cascade manner, as shown in Fig.2 (a). In this structure, foreground localization can be protected from category knowledge because the loss of identification is diluted by the latter decoder. Moreover, the identification process can utilize the localization information because it leverages the former decoder’s output embeddings as the input queries.

Since existing datasets [24, 25] are manually annotated with predefined categories, current benchmarks cannot comprehensively measure the detection performance of open-world detectors for unknown objects due to the lack of bounding box annotations for unknown entities in the test scenarios. UnSniffer [38] chooses testing images with only a few unknown foreground objects (no more than 3) from the MSCOCO dataset [24] to evaluate the detector’s ability to identify unknown objects. Inspired by this, we propose two benchmarks, named StandardSet$\heartsuit$ and IntensiveSet$\spadesuit$ respectively, based on the complexity of their testing scenarios. Different from UnSiffer follows the manual annotation guidelines of the MSCOCO [24] dataset, inspired by the more detailed annotation criteria of finer-grained datasets [39], we manually select suitable evaluation scenes (with no fewer than three unknown objects) and provide more meticulous manual annotations of unknown objects. Moreover, our IntensiveSet$\spadesuit$, features an average of over 33 unknown open-world object instances per test scene. Our contributions can be summarized fourfold:

• $\bullet$

  We observe that the simple knowledge distillation could convert the open-world knowledge in the large pre-trained vision-language grounding model for the specialized OWOD task and propose a simple framework with surprisingly good performance.

• $\bullet$

  To mitigate the effect of distilled knowledge on the detection performance of known objects, we propose the down-weight training loss function for the detector’s mixed learning process of known ground truth, distilled unknown knowledge, and the pseudo unknown knowledge in OWOD algorithm. Meanwhile, a cascade decoupled detection transformer structure is proposed to alleviate the influence caused by unknown objects on detecting known objects.

• $\bullet$

  We propose two novel benchmarks to comprehensively evaluate the ability of the open-world detectors to detect unknown open-world objects, named StandardSet$\heartsuit$ and IntensiveSet$\spadesuit$ respectively, based on the complexity of their testing scenarios.

• $\bullet$

  Our extensive experiments on existing and proposed benchmarks demonstrate the effectiveness of our framework. Our model exceeds the distilled large pre-trained vision-language grounding model for OWOD and state-of-the-art methods for OWOD and IOD.

$\mathcal{K}^{t}=\{1,2,\ldots,C\}$ denote the set of known object classes and $\mathcal{U}^{t}=\{C+1,\ldots\}$ denote the unknown classes which might be encountered at the test time, at the time $t$. We labeled the known object categories $\mathcal{K}^{t}$ in the dataset $\mathcal{D}^{t}=\{\mathcal{J}^{t},\mathcal{L}^{t}\}$ where $\mathcal{J}^{t}$ denotes the input images and $\mathcal{L}^{t}$ denotes the corresponding labels at time $t$. The training image set consists of $M$ images $\mathcal{J}^{t}=\{i_{1},i_{2},\ldots,i_{M}\}$ and corresponding labels $\mathcal{L}^{t}=\{\ell_{1},\ell_{2},\ldots,\ell_{M}\}$. Each $\ell_{i}=\{\mathcal{T}_{1},\mathcal{T}_{2},\ldots,\mathcal{T}_{N}\}$ denotes a set of $N$ object instances with their class labels $c_{n}\subset\mathcal{K}^{t}$ and locations, $\{x_{n},y_{n},w_{n},h_{n}\}$ denote the bounding box center coordinates, width and height respectively.

The artificial assumptions and restrictions in closed-set object detection are removed in Open-World Object Detection. It aligns object detection tasks more with real life. It requires the trained model $\mathcal{M}_{t}$ to detect the previously encountered known classes $C$ and identify an unseen class instance as belonging to the unknown class. In addition, it requires the object detector to be capable of incremental updates for new knowledge, and this cycle continues over the detector’s lifespan. In the incremental updating phase, the unknown instances identified by $\mathcal{M}_{t}$ are annotated manually. Along with their corresponding training examples, they update $\mathcal{D}^{t}$ to $\mathcal{D}^{t+1}$ and $\mathcal{K}^{t}$ to $\mathcal{K}^{t+1}=\{1,2,\ldots,C,\ldots,C+\text{n}\}$. The model adds the $n$ new classes to known classes and updates itself to $\mathcal{M}_{t+1}$ without retraining from scratch on the whole dataset $\mathcal{D}^{t+1}$.

This section elaborates on the proposed framework in detail. We start by illustrating the overall scheme of our proposed framework in Sec.4.1 Furthermore, we introduce the open-world object detector in Sec.4.2, the distillation process in Sec.4.3, and the matching and pseudo-labeling procedure of the owod algorithm in Sec.4.4. Last but not least, we describe the down-weight training strategy in Sec.4.5 and the inference phrase in Sec.4.6.

Since existing datasets [24, 25] are manually annotated with predefined categories, current benchmarks cannot comprehensively measure the detection performance of open-world detectors for unknown objects due to the lack of bounding box annotations for unknown entities in the test scenarios. In this paper, we propose two benchmarks, named StandardSet$\heartsuit$ and IntensiveSet$\spadesuit$ respectively, based on the complexity of their testing scenarios. For the proposed benchmark, the known categories are defined as the PASCAL VOC [25] dataset, and we train the open-world detectors by the images and annotations of the PASCAL VOC [25] training set. Meanwhile, inspired by the detailed annotation criteria of finer-grained datasets [39], we manually select suitable evaluation scenes (with no fewer than five unknown objects) and provide more meticulous manual annotations of unknown objects to construct the testing set, as shown in TABLE.I.

Open-world object detection makes artificial intelligence smarter to face more problems in real life. It takes object detection to a cognitive level, as the model requires more than simply remembering the objects learned, it requires deeper thinking about the scene. When it comes to applications like autonomous driving, the significance of open-world object detection becomes even more pronounced. In such scenarios, vehicles need to rapidly and accurately identify and comprehend various objects and obstacles on the road, including but not limited to pedestrians, other vehicles, traffic signals, and road signs. The breakthrough in open-world object detection will render autonomous driving systems more intelligent as they can handle unforeseen or rare situations, not limited to pre-trained object categories.

Although our results demonstrate significant improvements over existing state-of-the-art methods, the performances are still on the lower side due to the challenging nature of the open-world detection problem. In this paper, we are mainly committed to enhancing the model’s ability to explore unknown classes. However, the confidence level and recognition ability of our model for the detection of unknown objects still need to be improved, and this is what we will strive for in the future. Currently, our model still faces the issue of mistakenly detecting the background as unknown objects, and the benchmark we proposed is not yet comprehensive, including only a single-moment task.

In our future work, these two points will be the main focus of our research. In addition, post-processing operations for prediction boxes of unknown objects are urgently needed, so developing a post-processing operation similar to NMS, dedicated to unknown objects, is also a direction that requires our research. To facilitate the integration of open-world object detection algorithms into everyday use and contribute positively to society, we are fully committed to diligently pursuing this goal with great care and effort.

In this paper, we start by observing that the simple knowledge distillation could convert the open-world knowledge in the large pre-trained vision-language grounding model for the specialized OWOD task and propose a simple framework with surprisingly good performance. Meanwhile, we propose the down-weight training loss function for the detector’s mixed learning process of known ground truth, distilled unknown knowledge, and the pseudo unknown knowledge in OWOD algorithm to mitigate the effect of distilled knowledge on the detection performance of known objects. Besides, the cascade decoupled detection transformer structure is proposed to alleviate the influence caused by unknown objects on detecting known objects. Last but not least, we propose two novel benchmarks to comprehensively evaluate the ability of the open-world detectors to detect unknown open-world objects, named StandardSet$\heartsuit$ and IntensiveSet$\spadesuit$ respectively, based on the complexity of their testing scenarios. Our extensive experiments on existing and proposed benchmarks demonstrate the effectiveness of our framework. Our model exceeds the distilled large pre-trained vision-language grounding model for OWOD and state-of-the-art methods for OWOD and IOD.

This work is supported by National Natural Science Foundation of China (grant No.61871106 and No.61370152), Key R&D projects of Liaoning Province, China (grant No.2020JH2/10100029), and the Open Project Program Foundation of the Key Laboratory of Opto-Electronics Information Processing, Chinese Academy of Sciences (OEIP-O-202002).