Object Retrieval for Visual Question Answering with Outside Knowledge

page\cref@result Early approaches frequently employed for object matching is to generate visual representations from the activations of convolutional layers [32] and transformer blocks [33]. These studies extensively utilized deep metric learning methods, which aims to learn discriminative features from images by minimizing the distance between samples of the same class and maximizing the distance between samples of different classes. In early deep metric learning methods, contrastive loss [34] was effectively employed to optimize the pairwise distances between samples. To explore more intricate relationships between samples, various metric loss functions, including triplet loss [35] and lifted structured loss [36], have been developed. These methods rely on supervised learning using image labels. However, in real-world object retrieval scenarios, objects are often unknown and lack labels. To learn representations of unlabeled objects, several techniques have been proposed. He et al. [37] introduced the momentum contrast (MoCo) method for unsupervised visual representation learning. Chen et al. [38] presented the simCLR framework, which is based on contrastive learning, for effective unsupervised visual representation learning.

In an effort to generate more robust pseudo-labels for unsupervised deep metric learning, Nguyen et al. [39] introduced a deep clustering loss to learn centroids. More recently, Kan et al. [40] proposed a relative order analysis (ROA) and optimization method for enhancing the relative ranking of examples in unsupervised deep metric learning. Li et al. [41] developed spatial assembly networks (SAN) to facilitate both supervised and unsupervised deep metric learning. Kim et al. [19] introduced an effective teacher-student learning pipeline for embedding learning of unlabeled images.

The methods mentioned above have primarily been investigated using evenly distributed class examples with images of the same size. Their performance may deteriorate when applied to VQA. Our work aims to develop a general object embedding method that is more representative of the scenario of VQA, setting it apart from the aforementioned works.

With the rise of large language models (LLMs) and retrieval-augmented generation (RAG), VQA with external knowledge has become increasingly popular [5, 6]. RAG addresses limitations of LLMs, particularly their tendency to produce ‘hallucinations’ when handling queries outside their training data or requiring real-time information. Prior RAG-based approaches have focused on retrieving relevant text descriptions [1] or identifying objects within large images to localize answer-relevant areas [3]. RAG methods are generally categorized into query-based, latent representation-based, logit-based, and speculative approaches [5]. In query-based RAG, user queries are enhanced with retrieved information, which is then fed into the generator’s input—exemplified by RetrieveGAN [42], which integrates image patches and bounding boxes to improve relevance and precision. Latent representation-based RAG leverages cross-attention to fuse retrieved objects as latent features [43]. Logit-based RAG introduces retrieval data at the decoding stage [44], while speculative RAG optimizes response time by selectively combining retrieval with generation. Our object retrieval method for VQA falls under query-based RAG, where we focus on enhancing retrieval accuracy to improve VQA performance.

fig:framework offers an overview of the MS-GCEL method proposed for object retrieval. Our primary objective, as shown in the object retrieval pipeline, is to train a universal object embedding network capable of representing any object. This network enables the extraction of embeddings for both query and gallery objects. Subsequently, we compute the Euclidean distances between the query embedding and the gallery embeddings. These distances are used for ranking, resulting in the retrieval of the top-k objects, which are then located within their respective images. Finally, these retrieved images are used to assist VQA for LVLM. Our method includes a potential object extraction network for object extraction and an MS-GCEL network for object embedding learning. The potential object extraction network is the SAM [15]. The MS-GCEL network can be configured using popular CNN-based or Transformer-based networks. In our experiments, we explored various backbone networks. In the literature, the teacher-student training paradigm has been demonstrated as effective for embedding learning [19]. Consequently, we have also implemented the teacher-student training pipeline to train the MS-GCEL network. In this approach, the teacher network $\phi^{t}$ maintains the same architecture as the student network $\phi^{s}$, and its weights are updated with momentum using the exponentially moving average weights of the student network. The details of each part are described in the following sections.

fig:ms-gcel portrays the intricate pipeline of our MS-GCEL. As elaborated upon in \crefsec:motivation, the effectiveness of embedding learning is closely associated with the accuracy of labeling, which, in turn, is linked to the scale of the image. Generally, images with larger scales tend to yield higher labeling accuracy, while those with smaller scales may result in lower labeling accuracy.

Building upon the preceding analysis, we propose categorizing images according to their sizes. As depicted in the initial step of \creffig:ms-gcel, the data distribution exhibits a long-tailed nature. The horizontal axis represents the image sizes, arranged from small to large, while the vertical axis denotes the number of objects. To uniformly sample data across different scales, we segment this data into multiple groups with uniform distributions (e.g., 4), spanning from small to large based on image area.

Then, we explore the use of k-nearest neighbors (K-NN) to sample data from each group, forming a batch of images, as illustrated in the second step of \creffig:ms-gcel. Subsequently, we compute the collaborative knowledge distillation loss based on the grouped embeddings. Specifically, for the $m$-th group, we denote the student output embedding vectors generated by $l^{s}_{m}$ as $\mathcal{F}^{l^{s}_{m}}$. To initiate the process, we employ the k-means clustering algorithm to cluster all the training embeddings, resulting in a set of clustering centroids, denoted as $\mathcal{C}$. We then compute the normalized similarity matrix $\bm{S}_{m}$ between embeddings in group $m$ and cluster centroids in $\mathcal{C}$. For collaborative knowledge distillation (CKD) between group $a$ and group $b$, we utilize the Kullback-Leibler divergence to calculate their CKD loss, which is defined as follows

In the formula, $G_{a}$ and $G_{b}$ represent the object image sets in group $a$ and $b$, while $G_{a\cap b}$ denotes the set of identical object images found in both group $a$ and $b$. The term $|G_{a\cap b}|$ corresponds to the number of images in $G_{a\cap b}$. $\bm{S}_{a}^{i}$ represents the indexed similarity vector between the embeddings of object $o_{i}$ and $\mathcal{C}$ within the matrix $\bm{S}_{a}$. The function of Kullback-Leibler divergence $\mathcal{L}_{\text{KL}}(\bm{S}^{i}_{a},\bm{S}^{i}_{b})$ is defined as follows

where $L=|\mathcal{C}|$, which is the total similarities between the embeddings of object $o_{i}$, and $\mathcal{C}$. Finally, the CKD loss in all groups is computed as follows

where $|Q|$ is calculated as $|Q|=\frac{{k(k-1)}}{2}$, with $k$ representing the number of groups.

To train the network, we utilize a combination of knowledge distillation and contrastive loss functions. The primary objective of the knowledge distillation loss is to extract more informative features from the more dependable embeddings. Meanwhile, the contrastive loss function serves as a metric loss, with the aim of learning a Mahalanobis distance metric. They were defined according to the following definitions. First, we define the loss functions $\mathcal{L}^{\text{self}}_{m}$ and $\mathcal{L}^{\text{con}}_{m}$ following the works of STML [19]. Let the student output embedding vectors generated by $h^{s}_{m}$ and $l^{s}_{m}$ in the $m$-th group be denoted as $\mathcal{F}^{h^{s}_{m}}$ and $\mathcal{F}^{l^{s}_{m}}$. The self-distillation loss $\mathcal{L}^{\text{self}}_{m}$ is computed based on Kullback-Leibler divergence, as follows:

where $d_{ij}^{l^{s}_{m}}:=||\bm{f}_{i}^{l^{s}_{m}}-\bm{f}_{j}^{l^{s}_{m}}||_{2}$ / $(\frac{1}{n}\sum_{k=1}^{n}||\bm{f}_{i}^{l^{s}_{m}}-\bm{f}_{k}^{l^{s}_{m}}||_{2}$) represents the relative distance between $\bm{f}_{i}^{l^{s}_{m}}$ and $\bm{f}_{j}^{l^{s}_{m}}$. Similarly, $d_{ij}^{h^{s}_{m}}$ represents the relative distance between $\bm{f}_{i}^{h^{s}_{m}}$ and $\bm{f}_{j}^{h^{s}_{m}}$. The function $\psi(\cdot)$ corresponds to the softmax operation, and $n$ represents the number of samples in the $m$-th group. The relaxed contrastive loss $\mathcal{L}^{\text{con}}_{m}$ is defined as follows

where $w_{ij}=exp(-\frac{||\bm{f}_{i}^{t}-\bm{f}_{j}^{t}||_{2}^{2}}{\sigma})$ represents a pairwise similarity score between teacher embeddings $\bm{f}_{i}^{t}$ and $\bm{f}_{j}^{t}$, and $\delta$ is a margin. It’s worth noting that the same training approach is applied to $l^{s}_{m}$ as well. The overall loss function $\mathcal{L}_{\text{MS-GCEL}}$ is defined as follows

After training the network, the OR-OK-VQA task is performed in a human–computer interaction manner. First, gallery image features are extracted and indexed using the learned network with the MS-GCEL loss function. Given an input image and question, we crop a question-relevant object image and extract its features using the trained network. These object features are then used to retrieve relevant images from the gallery. Finally, the retrieved images and the question are fed into the large vision-language model (LVLM) to assist with the VQA task.

Our training dataset is curated from pre-existing open-access object detection datasets, i.e., COCO 2017 [28] and VOC 2007 [29]. The testing dataset consists of publicly available object retrieval and object detection datasets, which serve to evaluate the performance of general object retrieval. We also established an OK-VQA evaluation benchmark using images from the BelgaLogos [7], Visual Genome [30], and LVIS [31] datasets. The details of our experimental datasets are summarized in  \creftab:dataset_statistics. In the following, we provide a simple description of the datasets.

General object retrieval dataset. The training and validation sets are constructed by extracting objects from the training sets of COCO 2017 and VOC 2007, as well as the validation set of VOC 2007. The training set comprises 121,298 images, while the validation set contains 2,000 images. The test set is composed of objects extracted from the validation set of COCO 2017 and the test set of VOC 2007, totaling 9,904 images. To perform object extraction, we employed SAM [15]. The resulting partitioned sets consist of more than 8 million objects for the training set, 173K for the validation set, and 862K for the test set. In the zero-shot object retrieval setup, we partition the training set into four distinct groups, with each group comprising 20 object classes. These four group datasets are employed to create four tasks, with each task progressively introducing an additional set of 20 new classes. The classes utilized for training (known classes highlighted in light yellow) and testing (unknown classes highlighted in light blue) in each task are presented in Table 9. In Task 1, the training set consists of the most popular 20 classes, while the remaining 60 classes are used for testing. For Task 2, 20 new classes are introduced along with the original Task 1 classes for training, and the remaining 40 classes are designated for testing. Task 3 involves the addition of another set of 20 new classes to Task 2, with the remaining 20 classes used for testing. Finally, Task 4 incorporates all 80 classes for both training and testing phases.

Generalization evaluation dataset. To assess the generalization ability of the trained MS-GCEL model, we conducted evaluations on the BelgaLogos [7], Visual Genome [30], and LVIS [31] datasets. The BelgaLogos dataset consists of 10K images and 55 query logos, primarily used to evaluate the logo retrieval performance of an algorithm. The provided 55 query logo images of the BelgaLogos dataset are utilized as the query set, while the gallery set comprises 939,361 objects extracted using SAM[15]. In the case of the Visual Genome dataset, which contains 108K images and 2,516,939 labeled bounding boxes, we define a set of 79,921 objects as the query set and directly performed object retrieval using the trained model on the whole dataset. As for the LVIS dataset, it shares the same images as the COCO 2017 dataset but features 1000 class labels. We compute evaluation scores based on the labeled objects in the validation set of the LVIS dataset.

OK-VQA evaluation dataset. To evaluate the performance of OK-VQA, we constructed a specialized VQA dataset by selecting images from the BelgaLogos, Visual Genome, and LVIS datasets. A total of 36 questions are crafted for these images, with most answers requiring external knowledge beyond the information directly available in the images. To correctly answer these questions, relevant images must be retrieved from external datasets to provide additional contextual information.

Evaluation Metrics. We use Recall@1 and mean average precision (mAP) as the evaluation metrics for object retrieval. The Recall@1 is defined as follows: Recall@1 = $\frac{1}{q}\sum_{i=1}^{q}r_{i}$, where $q$ represents the number of queries, and $r_{i}$ is equal to 1 if the top-1 returned objects contain the ground truth object, and 0 otherwise. The mAP is calculated as: mAP = $\frac{1}{q}\sum_{i=1}^{q}\sum_{j=1}\frac{j}{p_{j}}$, where $q$ represents the number of queries, and $p_{j}$ is the ranking index of a returned object that matches the ground truth. We calculate image-level and object-level retrieval scores. The image-level score is computed using an IoU threshold of $1e^{-10}$, and the object-level score is computed using an IoU threshold of 0.3.

Implementation Details. In our experiments, we utilize SAM [15] and MViT [48] for object extraction. Specifically, the ViT-H backbone network was utilized for object extraction based on SAM. For object extraction based on MViT, the text prompts ’all objects,’ ’all entities,’ ’all visible entities and objects,’ and ’all obscure entities and objects’ were employed. Our teacher-student network is trained on two RTX 3090 (24GB) GPUs with a batch size of 120. The nearest neighbors are set to 5. Each batch contains images of various sizes for MS-GCEL learning. The group number of the MS-GCEL is set to 4, and the clustering number is set to 100 unless specified otherwise. The dimension of teacher embeddings $f^{t}$ was set to 1024, while the dimensions of student embeddings $f^{l^{s}}$ and $f^{h^{s}}$ were both set to 512. The $\sigma$ to calculate $w_{ij}$ and the margin $\delta$ are 3 and 1 respectively. The initial learning rate is set as $10^{-4}$ on the backbone of GoogLeNet, and $3\times 10^{-5}$ for MiT-B2 and ViT-B/16, both of which are scaled down by the cosine decay function[50]. During training, The Adamp[51] optimizer was employed with a weight decay set to 0.01. and the K-nearest neighbor sampling distance matrix was updated once every 1000 iterations.

To assess the effectiveness of the MS-GCEL method, we conducted experiments using different backbone networks and various initialization methods. The results on the curated test set are presented in \creftab:effectiveness. In this table, methods based on the GoogLeNet and MiT-B2 backbone networks are initialized with parameters trained on the ImageNet-1K [20] dataset. Our method based on the ViT-B/16 [52] backbone network are initialized using parameters trained by the MoCo-V3 [17] and CLIP [53] methods, respectively, and then fine-tuned on our curated training dataset using the proposed MS-GCEL method.

From \creftab:effectiveness, it is evident that our method outperforms the state-of-the-art unsupervised STML [19] method, achieving an improvement of 2.65% and 10.03% in image level mAP for GoogLeNet and MiT-B2 backbone networks, respectively. To further verify the effect of multi-scale group collaborative embedding learning (MS-GCEL), we present the results for different sizes of query objects, as shown in \creftab:coco_MS_results_google and \creftab:coco_MS_results_MiT. Specifically,  \creftab:coco_MS_results_google reports object retrieval results based on the GoogLeNet backbone, while \creftab:coco_MS_results_MiT presents results using the MiT backbone. We can see that the MS-GCEL obtained better performance in both the object- and image-level retrieval scores for different sizes of query objects. Moreover, the larger size of objects, the higher the performance for both the STML [19] and the MS-GCEL methods.

Moreover, we conduct evaluation experiments on the BelgaLogos [7], LVIS [31] and Visual Genom datasets using the trained model. Results are shown in \creftab:comparison. We have observed that our approach using GoogLeNet [54] and MiT-B2 [55] backbone networks effectively enhances retrieval performance across various datasets. The object-level mAP scores demonstrated a 6.69% improvement on the Visual Genome dataset and a 0.89% improvement on the LVIS dataset when using the MiT-B2 backbone network. The highest mAP scores and Recall@1 scores on the BelgaLogos, Visual Genome, and LVIS dataset are achieved with the parameters initialized by the MoCo-V3. However, on the Visual Genome dataset, the performance of the MS-GCEL method with MoCo-V3 slightly decreases due to the lower-dimensional feature embeddings (512 v.s. 1000).

The object retrieval results for OK-VQA on the curated evaluation dataset are shown in  \creftab:or-ok-vqa. We observe that enhancing the object retrieval model leads to improved VQA performance. \creffig:Results-case, \creffig:ok-vqa-1 and \creffig:ok-vqa-2 present VQA examples using GPT-4o, comparing results without and with the integration of STML and MS-GCEL methods. We can see that incorporating MS-GCEL results enables GPT-4o to answer the question accurately.

We conducted ablation studies on our curated object retrieval dataset using the GoogLeNet [54] backbone. In the subsequent ablation experiments, we utilize objects extracted by SAM for both training and validation.

(1) Impact of the MS-GCEL group number. We assessed the effect of the number of groups on the performance of the MS-GCEL method, with the cluster number fixed at 100. The results on the validation set are displayed in \creftab:ab-group, and they indicate that the best performance is attained when the number of groups is set to 4. Thus, in all of our experiments, we set the number of groups as 4.

(2) Impact of the clustering number. We examined the impact of the clustering number on deep feature embedding performance, with the number of groups fixed at 3. The results on the validation set are presented in \creftab:ab-cluster, and it is evident that the best performance is achieved when the number of clusters is set to 100.

(3) Impact of the MS-GCEL method. We assessed the influence of the PEL module and the CKD loss function in the proposed MS-GCEL approach on the object retrieval validation set. The ablation experiments are shown in \creftab:ab-gcml. The utilization of the PEL, which focuses on obtaining scale-specific information for feature encoding, enhances object-level retrieval performance. This indicates that the PEL assists in tackling the multi-scale challenge in object retrieval. The incorporation of the CKD loss function, which promotes information sharing and transfer among several PELs, significantly improves the model’s retrieval performance.

page\cref@result \creffig:compare_case.png illustrates the comparison of three object retrieval examples between the MS-GCEL and STML[19] methods. The first retrieval example is based on the GoogLeNet backbone network, while the second and third retrieval examples are based on the MiT-B2 backbone network. These examples demonstrate that the MS-GCEL method is capable of returning more relevant objects associated with the query object.

fig:retrieval-examples provides logo retrieval example showcasing the effectiveness of the proposed MS-GCEL method. These examples illustrate the ability to ground multiple objects within the retrieved images.

In \creffig:supplemental_more_case, we present six retrieval examples. These examples were retrieved using the fine-tuned model based on the MS-GCEL method with parameters initialized using MoCo-V3 [17]. During retrieval, the top five images containing the query object were retrieved. Subsequently, object localization was conducted in these retrieved images using a similarity score (feature distance).

fig:failure_case.png shows two failure cases, the first line of the query image in the \creffig:failure_case.png is antenna, but the search returns ski-pole and mast. This is because antenna, ski-pole and mast are essentially the same in shape, and distinguishing between them requires more contextual semantic information. The image retrieved in the second row is the BASE logo. However, the fourth image is not the BASE logo, but other text patterns.