Leveraging Retrieval-Augmented Tags
for Large Vision-Language Understanding
in Complex Scenes

Large vision-language models (LVLMs [5, 6]) have emerged as a significant advancement in multimodal AI, bridging the gap between visual understanding and natural language processing. These models aim to combine the strengths of large language models (LLMs) and vision transformers (ViTs) to tackle a variety of tasks, such as visual question answering, image captioning, and multimodal reasoning [7].

Recent works have explored various architectures and training paradigms to enhance the integration of visual and textual modalities. Some approaches utilize pretrained LLMs as the backbone, treating images as "foreign languages" by embedding visual inputs into tokenized representations [8, 9]. This method enables the LLM to process visual and textual information jointly, thereby achieving strong performance on vision-language tasks [10, 11, 12]. Other studies focus on scaling vision foundation models and aligning them with LLMs through advanced fine-tuning strategies, resulting in improved performance on diverse benchmarks [13]. Furthermore, retrieval-augmented frameworks have been proposed to incorporate external visual knowledge into LVLMs, providing more accurate and detailed context for multimodal reasoning [2, 14, 15].

In addition to architectural innovations, LVLMs have also been evaluated for their scalability and robustness. Research demonstrates that these models benefit significantly from large-scale multimodal datasets, which improve their generalization to unseen visual concepts and fine-grained object understanding [16, 17]. However, challenges remain, such as aligning modalities effectively and reducing hallucinations during generation. Techniques like preference fine-tuning and reinforcement learning have been introduced to address these issues, enhancing both accuracy and interpretability in complex visual tasks [18, 19].

Overall, LVLMs have shown remarkable progress in unifying vision and language understanding. These advances provide a solid foundation for developing more robust, efficient, and interpretable multimodal systems capable of reasoning over complex visual and textual data.

Object-aware knowledge retrieval is an emerging area that aims to improve multimodal reasoning and understanding by integrating object-level knowledge into vision-language models. This research addresses key challenges such as recognizing novel objects, mitigating hallucinations, and accurately capturing object attributes and relationships in complex visual scenes.

Recent studies have focused on enhancing the object-awareness of large language models (LLMs) and multimodal large language models (MLLMs). A prominent approach involves retrieval-augmented frameworks, where external object-level knowledge is dynamically retrieved and incorporated into the model’s reasoning pipeline [2, 20]. Such frameworks typically generate structured object tags, including attributes and relationships, that enrich the textual representation of the input image. This methodology has shown improvements in tasks like visual question answering and contextual image captioning.

Another line of research explores the integration of generative frameworks for multi-modal knowledge retrieval. These approaches treat LLMs as virtual knowledge bases, aligning visual features into the textual feature space of the LLM to facilitate object-aware reasoning [21]. Advanced techniques, such as prefix-tuning and meta-knowledge integration, have been proposed to guide multi-grained visual learning and improve the alignment of visual and textual modalities [20].

While significant progress has been made, challenges remain. One critical issue is the alignment of retrieved object-level knowledge with model-generated responses. Techniques such as fine-tuning with contrastive loss and preference alignment have been explored to address this limitation [2]. These advancements provide a robust foundation for further exploration into retrieval-augmented object-aware reasoning.

The VRAP framework processes an image-text pair $(x,q)$, where $x$ is the input image and $q$ is the textual query, to produce a response $y$. The workflow consists of three stages:

• •

  Visual Encoder: Extracts spatial and semantic features from the input image $x$.

• •

  Retrieval-Augmented Tag Generator: Produces a set of structured object tags $T$ from the visual features.

• •

  Generative LLM: Generates the final response $y$ based on the input query $q$ and the augmented tags $T$.

The pretrained visual encoder $\mathcal{E}_{v}$ maps the input image $x$ into a feature representation:

where $H$ and $W$ represent the spatial dimensions of the feature map, and $D$ is the dimensionality of each feature vector. These features encode the spatial and semantic information of the objects in the image.

To capture fine-grained object information, we use a scene graph parser $\mathcal{P}$ to extract objects $\{o_{i}\}$, their attributes $\{a_{i}\}$, and relationships $\{r_{k}\}$ from the visual features $f_{v}$. These elements are further enriched through a retrieval mechanism that incorporates external knowledge, forming a structured tag set $T$:

where $N$ is the number of detected objects, and $K$ is the number of relationships. The tags $T$ are then serialized into a textual format to serve as input to the LLM.

The augmented prompt $P$ integrates the input query $q$ with the retrieval-augmented tags $T$ in a structured manner:

The LLM $\mathcal{M}_{\text{LLM}}$ generates the response $y$ conditioned on the prompt $P$:

This design allows the LLM to leverage external object-level knowledge without modifying its architecture, relying solely on enriched textual prompts.

To train the VRAP framework, we employ a multitask learning objective comprising three loss functions: a generative loss for response alignment, a contrastive loss for tag relevance, and an auxiliary loss for tag generation.

The total loss combines the above objectives with balancing coefficients $\lambda_{\text{gen}}$, $\lambda_{\text{contrast}}$, $\lambda_{\text{tag}}$:

During inference, VRAP integrates the pre-generated tags $T$ with the input query $q$ to form the augmented prompt $P$. The LLM directly generates the response $y$ without requiring additional retrieval:

This streamlined inference strategy ensures efficiency while retaining the model’s robust object-aware reasoning capabilities.

The results of our comparison are presented in Table 1. VRAP achieves superior performance across all benchmarks, demonstrating its effectiveness in fine-grained reasoning tasks.

To assess the contributions of individual components, we conduct an ablation study by systematically disabling or modifying key modules in VRAP. Table 2 shows the results, highlighting the importance of retrieval-augmented tags and contrastive learning.

To complement the quantitative analysis, we conducted a human evaluation study. We randomly sampled 200 examples from the VQAv2 dataset and asked evaluators to compare the outputs of VRAP and the strongest baseline, ShareGPT4V. The evaluation focused on three aspects:

• •

  Accuracy: The factual correctness of the response.

• •

  Relevance: The contextual appropriateness of the answer.

• •

  Detail: The level of nuanced and specific information provided.

The results are summarized in Table 3. VRAP significantly outperformed ShareGPT4V across all criteria, with especially strong results in relevance and detail.

To gain a deeper understanding of the performance of VRAP, we analyze its behavior from multiple perspectives, including robustness to unseen objects, scalability to larger datasets, interpretability of generated tags, and computational efficiency. Each analysis provides insights into why VRAP outperforms other approaches.