LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in Vision-Language Models

To address the degradation of complex semantic information in VLMs caused by the processing of images as set of patches, we introduce the SGE module into the VLM, as shown in Fig. 2(b). Using LLaVA [18] as the base framework, the image is input into the pretrained vision encoder, and the obtained visual features are used as the input of SGE module, that is, $F_{v}$. Utilizing the pretrained visual perception module, we obtain the semantic segmentation masks of the input image. The input text is tokenized, and the embeddings are used as the prompt feature for SGE module, denoted as $F_{t}$. Together, $F_{v}$, $M$, and $F_{t}$ serve as inputs to the SGE module, constructing the scene graph $G^{\prime}$ activated by the prompt feature. Similar to the projection layer $W_{v}$ after the pretrained vision encoder, we apply a trainable projection layer $W_{g}$ to convert the nodes in $G^{\prime}$ into language embedding tokens. With the same dimensionality, the converted scene graph tokens are then fed into the Large Language Model, along with the visual tokens and the text tokens. We add two special tokens to the LLM, namely <sg> and <text>, which are inserted before and after the scene graph tokens, respectively. Consequently, the sequence of these tokens is fed into the LLM to generate language response tokens as follows:

where $H_{a}$ denotes the output of the LLM, serving as the response to the input image and text.

Following LLaVA [5], we train our LLaVA-SG on the prediction tokens of the LLM using an auto-regressive training objective. Building on LLaVA’s original two-stage training, we insert an additional training stage specifically for our SGE module. The training of LLaVA-SG thus includes the following three stages:

Visual Feature Alignment. We initialize the image encoder and the LLM of LLaVA-SG with pretrained weights and keep them frozen. We do not use the SGE module at this stage. Only the visual projection layer $W_{v}$ is trainable. The dataset of 558K LAION-CCSBU image-text pairs[21, 5] is adopted for training in this stage.

SGE training. In the second stage, we focus training the SGE module and the corresponding projection layer $W_{g}$. The visual encoder, visual projection layer, and LLM are all frozen during this stage. Considering that a scene graph contains both entities and the relationships between them, we endow the SGE module with the ability to preserve and express visual semantic information from both perspectives. For entity recognition, we use the pretrained visual perception module. For expressing the relationships between entities, we train the SGE module and the projection layer $W_{g}$ on visual relationship understanding datasets. Specifically, the visual relationship understanding datasets we use are derived from two sources. First, existing fine-grained visual understanding datasets, such as Visual Genome [22] and Open Image V6 [23], are reformatted into the visual question-answering format. However, the relationships in these datasets are limited and not open-vocabulary. Therefore, we also construct an open-vocabulary visual relationship understanding dataset based on the large-scale visual grounding dataset GRIT [24], utilizing GPT-4v to produce the data.

Fine-tuning. In the third stage, only the visual encoder remains frozen, while all other parameters are trained in LLaVA-SG. We use the 665K image-text instruction dataset from LLaVA-1.5 [18], which contains diverse instructions for fine-tuning in this stage.

In LLaVA-SG, we adopt CLIP ViT-L/14@336p [4, 25] as the vision encoder and Vicuna [26] as the LLM. The pretrained visual perception module includes RAM [27] for image tagging, Grounding-DINO [28] for detection, and SAM [29] for semantic segmentation. The message passing and attention mechanism in the SGE module are implemented using lightweight transformers. In the first stage, the learning rate for the trainable parameters is set to $2e-3$. In the second and third stages, we use a learning rate of $2e-5$.

The comparison results of LLaVA-SG and baseline models are summarized in Table I following the evaluation metrics of [30]. The reported results of the compared models are collected from the corresponding papers. Analyzing the experimental results in Table I, it is evident that our LLaVA-SG model shows significant and consistent improvements over the baseline models. The improvements of LLaVA-SG-7B over LLaVA-1.5-7B highlight that our SGE module effectively preserves and expresses visual semantic information with a nearly negligible increase in parameters, achieving significant performance gains. With Vicuna-13B, the LLaVA-SG-13B achieves better results than LLaVA-SG-7B.

The detailed results on MMBench are presented in Table I. MMBench assesses large vision-language models across multiple capability dimensions including LR for Logical Reasoning, AR for Attribute Reasoning, RR for Relation Reasoning, FP-S for Fine-grained Perception (Single Instance), FP-C for Fine-grained Perception (Cross Instance) and CP for Coarse Perception. Analyzing the comparative results in this table, our LLaVA-SG shows significant improvements over LLaVA-1.5 in the capabilities that require entity perception and relationship analysis, specifically in RR, FP-S, FP-C, and CP.

We present example outputs of LLaVA-1.5 and LLaVA-SG in Fig. 3. The middle column shows the masks of entities included in the scene graph of the SGE module. With the visual semantic information expressed by SGE, LLaVA-SG exhibits enhanced multimodal capabilities. For example, in the first case shown in Fig. 3, without explicitly preserving the visual semantic information in the image, the counting problem becomes difficult for LLaVA. Relying solely on the fragmented visual tokens output by ViT makes it challenging to accurately determine the number of dogs in the image. However, equipped with the SGE module, our LLaVA-SG can leverage tokens that explicitly represent entities in the image, enabling it to provide an accurate count of the dogs.

We perform ablation experiments to explore the effect of the SGE module and the effect of training strategies. These ablation experiments are based on LLaVA-SG-7B.

The Effect of the SGE Module. We conduct ablation studies on the components within the Scene Graph Expression module. The results are shown in Table II. ”SG” denotes the construction and use of a basic scene graph without message passing and prompt feature adaptation. ”MP” denotes the message passing among nodes in $G$. ”Prompt” denotes the attention mechanism with the prompt feature. ”Rel” indicates that the test sets from Visual Genome and Open Images are used for evaluating visual relationship classification. As shown in Table II, each component contributes to the improvement of the LLaVA baseline.

The Effect of Training Strategies. We performed ablation experiments on the additional visual relationship understanding data used for training the SGE module and the separate SGE training stage. The results are shown in Table III. The first row shows the results without using the SGE module, where the visual relationship understanding data used for the second training stage, i.e., the SGE-D in Table III is simply appended to the fine-tuning data for training the LLaVA model. The second row shows the results without a separate SGE training stage, where SGE-D is incorporated into the fine-tuning stage and SGE is trained together with the LLM. The third row shows the performance of LLaVA-SG under the full training strategy. The comparison between the first and third rows of Table III indicates that the improvement of LLaVA-SG over LLaVA is not primarily due to the additional visual relationship understanding data. Instead, the SGE module plays a crucial role. The comparison between the second and third rows highlights the necessity of the separate SGE training stage. The separate SGE training stage allows the SGE module to focus more specifically on expressing the visual semantic information within the input images.