Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

In this module, we first use a pre-trained MLLM model to generate general text descriptions corresponding to the images. These descriptions do not involve the category names used in the final classification. Next, the images and their corresponding text descriptions are fed into frozen image encoders (such as ViT [43] or ResNet [44]) and text encoders (such as BERT [45]), respectively, to generate image features and text features. Finally, the image features and text features are combined into multi-modal features through feature concatenation. Mathematically, suppose $X_{i}$ is the input image, and $\text{MLLM}(\ast)$ represents the MLLM model, the text description of image $X_{i}$ obtained through the MLLM model can be expressed as

Let $I(X_{i})$ and $T(X_{i}^{t})$ represent the series of tokens obtained from the frozen image encoder (ViT/ResNet) and text encoder (BERT), respectively. These can be expressed as

where each token is of dimension $E$. The image tokens encode information from each patch location, while the text tokens encode information from each word location. $V_{i}^{I}$ and $V_{i}^{T}$ are sets of tokens. Each element in $V_{i}^{I}$ represents the $i$-th token in the image, and each element in $V_{i}^{T}$ represents the $i$-th token in the text. $N$ denotes the number of tokens. All tokens form a series of text and image vertices:

where $\text{Concat}(\ast)$ represents feature concatenation, and $V_{i}$ represents a node after concatenating text features and image features.

To uncover the structural knowledge in the text embedding space for downstream tasks, i.e., the relationships between different semantics, and given the diverse visual features of different samples, we can measure finer-grained relationships between different semantics in the visual and textual space. Thus, we can construct a multi-modal graph structure $G=\{V_{i},E\}=\{V_{i}^{I},V_{i}^{T};E\}$, where $V_{i}^{I}$ and $V_{i}^{T}$ can be seen as the sets of image vertices and text vertices, respectively. $E$ represents the set of edges.

We build the graph using the similarity of multi-modal features via a predefined threshold $\gamma$. When the similarity between two multi-modal features is greater than $\gamma$, an undirected edge is created between these two vertices, representing the adjacency relationships between all multi-modal features:

where

represents the similarity between multi-modal nodes $V_{i}$ and $V_{j}$. $\gamma$ is the similarity threshold, and an edge is constructed when the similarity between two vertices in the graph exceeds this threshold.

We propose a parameter-efficient fine-tuning method based on graph networks called Graph Adapter Net (GA-Net), where the rest of the network is frozen and only the GCN [46] is fine-tuned during downstream tasks. The advantage of this method is that it can adapt to downstream tasks and improve model performance without significantly increasing the number of model parameters. Furthermore, since adapter fine-tuning only requires training a small number of parameters, it can significantly reduce computational resources and time costs for fine-tuning. The unique aspect of GA-Net is its ability to update features based on our constructed multi-modal graph structure, allowing fine-tuning while preserving adjacency relationships. The process of GA-Net can be represented by the following formula:

where $C_{x}$ represents the vertex feature matrix. The GA-Net utilizes a bottleneck architecture, including a down-projection $W_{\text{down}}:\mathbb{R}^{n_{\text{in}}}\rightarrow\mathbb{R}^{n_{\text{mid}}}$, a Graph Convolutional Network (GCN) layer, and an up-projection $W_{\text{up}}:\mathbb{R}^{n_{\text{mid}}}\rightarrow\mathbb{R}^{n_{\text{out}}}$. For vertex feature aggregation, we utilize the GCN, and the update formula for the vertex features in each layer can be formulated as:

where $\hat{A}$ is the normalized adjacency matrix, defined as $\hat{A}=A+I$, where $A$ is the original adjacency matrix and $I$ is the identity matrix. $\hat{D}$ is the degree matrix of $\hat{A}$, with diagonal elements $\hat{D}_{ii}=\sum_{j}\hat{A}_{ij}$. $W^{(l)}$ is the trainable weight matrix of the $l$-th layer, with dimensions $F_{l}\times F_{l+1}$. $\sigma$ is the activation function, applied element-wise to the matrix. $C^{(l)}$ represents the node feature matrix of the $l$-th layer of the GCN, with dimensions $N\times F_{l}$, where $N$ is the number of nodes and $F_{l}$ is the feature dimension of the $l$-th layer. $C^{(l+1)}$ represents the node feature matrix of the $(l+1)$-th layer of the GCN, with dimensions $N\times F_{l+1}$, where $F_{l+1}$ is the feature dimension of the $(l+1)$-th layer.

In the prediction stage, we introduce Elastic Weight Consolidation (EWC) regularization [12] into the generic cross-entropy loss function. By incorporating the importance of parameters and their association with the loss function, the EWC algorithm can effectively retain knowledge from previous tasks and reduce interference when learning new tasks, thereby addressing the problem of catastrophic forgetting in task learning.

First, we calculate the importance of parameters based on the training results of previous tasks, using the Fisher Information Matrix $I_{A}$. The Fisher Information Matrix is calculated as follows:

where $I_{A}$ is the Fisher Information Matrix representing the importance of parameters; $\mathbb{E}$ is the expectation operator; $\mathcal{L}(\theta)$ is the cross-entropy loss function, representing the model’s CE loss under parameters $\theta$, with $L_{CE}=\mathcal{L}(\theta)$; $\theta$ are the model parameters; $\theta^{*}$ are the optimal parameters obtained from previous task training; $\frac{\partial^{2}\mathcal{L}(\theta)}{\partial\theta^{2}}$ is the second derivative of the loss function with respect to the parameters, indicating curvature information; $\frac{\partial\mathcal{L}(\theta)}{\partial\theta}$ is the first derivative of the loss function with respect to the parameters, indicating gradient information.

The EWC loss is defined as:

where $\mathcal{L}_{\mathcal{B}}(\theta)$ is the loss function for the current task $B$; $\lambda$ is a hyperparameter that balances the learning of new tasks and the retention of old tasks; $I_{A}(i)$ are the diagonal elements of the Fisher Information Matrix, representing the importance of parameter $\theta_{i}$ in task $A$; $\theta_{i}^{*}$ are the parameters learned in task $A$. The model, regularized by EWC, is used for the final prediction, improving performance on new tasks while mitigating the forgetting problem.

The total loss is the sum of the ordinary cross-entropy loss $L_{CE}$ and the EWC loss $L_{ewc}$:

We validated our model on three downstream classification tasks: Oxford Pets [47], Flowers102 [48], and Food101 [49]. All these datasets belong to the fine-grained classification category. The Oxford Pets dataset contains 37 categories (25 dog breeds and 12 cat breeds) with a total of 7,349 images. The Flowers102 dataset includes 102 categories with a total of 8,189 images. The Food101 dataset consists of 101 food categories with a total of 101,000 images. These datasets are not only rich in categories but also possess high fine-grained characteristics, making them ideal for evaluating the model’s performance in distinguishing similar categories.

We compared the proposed model with several state-of-the-art parameter-efficient fine-tuning methods, including TaskRes [8], CoOp [7], CLIP-adapter [27], and Tip-Adapter [28], on the Oxford Pets, Flowers102, and Food101 datasets. As shown in Table 1, the experimental results demonstrate that our model consistently outperforms previous parameter-efficient fine-tuning models on the average performance of the benchmark datasets. Our model achieved an average performance of 90.03%, outperforming Tip-Adapter by 3.19% and TaskRes by 2.82%. The model’s test accuracy improved by 4.45% and 2.92% on the Oxford Pets and Flowers102 datasets, respectively, compared to the state-of-the-art methods. Our model still performed the best on the Food101 dataset. The more significant improvement on the Oxford Pets and Food101 datasets is due to the higher need for multi-modal associations, which are better modeled through GNN. Similarly, Tip-Adapter performs better than other SOTA methods as it combines the strengths of prompts and adapters, introducing task-related prompts into the model to provide more multi-modal associations.

As shown in Table 2, we conducted experiments on the parameter quantities of different methods. The parameter consumption of Tip-Adapter is exceptionally high and is not in the same range as other methods. Among the remaining three models, TaskRes has a higher parameter count than CLIP-Adapter, with CoOp having the least parameter count. Our model has a parameter count of 0.056M, which is less than most models and only slightly higher than CoOp. However, the accuracy of our model significantly surpasses that of CoOp, indicating that our model achieves a good balance between accuracy and parameter count.

When the image encoder and text encoder are not frozen, the parameter size is 195,337,765. When the image encoder and text encoder are frozen, the parameter size is 56,869. Therefore, with only 0.029% of the parameters being trainable, our model can surpass the state-of-the-art (SOTA) models on multiple datasets. Our model ranks second in terms of model efficiency among SOTA models. This is partly because the text descriptions in the CoOp model are generated using fixed handcrafted sentences, which are relatively simple text descriptions. Compared to our model, even though the CoOp model has fewer training parameters, the quality of the textual information is lower, resulting in significantly lower accuracy on two datasets.

As shown in Table 3, the baseline is a simple linear layer trained on single-modal features, which are also extracted using a pre-trained model. Without using our foundational modules, the accuracy is only 86.20%. After applying EWC regularization, the accuracy improves by 1.65% to 87.85%. When using multi-modal learning, the performance further increases by 2.23% to 90.08%. The improvement with multi-modal learning is because leveraging two modalities simultaneously provides greater capability compared to using a single modality. Finally, by integrating the complete graph method, the accuracy improves by 2.55% to 92.23%. The performance boost from incorporating the graph is due to its ability to better model the relationships between tokens, thereby demonstrating the effectiveness of our proposed method.

The text descriptions for each image are generated by MLLM. In the baseline experiments, replacing all text descriptions uniformly with ”A photo of a pet/flower” eliminates text information during training. Ablation experiments show that different combinations of methods and features enhance the model’s performance to varying degrees. The GA-Net model achieves the best performance by combining EWC[12], multi-modal features, and GNN. Introducing EWC regularization improves model performance by 1.65%; introducing GNN enhances performance by 2.55%; and incorporating multi-modal learning boosts performance by 2.23%. The significant performance improvements from multi-modal learning and GNN indicate that our model can effectively capture complex associations between modalities. EWC regularization helps mitigating the forgetting problem in task learning, making its effect more pronounced with larger datasets.

In terms of memory cost, Tip-Adapter has the lowest memory consumption, while CoOp has the highest. This is contrary to the parameter usage, indicating that, to some extent, a greater number of parameters can lead to lower memory consumption. Although our model also incurs a significant memory cost, it is still less than CoOp and performs better. While Tip-Adapter consumes less memory than our model, our model requires far fewer parameters and delivers significantly better performance. Furthermore, our model needs to store adjacency relations between different tokens, which inevitably consumes some memory space. This aspect contributes to the performance of our model. This demonstrates that our model achieves a good balance between accuracy and parameter consumption.

To investigate the impact of hyperparameters, specifically similarity thresholds, we analyzed different similarity thresholds on the OxfordPets and Flowers102 datasets, as shown in Figure 3. The results indicate that different datasets are affected differently by varying similarity thresholds. For both datasets, the accuracy increased most significantly within the threshold range of 0.3 to 0.5. Additionally, the accuracy improvement for the Flowers102 dataset was more pronounced within this range compared to the OxfordPets dataset, suggesting that the Flowers102 dataset is more sensitive to adjacency relations in the graph. When the similarity threshold reaches 0.7, the model achieves peak accuracy on both datasets. For thresholds less than 0.7, accuracy shows an increasing trend with rising similarity thresholds. However, once the threshold exceeds 0.7, accuracy changes minimally, but the training burden increases, indicating that a similarity threshold of 0.7 is optimal.