Learning Modality Knowledge Alignment for Cross-Modality Transfer

We consider the knowledge transfer between source modality $\mathcal{M}^{s}$ and target modality $\mathcal{M}^{t}$. Data in source modality, such as vision or language, is easier and cheaper to obtain, and large pretrained models are publicly available. Instead, the target modality considered in this paper has insufficient data to pretrain its own large models. The two modalities differ in both input space and label space, i.e., $\mathcal{X}^{s}\neq\mathcal{X}^{t}$, $\mathcal{Y}^{s}\neq\mathcal{Y}^{t}$. Cross-modal transfer aims to leverage a source pretrained model, parameterized by $\boldsymbol{\theta}^{\mathcal{S}}$, to help a given target task with a small set of labeled data $\{\boldsymbol{x}_{i}^{t},y_{i}^{t}\}_{i=1}^{n_{t}}$.

Following previous works (Lu et al., 2022; Shen et al., 2023), our model architecture $g_{\boldsymbol{\theta}}$ includes an embedder $e(\cdot;\boldsymbol{\theta}_{e})$, a transformer encoder $f(\cdot;\boldsymbol{\theta}_{f})$ and a predictor $h(\cdot;\boldsymbol{\theta}_{h})$, and the parameter of the full model is denoted as $\boldsymbol{\theta}=\{\boldsymbol{\theta}_{e},\boldsymbol{\theta}_{f},\boldsymbol{\theta}_{h}\}$. Particularly, the pretrained transformer has its own embedder and predictor, and thus we denote the pretrained weights of the source model as $\boldsymbol{\theta}^{\mathcal{S}}_{0}=\{\boldsymbol{\theta}_{e_{0}}^{\mathcal{S}},\boldsymbol{\theta}_{f_{0}}^{\mathcal{S}},\boldsymbol{\theta}_{h_{0}}^{\mathcal{S}}\}$.

The embedder maps the input data into a shared input embedding space $\hat{\mathcal{X}}$, and the encoder extracts features from the embedded input. The predictor is a linear layer that maps the encoder output to the label space. For our target model $g_{\boldsymbol{\theta}^{\mathcal{T}}}:\boldsymbol{\theta}^{\mathcal{T}}=\{\boldsymbol{\theta}^{\mathcal{T}}_{e},\boldsymbol{\theta}^{\mathcal{T}}_{f},\boldsymbol{\theta}^{\mathcal{T}}_{h}\}$, both embedder and the predictor are specifically re-designed to accommodate the input and label space of the target task, while we use $\boldsymbol{\theta}_{f_{0}}^{\mathcal{S}}$ to initialize the encoder weight $\boldsymbol{\theta}^{\mathcal{T}}_{f}$.

The flexibility of such architecture enables end-to-end training on the target task. Vanilla finetuning simply updates all the parameters of the target model by minimizing the task-specific loss on the given training dataset:

where $\ell$ is the task loss function such as cross-entropy.

Learning directly from target supervision in this way encourages the model to learn knowledge that help discriminate the target data. As the pretrained model already contains source discriminative knowledge, it is natural for cross-modal transfer to expect that source and target knowledge share similarities in some aspects so that the source knowledge can be reused to promote target learning. In the following, we 1) conduct experiments to show that this similarity depends on the modality, and 2) provide interpretation of modality knowledge and formalize the knowledge discrepancy.

We look for a quantitative way to compare the extent of knowledge reuse among various cross-modal transfer scenarios. In this section, we select image modality as knowledge source and choose four target tasks from different modalities. We include two tasks closely related to images: CIFAR-100 (Krizhevsky et al., 2009), Spherical (Cohen et al., 2018) that contains spherically projected images, and two tasks dissimilar to image modality: NinaPro (Atzori et al., 2012) that represents hand gestures with electromyography signals, FSD50K (Fonseca et al., 2017b) that contains audio clips of sound events. To be specific, we adopt Swin Transformer Base pretrained on ImageNet-22k as the source model and examine the properties of the model after finetuning it on different tasks.

Given that the comparison is conducted across distinct modalities, there lacks a general metric measuring the degree of knowledge reuse during transfer. Therefore, we turn to compare the distortion of source knowledge. Specifically, we would expect smaller distortion if more source knowledge is reused to solve target task, and vise versa. So we leverage the pretrained source model to extract the visual representation of CIFAR-10, a surrogate image dataset unseen by the model. Samples in this particular source dataset are denoted as $\{\boldsymbol{x}^{s}_{i},y^{s}_{i}\}$ and their corresponding feature set $\{\boldsymbol{f}^{s}_{i}=f(e(\boldsymbol{x}^{s}_{i};\boldsymbol{\theta}_{e_{0}}^{\mathcal{S}});\boldsymbol{\theta}_{f_{0}}^{\mathcal{S}})\}$. Then, we finetune the pretrained model on the four target tasks using Eq. (1) respectively. After the finetuning process, we once again extract the representation of CIFAR-10 using the finetuned encoder and obtain $\{\boldsymbol{f}^{s}_{i}(\mathcal{M}_{t})=f(e(\boldsymbol{x}^{s}_{i};\boldsymbol{\theta}_{e_{0}}^{\mathcal{S}});\boldsymbol{\theta}_{f}^{\mathcal{T}},\mathcal{M}_{t})\}$. Fig. 2 shows the T-SNE visualization results of the five different sets of CIFAR-10 image features.

The figure illustrates that encoders finetuned on CIFAR-100 or Spherical maintain or even improve their discriminability on image samples in CIFAR-10, whereas the encoders finetuned on NinaPro and FSD50K can no longer extract class discriminative features for images. Considering that finetuning on target modality makes the encoder focus on classifying target data and learning target discriminative function, what this observation indicates is that the knowledge required for discriminating samples in CIFAR-100 and Spherical are more aligned with that for CIFAR-10, compared to the latter two modalities. Such conclusion aligns with our intuition, since CIFAR-100 is vision dataset and Spherical is originated from natural images, whereas NinaPro and FSD50K are less relevant to images.

On the flip side, the results shows that CIFAR-100 and Spherical can better reuse the source knowledge in the pretrained encoder for task solving while NinaPro and FSD50K require the encoder to make greater adjustments in order to adapt to target tasks.

To investigate the source knowledge reuse (or distortion) during cross-modal transfer more quantitatively, we use linear probing on CIFAR-10 to evaluate the quality of extracted representations with encoders finetuned 1) on different target modalities, 2) with different epochs, and 3) with different transfer methods. Additional to vanilla finetuning, we consider the following two baselines:

• •

  ORCA (Shen et al., 2023) adds an embedder training stage before finetuning. This first stage only updates the target embedder parameters $\boldsymbol{\theta}_{e}^{\mathcal{T}}$ to minimize the Optimal Transport Dataset Distance (Alvarez-Melis & Fusi, 2020) between source and target embeddings in the shared input space $\hat{\mathcal{X}}$.

• •

  We propose another baseline modified from previous works (Kumar et al., 2022; Zhang et al., 2023b), Embedder warmup (Emb), which is also a two-stage training method. The first stage solely updates the target embedder using the same task loss as vanilla finetuning while keeping the rest of the network frozen. The second stage finetunes the full network.

Fig. 3 shows the error rate of linear probing, where the dash line shows the linear probing results on pretrained encoder as a reference. Note that all these results are error rates on CIFAR-10 dataset that reflects the extent to which the model retains the source modality knowledge. The performance comparison on target modalities is not what we concern right now and can be found later in table. 2. From experiments we observe that modality has the greatest effect on linear probing results. Finetuning on FSD50K significantly distorts the encoder and impairs its discriminability on image data. We also notice that tuning on target dataset for more epochs leads to larger distortion of source knowledge on all target modalities except for image modality (CIFAR-100). These observation leads to the conclusion that knowledge for discriminating samples in different modalities differ in varying degrees, which we refer to as the misalignment of modality semantic knowledge. We argue that a large discrepancy may hinder the effectiveness of cross-modal transfer, and thus the assumption that source modality pretraining is beneficial to target modality should depends on the discrepancy.

We make additional observations about the source knowledge preservation effect of two-stage training methods. We observe that both ORCA and Emb achieves lower source error compared to vanilla finetuning, and Emb performs better than ORCA. This suggests that the target embedder trained in their first stage implicitly learns a mapping from $\mathcal{X}^{t}$ to $\hat{\mathcal{X}}$ that mitigates the knowledge misalignment between target and source, and thus reduces the model distortion during its adaptation towards target tasks.

The above experiments motivate us to formalize the discrepancy of modality semantic knowledge, and to propose an improved objective for training target embedders that reduce such discrepancy.

We consider representing the semantic knowledge within a modality using the conditional distribution $P(Y|X)$, which describes the relationship between raw data space and semantic space of the modality. This is because for neural networks, acquiring the semantic knowledge means learning a mapping from data space to semantic space that resembles the true conditional distribution.

However, to measure the degree of alignment or “similarity” of such knowledge between two modalities is quite challenging. The difficulty lies in the fact that both the data space $\mathcal{X}$ and the label space $\mathcal{Y}$ are different and even non-overlapping across modalities.

Therefore, we need to modify the conditional distribution to make it comparable across modalities. Modifying the input space is rather easy, as we can simply embed the inputs into a shared space $\hat{\mathcal{X}}$ using modality-specific embedders. However, modifying the label space is more sophisticated.

Considering that source modalities, like vision and language, having large pretrained models are both rich in semantics, we make the following assumption: The cardinality of source modality label space is larger than the cardinality of the label space of the target modality, i.e., $|\mathcal{Y}^{s}|\geq|\mathcal{Y}^{t}|$.

This assumption is easily satisfied in practice. For example, vision transformers trained on ImageNet learns a discriminative function of one thousand categories whereas only four classes are considered in an electrocardiogram classification task. With the assumption, we can select a subset of the source modality label space $\mathcal{Y}^{s}_{\mathcal{B}}\subset\mathcal{Y}^{s}$ such that $|\mathcal{Y}^{s}_{\mathcal{B}}|=|\mathcal{Y}^{t}|$. We further introduce a category permutation $\pi(\cdot)$ that adjusts the order of source classes. To this end, we can define a new label space of the source modality, namely the source subset after permutation $\mathcal{Y}^{s}_{\pi,\mathcal{B}}\triangleq\pi(\mathcal{Y}^{s}_{\mathcal{B}})$. By measuring the discrepancy between the modified conditional distributions $P(Y^{s}_{\pi,\mathcal{B}}|\hat{X})$ and $P(Y^{t}|\hat{X})$, we can formalize the degree of alignment of the modality semantic knowledge as follows:

The definition basically says that, if we can find an optimal subset within source semantics that, with a proper one-to-one matching between source semantics and target semantics, shares similar conditional distribution with target modality, then the knowledge discrepancy is considered to be small. The source model should be able to correctly distinguish target samples like the way it discriminates source samples within the subset.

With the definition, we calculate the modality semantic knowledge discrepancy between image modality and the four target tasks using an extreme approximating algorithm. Here we only demonstrate the results in Fig. 4 while leaving the implementation details in supplementary. Our calculation aligns with previous observations, showing that modalities do have different degree of knowledge discrepancy, and FSD50K is the most dissimilar modality from image modality among the four.

In the previous experiments we find that Embedder warmup, in spite of its simplicity in training objective, preserves source knowledge better than other methods. Correspondingly, we turn to examine its performance on target modalities. Table 2 shows that Emb likewise surpasses its counterparts. We argue that during the embedder warmup, in order to minimize the task loss, the embedder are forced explicitly to project target original inputs into embeddings that are distinguishable by the source model, which is frozen and extract features according to the source knowledge.

Combined with our previous analysis, we hypothesize that the key to effective transfer is to learn a target embedding function $e^{\mathcal{T}}:\mathcal{X}\to\hat{\mathcal{X}}$ that makes the target conditional distribution $P(Y^{t}|\hat{X})$ more aligned with the source knowledge. Consequently, we propose to train the target embedder solely using objective in the next section to learn such embedding function ahead of the full finetuning process.

Since we cannot estimate the target conditional probability without training a model, adopting the modality knowledge divergence directly as a objective for optimization is difficult. As an alternative, we propose to leverage meta learning pipeline to simulate the process in Fig. 3 and optimize the representation quality of source data after finetuning. Specifically, an ideal target embedder aligns the modality knowledge, allowing the encoder to retain its discriminability on image data during target finetuning. Therefore, if we use a source dataset to evaluate the finetuned encoder that is initialized by this ideal target embedder, we would obtain minimal error on the source data.

Such process is a standard bi-level optimization problem widely studied in meta-learning (Finn et al., 2017). Particularly in our case, the outer-loop updates the target embedder based on the outer-loop loss, which is computed using target encoder after inner-loop optimization. Fig. 5(a) illustrates a single step update of the embedder parameter $\boldsymbol{\phi}_{e}$ in the outer-loop during meta learning, and Fig. 5(b) shows the process of bi-level optimization.

More specifically, the inner-loop is the optimization of the model on target dataset, subjected to the condition that the target embedder is initialized by $\boldsymbol{\phi}_{e}$, which is

where $\mathcal{L}_{inner}$ is the same loss as in Eq. (1), and

This inner-loop optimization simulates the full finetuning process in the second stage, and returns an encoder that is already adapted to target modality. Note that the whole optimal target model in the inner-loop depends on the initialization of the target embedder. Therefore we have $\boldsymbol{\theta}^{\mathcal{T}^{*}}(\boldsymbol{\phi}_{e})=\{\boldsymbol{\theta}^{\mathcal{T}^{*}}_{e}(\boldsymbol{\phi}_{e}),\boldsymbol{\theta}^{\mathcal{T}^{*}}_{f}(\boldsymbol{\phi}_{e}),\boldsymbol{\theta}^{\mathcal{T}^{*}}_{h}(\boldsymbol{\phi}_{e})\}$.

The outer-loop is an optimization problem with respect to the target embedder. Our goal is to find optimal embedder parameters ${\phi}_{e}^{*}$ such that the resulting optimal target encoder $\boldsymbol{\theta}^{\mathcal{T}^{*}}_{f}(\boldsymbol{\phi}_{e}^{*})$ generates high quality representations of source data. To calculate the loss, we leverage a small labeled dataset $\{\boldsymbol{x}_{i}^{s},y_{i}^{s}\}$ in source modality as a surrogate and compute their features $\{\boldsymbol{f}^{s}_{i}=f(e(\boldsymbol{x}^{s}_{i};\boldsymbol{\theta}_{e_{0}}^{\mathcal{S}});\boldsymbol{\theta}_{f}^{\mathcal{T}^{*}}(\boldsymbol{\phi}_{e}))\}$. Then we normalize these features onto a unit sphere and measure the alignment and uniformity of the source features (Wang & Isola, 2020). In particular, the alignment loss measures whether features from the same class are close, and the uniformity loss measures whether features from different classes are evenly distributed on the sphere.

Our outer-loop objective that measures the source discriminability of the induced encoder takes the following form:

Notably, the source knowledge cannot be well-preserved at the beginning of the embedder training. To prevent the embedder from overly focusing on source modality, and also to keep the optimization process stable, we strike a balance between source and target knowledge learning by jointly minimizing the two objectives with a trade-off parameter $\lambda$:

In practice, we adopt single step update in the inner loop, a simplification that will be discussed in the analytical experiment section. This enables us to reuse the loss $\mathcal{L}_{inner}$ calculated during the inner-loop virtual update to compute this combined objective $\mathcal{L}_{outer}^{{}^{\prime}}$ efficiently. To this end, our proposed MoNA updates the target embedder in the first stage using:

With the modality knowledge being better aligned, MoNA conducts vanilla finetuning in the second stage. The complete algorithm of MoNA is demonstrated in Alg. 1.

Several experiments are conducted on this benchmark. The benchmark involves seven tasks with 2D inputs and three tasks with 1D inputs. Table 1 shows the results comparison using the full training set in each modality. We compare different kinds of baselines including training hand-designed and NAS-based architectures (NAS-Bench-360, DASH (Shen et al., 2022)) solely on target modalities, general purpose networks Perceiver IO (Jaegle et al., 2022), and cross-modal transfer methods FPT (Lu et al., 2022) and ORCA. We observe that MoNA achieves top performance on nine out of ten tasks, and surpasses previous cross-modal transfer methods on all tasks.

We also compares several variants of cross-modal transfer. In single stage methods, finetuning refers to the vanilla finetuning in Eq. (1). Finetuning++ initializes the target embedder using source embedder weights under certain modifications to map the dimension. Frozen encoder resembles to previous works (Zhang et al., 2023b) and it keeps the pretrained weight frozen during finetuning. In two-stage methods where a warmup stage is conducted before finetuning, we consider ORCA, Emb and MoNA. These experiments serves as an ablation study of our method and their results is reported in table 2.

The results lead to following conclusions: 1) finetuning the encoder helps the pretrained model adapt to target modality and performances better than frozen encoder. 2) Two-stage methods are generally superior than single stage methods, indicating that a proper target embedding function leads to better knowledge transfer. 3) Combined with the results on source knowledge preservation experiment in Fig. 3, we see that methods that better align modality knowledge achieves higher cross-modal transfer performance.

To show that MoNA achieves consistent performance improvement across different pretrained models, we conduct experiments on other two vision backbones, which are ViT-Base (Dosovitskiy et al., 2020) pretrained on ImageNet-22K and CLIP ViT-Base/16 (Radford et al., 2021) pretrained on WIT-400M. The results in table 3 show that, although the performance on target modalities varies due to the different capability of the pretrained models, MoNA consistently improves knowledge transfer and is superior to the State-of-the-Art method on different pretrained models.

The benchmark includes eight PDEs with 1D/2D inputs, which we address similarly using language and vision pretrained models. We consider three baselines in the original paper, namely U-Net (Ronneberger et al., 2015), PINN (Raissi et al., 2019), FNO (Li et al., 2020), where the last two methods are specifically designed for PDEs. We also compare ORCA as the cross-modal transfer baseline. The result is shown in table 4.

We observe that MoNA achieves state-of-the-art on four out of eight tasks on PDEBench. Notably, it outperforms ORCA on seven tasks with significant improvements on Advection Equation and Navier-Stokes’ Equation, and it achieves competitive results with specialized method FNO.

To further demonstrate the scalability of MoNA, we conduct experiments on more datasets and modalities, including AudioSet (Gemmeke et al., 2017) and ESC50 in audio modality, and UCF101 (Soomro et al., 2012) in video modality.

As shown in table 5, MoNA surpasses all baselines including ORCA. These results validate the scalability of our method, indicating that MoNA is a general cross-modality transfer method that can be applied on a wide range of tasks.

Ablation Studies. To validate the effectiveness of our design, we conduct several ablation studies on four tasks in NAS-Bench-360. The first ablation study investigates the effect of different training objectives as shown in table 6. We begin with excluding the inner-loop loss $\mathcal{L}_{inner}$ from the total objective of the outer-loop in Eq. (5). Comparing to MoNA, the performance drop on all four tasks shows that $\mathcal{L}_{inner}$ is crucial to ensure the adaptation of the embedder.

We move on to replace the alignment and uniformity loss (i.e., $\mathcal{L}_{outer}$) by two variants. The contrastive loss refers to the supervised contrastive loss (Khosla et al., 2020), and the clustering metric we adopt is the Davies-Bouldin index. The results in table 6 shows that contrastive loss achieves slightly worse performance than MoNA, while the clustering metric leads to much worse results. We hypothesize the reason is that the DB index becomes less informative when the feature dimension is high.

The ablations on different training strategies can be found in table 2. Comparing two-stage MoNA against one-stage finetuning method, we find that the knowledge alignment stage brings significant improvement.

MoNA reduces modality knowledge misalignment. We empirically validate that MoNA reduces the discrepancy between source modality and four target tasks (Fig. 4), and by which better retains source knowledge during finetuning than other methods (Fig. 3). Our empirical results align with our hypothesis that reducing knowledge misalignment between modalities leads to more effective transfer and higher performance on target tasks.

Parameter sensitivity of $\lambda$. This parameter balances the trade-off between adapting to target knowledge and preserving source knowledge. Fig. 6(a)(b) shows results on Spherical and NinaPro tasks with varying $\lambda$. We observe that optimal value lies in the range of $[0.3,0.5]$, whereas much higher or lower values lead to performance drop.

Inner-loop update steps. We take different steps of gradient descent in the inner-loop and show the results in Fig. 6(c). We observe that increasing the inner-loop steps slightly improves the performance. However, it also causes an increasing accuracy variation as well as computational cost. Therefore, following standard gradient-based meta-learning algorithms like MAML (Finn et al., 2017), we adopt the single step update. This helps to reduce the optimization time as well as the requirements for large memory.

Training Efficiency. The double loop optimization paradigm of meta-learning has its intrinsic limitation. However, MoNA mitigates this issue in practice and achieves comparable efficiency with ORCA, as shown in table 7. The reason is two fold: first, we adopt single step update as discussed above to reduce the inner-loop optimization time. Second, MoNA requires only 5 epochs of target embedder training to achieve satisfactory performance, which is significantly lesser than ORCA, which requires more than 60 epochs in the first stage. Therefore, although MoNA cost longer time in training one epoch, the overall training time is comparable between the two methods.

Transfer learning is extensively investigated in the topic of domain shift (Pan & Yang, 2010). Domain Adaptation (DA) (Ben-David et al., 2010; Long et al., 2018; Ganin & Lempitsky, 2015; Li et al., 2021) studies the knowledge transfer between source and target domain that differ in data distribution yet share the same task. finetuning (Yosinski et al., 2014; Xuhong et al., 2018; You et al., 2020; Kumar et al., 2022; Ma et al., 2023) is another powerful pipeline that enables knowledge transfer from large source dataset to different downstream tasks. It initializes the target model for downstream tasks using weights pretrained on massive source data, achieving higher performances than training the model from scratch. Our work belongs to the second transfer pipeline, yet addressing the more challenging situation where knowledge required for solving target tasks may not be quite aligned with source.

Cross-modal transfer extends the boundary of transfer learning from same modality to different modalities (Shen et al., 2023). The possibility of leveraging model pretrained on one modality to benefit tasks on other modalities gains increasing attention in recent years (Dinh et al., 2022; Lu et al., 2022; Reid et al., 2022; Mirchandani et al., 2023; Pang et al., 2023). One stream of works focuses on the knowledge reuse and transfer from source pretrained models. FPT (Lu et al., 2022) empirically shows that a pretrained language model (PLM) can benefit a variety of non-language downstream tasks, whereas other works transfer PLM to specific modalities (Zhou et al., 2023; Pang et al., 2023; Reid et al., 2022). ORCA proposes a general workflow for cross-modal transfer that first aligns the data distribution between source and target embeddings and then finetunes the source model to adapt to target modality. Another line of researches aims to learn a general model for several modalities. Meta-transformer (Zhang et al., 2023b) proposes to design modality-specific embedders while keeping a multimodal pretrained model like CLIP (Radford et al., 2021) frozen as the unified backbone. OneLLM (Han et al., 2023) further adds projection layers on top of the pretrained encoder, interfacing it to large language models.

Despite the empirical success of previous works, we still lack understanding of the reason behind these success. As the target modalities can be greatly diverse, the common assumption that knowledge from source modality is beneficial to all target modalities requires further examination. In this work, we formalize the knowledge discrepancy between modalities as an interpretation to the actual feasibility of knowledge transfer between certain source to target modality, and propose a novel method to reduce modality knowledge misalignment.

Our work leverages the optimization-based meta-learning approaches that use bi-level optimization to embed learning procedures like gradient descent into the meta-optimization problem (Hospedales et al., 2021; Rajeswaran et al., 2019). MAML (Finn et al., 2017) is a representative work alone this line of meta-learning. It learns parameters that can serve as a general initialization to solve new downstream tasks with high efficiency. The method mimics the learning process on new tasks in the inner-loop, and updates the outer-loop parameters according to the final performances of each new task within the inner-loop. The spirit of such idea also inspires researches in domain generalization (Li et al., 2018; Balaji et al., 2018; Dou et al., 2019; Li et al., 2019; Liu et al., 2020), where the inner-loop simulates the model’s generalization in unseen target domains.