Start from Video-Music Retrieval: An Inter-Intra Modal Loss for Cross Modal Retrieval

Due to the pervasively existed video music pairs online, retrieving a music for a video is usually treated as a cross-modal contrastive learning task. Early works such as [8] performed binary classification for pairwise cross-modal samples. Ranking/triplet loss ordered the similarity score of samples when the query was given (e.g. [7],[8],[9]). The distance or similarity of hash codes [10] [11][12], the canonical correlation analysis (CCA) [13] and the relational network (RN) [14] were also applied to model the cross-modal distance. In recent works, CLIP (Contrastive Language–Image Pre-training)[15] used a large number of image-text pairs to self-supervise the pre-training of two encoders for image-text retrieval tasks. During pre-training, the distance between cross-modal feature pairs of each mini-batch was processed by a simple cross entropy loss function. CLIP was then extended to various types of cross-modal retrieval and pre-training tasks such as video-text (CLIP4clip[16]) and audio-text (AudioCLIP[17]). In addition to matching and retrieval tasks, some recent works (e.g. BLIP[18]) explored different pre-training methods that enable encoders to migrate well to caption and QA tasks.

There are also many works that try to optimize the cross modal retrieval task by redesigning of loss function. Some works [6] [19] integrated the label classification loss into the inter-modal similarity loss. In order to prevent intra-modal structure collapse during cross-modal training, [7] and [20] added intra-modal loss to triplet loss. [21] designed a three-part loss for robust cross-modal retrieval.

To find an appropriate BGM from a music library based on the contents of both music and the given video, early works were more likely to choose music by matching labels. Emotion was always regarded as a critical factor in music-video retrieval [3] [5]. In addition, some works also incorporated factors such as location ([1] [4]), style and genre ([22]), comments and lyrics text ([23]), and user personality ([4]). In recent works, algorithms [6] [7] [19] [8] are trained by self-supervised contrastive learning methods and capable to learn the rich factors and preferences of video creators to choose BGM automatically. There are also some works ([24] [25]) generated new BGM for videos.

In video-music retrieval task, most works retrieved music based on the distance between the pretrained visual and audio features. The performance of pretrained feature extractors also becomes a key factor restricting retrieval performance. These pretrained feature extractors are pre-trained on large-scale datasets, and all parameters are frozen during fine-tuning. When extracting video embeddings, multiple frames can be sampled and the frame-level features such as ResNet-50 [26], Inception-V3 [27], ViT [28], or CLIP-Vision[15] can be extracted. Also, video-level features like I3D[29], SlowFast[30], and Video Swin Transformer[31] can be extracted. As for audio, frame-level features VGGish [32] or PANNs [33] can be extracted.

Existing video-music retrieval usually takes one continuous fixed-duration (FD) clip from the original media to represent the whole sequence, e.g. cutting 30$s$ around the center of both video and music as in [7]. Those methods ignore the rest parts of video and music, so that the retrieved music may only be partially related to the video. To extract features of the entire video and the whole music, the global sparse (GS) sampling [34] is applied. For video $i$, it is split evenly into $T_{v}$ clips and the video feature sequence $v_{i}\in\mathbb{R}^{T_{v}\times E_{v}}$ is obtained where $E_{v}$ is the dimension of the feature. Similarly, the audio feature sequence $m_{j}\in\mathbb{R}^{T_{m}\times E_{m}}$ is obtained for music $j$. Note that the purpose of extracting feature sequences of fixed length for video and music of different durations is to eliminate duration information and enable the model to retrieve based on content.

To extract the temporal information from the frame-level video and music feature sequences, V and M are fed into two sequence encoders (biLSTM, transformer encoder, etc), respectively. After encoding, the encoded video feature $\textbf{V'}=\left\{v_{i}^{\prime}\right\}_{i=1}^{N},\;({v_{i}}^{\prime}\in\mathbb{R}^{1\times D})$ and music feature $\textbf{M'}=\left\{m_{j}^{\prime}\right\}_{j=1}^{N},\;({m_{j}}^{\prime}\in\mathbb{R}^{1\times D})$ are obtained, where $D$ is the fixed hidden dimension of the sequence encoders for both video and music modalities.

As illustrated in Fig 2, we consider an example of a batch with a size of 4. The video $v_{2}$ and the music $m_{2}$ constitute a pair of positive samples. Due to the similarity between music $m_{4}$ and $m_{2}$, $m_{4}$ can also be effectively utilized as background music (BGM) for video $v_{2}$. During cross-modal training, if solely relying on the conventional inter-modal loss, the distance between the features output by the encoder $v_{2}^{\prime}$ and $m_{2}^{\prime}$ would decrease, while the distance between the features $v_{2}^{\prime}$ and $m_{4}^{\prime}$ would increase, which is an undesired outcome. As depicted in Fig 2, the proposed intra loss aims to preserve the relative feature distribution of the four music samples in the encoder output, as close as possible to their pre-encoder state, thereby preventing $m_{4}^{\prime}$ from moving away from $v_{2}^{\prime}$.

In II-CLVM, a mini-batch consists of $N$ matched video-music pairs. In each batch, the loss is a weighted sum of the inter-modal similarity loss and the intra-modal features distance distribution loss as shown in Fig 1.

When choosing a background music (BGM) for a video, the video creator may also provide some text information, such as the video’s description, title or keywords. Our framework can easily incorporate these text inputs. We use the Inter-Intra Contrastive Learning for Video$\&$Text-to-Music Retrieval (II-CLVTM) model, as shown in Fig 3, to fuse and encode the raw video feature sequences $V$ and text feature vectors $T$ with biLSTM. The text feature vector is used as the initial hidden vector for biLSTM. As Fig 3 shows, when the query is multi-modal, we compute the intra-modal loss based on the similarity matrix of the features before and after the cross-modal encoder. We also note that the uncoded feature $\bar{V}_{T}$ is obtained by concatenating the raw text feature and the average video feature.

In this section, the performance of each module in II-CLVM is tested on video-music retrieval task of official YouTube8M dataset[35]. What’s more, a subjective evaluation is conducted to check the BGM selection performance of II-CLVM on YouTube8M.

In YouTube8M [35], there are 149,213 video samples labeled as Music Video, in which 116,098 videos are for training and 33,115 videos are for testing. Three testing sets $A_{1024},B_{1000}$ with 1024, and 1000 pairs are randomly generated from the test data. YouTube8M provides the official video features (Inception-v3 [27], 1024-d per second after whitening PCA) and music features (VGGish [32], 128-d per second). In the following experiments, these features are used as the raw pretrained features.

All experiments are conducted in a 32G Tesla V100 GPU accelerated environment. The model is built with PyTorch. In II-CLVM, $N=32$, $\alpha_{1}=\alpha_{2}=\beta_{1}=\beta_{2}=0.5$, and $\gamma_{1}=1,\gamma_{2}=3$. The initial value of $n_{t}$ is 0.07. When the encoder is biLSTM encoder, the initial value of $n_{t1}$ is also 0.07. When GS sampling method is applied, $T_{v}=T_{m}=100$. Since YouTube8M provides second-level pretrained feature sequences, GS sampling is directly applied on the feature. Each model is trained for 30 epochs on the training dataset and the $recall@k$ ($R@k$) metric is measured on three testing sets $A_{1024},B_{1000}$, respectively. $R@k$ is the most common evaluation metric in all kinds of retrieval tasks and is the abbreviation for recall at $k$-th in the ranking list, defined as the proportion of correct matchings in top-$k$ retrieved results. Specifically, in video-to-music retrieval, $R@k$ refers to the probability that the correct music is included in the top $k$ retrieved music results. Apart from $R@1$, $R@10$ and $R@25$ are also noteworthy due to the fuzziness of video-music retrieval rule.

This section evaluates the generality of II loss on different cross-modal retrieval tasks using the audio-text dataset Clotho, the image-text datasets MSCOCO and Flickr30K, and the video-text datasets MSVD, MSRVTT and VATEX. The details of each dataset are shown in Table V. In these supervised retrieval tasks, each image, video or audio has multiple corresponding texts(The number of texts is given in the NOTE column of Table V). But the training data still contains noise. For instance, in the MSCOCO training set, images with IDs 226419, 166798, 303404 etc. depict planes in the blue sky, and images with IDs 368402, 505728, 509149 etc. portray women in the kitchen. These image-text pairs with similar contents can introduce noise to training.

The experimental environment and training parameters are the same as IV-A. BiLSTM is used as the encoder for video and audio feature sequences. For text features, FC layer is used as the encoder. $R@1$ is calculated to test each group of models. It is worth mentioning that In each epoch, only one text is randomly selected from the multiple texts associated with each image, video or audio sample to form a positive pair for training.

Table 4(a), 4(c), 4(f) shows the $R@1$ results of these cross-modal retrieval task. Results for all experimental groups are measured at the steady state of each model after 30 epochs of training. Although there is no longer just one-to-one matches in the datasets of these cross-modal tasks, the noise of false negatives still exists. On CLOTHO dataset, II Loss significantly improves $R@1$ from 2$\%$ to 4$\%$. On the two datasets of image-text retrieval, II Loss improves $R@1$ from 1$\%$ to 2$\%$. On MSVD dataset of video-text retrieval, II Loss improves $R@1$ greater than 5$\%$. And on other datasets of video-text retrieval, II Loss also improves $R@1$ from 0.5$\%$ to 2$\%$. In the meantime, II loss works when different pretrained features are used.

Using the MSVD dataset as an example, the three line charts in Fig 5 show the inter loss (5(a)) and intra loss (5(b)) during training, and the $R@1$ (5(c)) during testing as a function of the training epoch. The blue solid line represents the group that only uses inter loss, and the orange dashed line represents the group that uses inter-intra (II) loss. Note that in the only-inter group, the intra loss is calculated and plotted in the line chart 5(b), but not back-propagated. As shown in Fig 5(a), compared with the only-inter group, the inter-intra group reached a platform where it could not continue to decrease at a higher inter loss value, suggesting that intra loss can prevent inter loss from overfitting to the noisy training set. In Fig 5(b), Although the intra loss of the inter-intra group is always lower than that of the only-inter group, it does not decrease to a very low value. This is caused by the interaction between inter loss and intra loss. Fig 5(c) demonstrates that the $R@1$ score on the testing set for each epoch is higher in the inter-intra group than in the only-inter group.

The three line charts in Fig 4 show the trend of $R@1$ indicator on three datasets: MSCOCO, VATEX, and Clotho, as the inter loss weight ($\gamma_{1}$) takes the value of 1 and the intra loss weight ($\gamma_{2}$) takes values of 0, 3, 6, and 10. It is obvious that the impact of the II loss is more pronounced in downstream tasks with smaller data volumes. For example, in the Clotho dataset, the $R@1$ values increase from 9.62 to 14.09 as $\gamma_{2}$ goes from 0 to 10. It is also worth noting that $\gamma_{2}$ has a noticeable effect on the model’s performance. As $\gamma_{2}$ takes on values of 0, 3, 6, and 10, the $R@1$ indicator generally exhibits an upward trend. The trend slows down gradually and reaches a peak. For smaller datasets (such as clotho), the peak comes later. This is because on small datasets, the model tends to overfit to noisy sample pairs, which gives II loss more room for improvement. This distinctive characteristic of the II loss allows retrieval tasks to achieve satisfactory training results with only a limited number of cross-modal samples collected. To determine the optimal $\gamma_{2}$ value, a limited number of experiments should be conducted. By analyzing these experiments and the specific $R@1$ values, researchers can select the best intra loss weight to maximize the performance of the model on various datasets and tasks.

This section presents two experiments that focus on examining whether II loss can effectively resist the noise that exists in self-supervised retrieval tasks. The first experiment was performed on the YouTube8M dataset. Based on all the other experimental settings of the group biLSTM(II-CLVM) in section 4.1, this experiment changed the proportion of noise matching pairs by changing the batch size of training, and thus checked the different optimization effects of II loss under different amounts of noise. In Fig 6, the $R@k$ metric fluctuates as the batch size changes within the YouTube8m dataset. Initially, the $R@k$ value increases with the growing batch size due to more sufficient contrastive learning. However, as the batch size becomes larger, the number of noisy matching pairs increases exponentially, thereby negatively impacting training and leading to a decline in the $R@k$ value. The effectiveness of II loss in dealing with noise is demonstrated when comparing the II-CLVM group to the CLVM group; the former experiences a considerably slower decrease in $R@k$, suggesting that the II loss is effective in mitigating the adverse effects of noise, and ultimately achieves the highest $R@k$ with a batch size of 128.

To test the effect of II loss on resisting different levels of noise interference in training, another experiment was conducted on an image classification dataset Caltech-256 [43] by manually adjusting the noise amount in training batches. The Caltech-256 dataset contains 256 categories of images, with about 100 images per category. The image retrieval task on Caltech-256 was defined as follows: given a query image, the aim was to retrieve another image from the same category as the query image, and $R@k$ was used to measure the retrieval performance. We randomly chose two images from each of the 256 categories as a positive sample pair for the testing set, and used the rest of the images as training data. During training, each pair of positive samples was randomly selected from the same category of training data. We also arranged multiple pairs of samples from the same category in each training batch to simulate different levels of noise. Four different experiments were performed: no_noise, with_noise, more_noise, and most_noise, with increasing amounts of noise. N represented the batch size of training. In the no_noise group, the N pairs of positive samples were selected from N different categories. In the with_noise group, we picked one pair from each of $\frac{N}{3}$ different categories and two pairs from each of another $\frac{N}{3}$ different categories. In the more_noise group, we picked two pairs from each of $\frac{N}{2}$ classes. In the most_noise group, we picked four pairs from each of $\frac{N}{4}$ classes. In this experiment, $N=128$ and $a=6$. As shown in Fig 7, when there was almost no noise in the no_noise group, II loss had very little effect on R@k, and even decreased it. As the noise increased, II loss group showed more and more improvement compared to non-II loss group. In real-world self-supervised retrieval datasets, all sample pairs are unlabeled, so there is a high chance that a batch contains similar positive sample pairs. These similar positive sample pairs can produce false negative pairs as mentioned above, which can lower the model’s retrieval performance. II loss is an effective method to enhance the model’s generalization ability under such circumstances.