RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrieval

To comply with disentangled video features, we also disentangle the textual features hierarchically. Every video clip has at least one descriptive textual sentence in a hierarchically structured way. For instance, while a sentence could define global features, action and entity-related words would define the local features.

By following recent works [11, 3], we utilize [36] for handling semantic role labeling to disentangle the structure of sentences in the form of graphs. To exploit the semantic relationship among the words, verbs are connected to the sentence by showing the temporal relations with their directed edges, and then the objects are connected to verbs. We define the relationship between actions and entities as $r_{ij}$, in which $i$ refers to action nodes and $j$ refers to verb nodes. Before applying GCN, we use bidirectional LSTM to have contextualized word embeddings. While we apply soft attention for sentence-level embedding, we implement max-pooling over verbs and objects for their corresponding embeddings.

Rather than using relational GCN [37] by learning weight matrix for each semantic role which would be computationally expensive, we follow Chen et al. [3] to utilize factorised weights in two steps. Firstly, the semantic role matrix $W_{r}$, which is specific matrix for different semantic roles are multiplied with initialized node embeddings $g_{i}$ $\epsilon$ {$g_{e}$, $g_{a}$, $g_{o}$} in Eq. 1. $r_{ij}$ is an one-hot vector referring to the edge type from node $i$ to $j$. $\odot$ refers to element-wise multiplication.

Secondly, a graph attention network is applied to neighbor nodes. Then $W_{t}$ matrix, which is shared across all relations types, is used to exploit attended nodes in Eq. 2. $\beta$ refers to the outcome after attention is implemented for every node.

After applying these two formulas, we gather the textual representation for global and local features; for sentences, verbs, and words, respectively.

While disentangling language queries into hierarchical features is relatively easy, parsing videos into hierarchical features could be challenging. We propose different transformer models for global and local levels. While we use only intra-modality features at the global level, we utilize both intra-modality and inter-modality specific features at the local levels.

The intuition behind this is that while the global features could guide the local representations through the cross-modal attention at the action or object level, the local features may give noise to the global representations when used at the appearance level. We use transformer encoder and decoders to implement this idea as inspired from [38, 10, 39, 34].

Calculating appearance-level video encodings is relatively easy. The 2D feature $F_{S}$ is fed into the self-attention layer to encode into $z_{s}$. Then, a feed-forward layer FF outputs the final contextualized appearance feature. Normalization of the layer is done under the Norm function.

We follow Vaswani et al. [38]’s multi-headed attention layers as formulated in Eq. 4. During the training process, all the W matrices are learned. While query Q is the same as key K and value V in the self-attention layers, the query Q could be from another source in the cross-modal attention layers. Layer normalization and the residual connection are also applied after every attention layer.

Complementary features are applied with cross-modal attention to calculate the verb $E_{A}$ and noun $E_{O}$ level encodings. For instance, we explain the formula for verb level in Eq. LABEL:eq:encoder. First, the appearance level $F_{S}$ and noun level features $F_{O}$ are concatenated. Then, a multi-headed self-attention is applied, followed by a feed-forward layer FF. This generates context-specific feature $s_{e}$ of the verb level.

Then, self-attention of verb level features is calculated as in the first row of Eq. LABEL:eq:encoder.

Finally, cross-modal attention is applied such that the contextualized verb-level features would attend to the contextualized appearance and object-level features. Then, a feed-forward linear layer FF and layer normalization are applied. We repeat the same process to calculate the noun-level embeddings. Then, these three-level video embeddings are used in the video-text matching part.

Some works such as [3, 11] average the similarities of the different embedding levels, assuming that they are equally important. Other works [10, 5] apply weighted sum but predicted from textual encodings. On the other hand, we apply weighted sum to the video encodings as the video encodings, especially at the local level, become contextualized via cross-modal attention with the other modalities. The guidance of the textual encodings may not be necessary.

We simply use cosine similarity with the video and textual embeddings to calculate the matching score by adding a weighting calculated on video embeddings only. We utilize contrastive ranking loss [3] in training by aiming to have positive and negative pairs larger than a pre-set margin. If $v$ and $c$ symbolize visual and textual representation, the positive pairs and the negative pairs could be presented like following, $(v_{p},c_{p})$ and $(v_{p},c_{n})$ / $(v_{n},c_{p})$, respectively. A pre-set margin is referred to as $\Delta$ to calculate the contrastive loss.

$W_{i}(v)$ represents the weight for the ith video expert feature. We feed the expert encodings into a linear layer and then apply a softmax operation. Since we have three linear layers, we would calculate three different weights.

YouCook2 dataset: This dataset [40] is collected from YouTube only from cooking-related videos with 89 different recipes. They are all captured in various kitchen environments from the third-person viewpoint. The video clips are labeled by imperative English sentences along with temporal boundaries referring to the actions. The annotation was done manually by humans. Video clips are extracted by using the formal annotation file. We feed our model with the validation dataset for evaluation following the other studies since the test set is not publicly available. Normally, the dataset has 10,337 clips and 3,492 clips in the training and validation set, respectively. However, some of the raw videos are not available due to being removed or becoming private. This makes  8% of training clips and  7% of validation clips missing in total. Thus, we use 9,496 clips and 3,266 clips in the training and validation set, respectively.

MSR-VTT dataset: It [41] contains 10K video clips with 20 descriptions for each video and collected by a commercial video search engine. The annotation is done by humans. This dataset has various train/test splits. One of the most used splits is called 1k-B. It has 6,656 clips for training and 1,000 clips for testing. The full split, which is another widely used split, contains 6,513 video clips for training, 497 for validation, and 2990 for testing. While we test our method mostly in this 1k-B split, we also report in the full split as well.

Video features: We extract our own features from the raw videos of YouCook2 by following Miech et al. [6]. To extract appearance features we use two dimensional Convolutional Neural Networks of ResNet-152 [42] model, pre-trained on Imagenet [43]. To extract action level features we utilize three dimensional Convolutional Neural Networks of ResNeXt-101 [44] model, pre-trained on Kinetics [45]. Later on, we implement max-pooling on temporal dimension, making the dimension (1, 2048) for each feature set. The process is also depicted in Figure 2. For MSR-VTT, we gather appearance level features of ResNet-152 model from Chen et al. [3] and action level features of ResNeXt-101 model from Liu et al. [8], respectively.

Evaluation metrics. Given a textual query, we rank all the video candidates such that the video associated with the textual query is ranked as high as possible. Recall (R@k) and Median Rank (MedR) are used to evaluate the model. For instance, R@k is calculated if the ground truth video is in the first kth in the ranking list, where k is {1,5,10}. Higher recall values refer to better performance. MedR denotes the median rank of correctly predicted video clips in the list. The lower MedR is, the performance is better.

Training. Glove embeddings [46] is used to set the word embedding with the size of 300 for the textual embedding. To parse the text embeddings hierarchically, we follow [3, 11]. Graph convolutions are applied with two layers, outputting 1024 dimensions. We apply a linear layer to each video expert feature embeddings to be compatible with the textual layer, making their dimension 1024 as well. The $\Delta$ margin is set to $0.2$. Our mini-batch size is 32. While the epoch is 100 for the YouCook2 dataset, it is 50 for the MSR-VTT dataset.

Table I compares our approach with SOTA methods for video retrieval task on YouCook2 dataset. It includes all the methods that report their results without pre-training. The above part of Table I shows the methods when using the same feature set. We overpass all the recent papers in all metrics, except the last metric for one work [31], using the same feature backbone. Please note that the HGR method by Chen et al. [3] is reported by Satar et al. [11]. All the models in the table use pre-extracted features; in other words, they are not end-to-end models, making the comparison fair.

The lower part of Table I introduces the results with pre-training and different visual feature backbones. We see higher results for two reasons. UniVL [18], COOT [31], and TACo [20] methods use S3D features pre-trained on HowTo100M. Since HowTo100M is similar to YouCook2 as an instructional video dataset, it would be more useful than pre-training on Kinetics. On the other hand, MMVC [33] uses S3D features pre-trained on Kinetics. However, this model exploits additional audio and Automatic Speech Recognition (ASR) features which could boost the performance.

Table II compares our approach with SOTA methods when using the same feature set on the MSR-VTT dataset. These include all the methods that report text-to-video retrieval results without pre-training. We overpass all the methods in all metrics. In particular, our method surpasses the performance by 2.9% on R@1 over the CE method [8], which uses four features, even though we only use two of them. Moreover, our method improves the performance by 1.9% on R@1 over the best SOTA approach [20].

For the full split, we overpass all the methods that we have similar feature sets. However, we surpass the CE method [8] only in the first two metrics. One of the main reasons that CE overpasses our method in the last two metrics could be the advantage of using various other visual experts such as audio and ASR. Secondly, the testing set becomes tripled than 1k-B split, increasing the advantage of having many visual expert features. Thirdly, our models’ textual encoding part may not handle having twenty captions for each video clip, as it is included in MSR-VTT. A transformer-based approach such as BERT may perform better, although we did not apply any experiments with BERT. Moreover, it is hard to make a comparison among some approaches. For example, Alayrac et al. [7] report their result when pre-trained on HowTo100M and a zero-shot setting. TeachText-CE [48] uses various textual feature experts as long as seven different visual feature experts, making it hard to compare with our result, which is still added to the table for reference. Nevertheless, we overpass the CE method in the first two metrics, which could be more critical in real-world applications.

Although the video-to-text retrieval task is not our primary motivation, we report it in Table III to show that our model could also be used in this aim. For YouCook2, only one method reports their video-to-text retrieval results when there is no pre-training, which is reported by [11]. We overpass the compared methods in all metrics with a high margin on both datasets without pre-training.

We share extensive ablation studies about design choices on weighted sum, model design, and video feature settings.