Learning Trajectory-Word Alignments for Video-Language Tasks

Recently, various techniques have been proposed in IL-BERT to learn vision-language connections. Most of them aim at capturing robust object-word alignments since such alignments construct the foundations of visual reasoning. In the beginning, a pre-trained Faster-RCNN [40] is used to extract a series of RoI poolings, where each one contains one or a few salient objects, to facilitate building object-word alignments [33, 47, 8, 27]. However, this Faster-RCNN is usually pre-trained by the object annotations from COCO [7] and VG [20], whose concept space is much narrower than the data used to train IL-BERT, which can be almost unlimitedly collected from the websites. Moreover, this Faster-RCNN is not updated during the end-to-end training of the IL-BERT, which means that the visual encoder may hardly learn novel knowledge from the web-collected data. Thus the performances of these IL-BERTs are limited.

To further release the potential of hugely web-collected data, the offline Faster RCNN is switched into vision Transformers [11, 31] and thus a homogeneous IL-BERT, i.e., all the components are transformer-based, is built, which is more easily trained end-to-end [33]. Compared with the RoI-based encoder, the vision Transformer outputs a series of patch embeddings and thus may lose object-level knowledge. To remedy such loss, various strategies are proposed to improve the vision-language alignments. For example, the align-before-fuse strategy [25] aligns the paired image-text embeddings before cross-modal fusion for facilitating the subsequent fusion. And the fine-grained contrastive objective [56] amplifies the local details for learning the object-word alignments.

Since the spatiotemporal contexts in videos can hardly be learnt by image extractors, only inheriting the techniques which are successful in IL-BERT to VDL-BERT is not enough. Thus, based on the fruits of IL-BERT, most VDL-BERTs aim at exploiting more spatiotemporal contexts to build cross-modal associations. One straightforward way is to learn such spatiotemporal contexts through video Transformers [4, 28]. Besides this, some more advanced techniques are proposed, e.g., VIOLET [12] tokenizes the dynamic video patches and predicts the labels of these tokens; BridgeFormer [14] erases the words (nouns or verbs) from the text and learn to match the visual embeddings queried by the erased words and the remained texts; or ALPRO [24] computes the similarities between the video embeddings with the generated entity prompts. TS2-Net [30] shifts token features and selects informative tokens in both temporal and spatial dimensions to produce the fine-grained spatial-temporal video representation. Although substantial improvements are observed, these methods use video patches in the cross-modal encoder, which neglects that an object usually acts as the trajectory in the videos, and thus they may over-exploit the trivial spatial contexts. To ameliorate this limitation, we propose TW-BERT to capture trajectory-word alignments for more robust vision-language alignment.

After embedding videos and texts, a cross-modal encoder is used to fuse them by calculating bi-directional associations: vision-to-language and language-to-vision. No matter what the direction is, the motivation is to assign semantic knowledge from one domain to another. Since a single word contains integral semantic knowledge, we follow previous VDL-BERT [24, 2, 23, 46] to set a traditional word-to-patch (W2P) attention to assign the words to a patch. However, different from the word, only the object instead of one grid conveys integral semantic knowledge. In videos, an object usually spans both spatial and temporal axes and thus the previously used patch-to-word (P2W) attention may fail to transfer the semantic knowledge. To amend this, we design a novel Trajectory-to-Word (T2W) attention for transferring semantic knowledge from videos to texts. Since W2P and T2W attentions have diverse structures, our cross-encoder is asymmetric.

Specifically, both W2P and T2W attentions are built on the Multi-Head Attention (MHA) operation, here we first formalize MHA and then introduce how to use it to build W2P and T2W attentions. Formally, MHA is^***To avoid symbol confusions, we use the calligraphic font ($\setminus$mathcal commend in LaTex) to denote the built-in variables of the MHA module.:

where $\mathcal{W}_{i}^{Q},\mathcal{W}_{i}^{K},\mathcal{W}_{i}^{V}$, $\mathcal{W}_{i}^{H}$ are all trainable matrices; $h$ is the head number and $d_{h}=d/h$; $\mathcal{A}_{i}$ is the $i$-th attention matrix corresponding to the $i$-th head matrix; $[\cdot]$ is the concatenation operation; and LN is the Layer Normalization.

In the first step, T2W attention uses $\bm{x}$ as the query to find the salient parts for each frame and then sequence these parts to construct the trajectory. Assuming $\bm{V}_{t}$ is the embedding set of the $t$-th frame, the salient part $\bm{y}_{t}$ is got as:

Then the salient parts at different time frames construct a continuous flow $\bm{Y}=\{\bm{y}_{1},...,\bm{y}_{T}\}$, which is the trajectory of the given word.

In the second step, to get the trajectory-to-word fusion embedding $\bm{z}_{T2W}$, we treat $\bm{x}$ as the query again while using the trajectory $\bm{Y}$ as the key and value in MHA:

By Eq. (3), T2W attention finds the salient parts for the given word at each frame, which enforces Eq. (4) to attend over the continuous frames instead of concentrating the attention only on one or some episodic frames as the previous P2W attentions. In this way, the whole T2W block exploits more temporal contexts to build vision-language associations, which facilitates the video reasoning tasks that usually require the correct recognition of the temporal patterns. Figure. 3 compares P2W and T2W attentions.

During fine-tuning, to improve the performance, previous studies tend to sample more frames than pre-training for introducing more temporal information. However, on the one hand, using too many frames will dramatically increase the computational resources, which causes infeasible training. On the other hand, in a video, different trajectories have varying graininess, e.g., some trajectories may cover longer frames and some cover shorter. In this way, uniform sampling may introduce redundant temporal information for these longer trajectories and deficient information for the shorter ones. Interestingly, the time duration of the trajectory is also reflected in the corresponding text. In this way, we can use the text to choose suitable frames and then only input them into the cross-modal attention for learning trajectory-word alignments.

Motivated by this, we propose a novel Hierarchical Frame-Selector (HFS) that can gradually select the most relevant video frames to reduce the computational burdens while providing more temporal knowledge. Figure 4 illustrates the architecture of our proposed frame-selector module. HFS contains a few frame selector layers which have the same structure with different parameters. For each frame-selector layer, given a sequence of video frame [CLS] tokens $\bm{V}_{cls}=\{\bm{v}^{1}_{cls},\bm{v}^{2}_{cls},...,\bm{v}^{T}_{cls}\}\in\mathbb{R}^{T\times D}$, we concatenate the text [CLS] embedding $\bm{x}_{cls}$ with each $\bm{v}^{t}_{cls}$: $\bm{f}_{t}=[\bm{v}^{t}_{cls},\bm{x}_{cls}]\in\mathbb{R}^{2D}(1\leq t\leq T)$ and feed them into a scorer network including a FC layer and a softmax layer:

Here, $T$ denotes the number of frames input into the video encoder, and $\bm{s}\in\mathbb{R}^{T}$ is the score vector of T frames. Then for this frame selector, we keep the tokens of $k$ frames with the top-$k$ $s_{t}$ for the further embedding in the subsequent video Transformer layers. During training, we employ the perturbed maximum method [5] to construct a differentiable Top-K operator. In the video encoder, we totally insert 2 selector layers and insert them in the 6-th and 12-th layers as in Figure 4.

To train TW-BERT, as the grey blocks shown in Figure 2, we totally use three losses which are masked language modeling (MLM), video-text matching (VTM), and video-text contrastive loss (VTC).

Following recent work [24, 14, 15, 48, 2], we pre-train TW-BERT on Google Conceptual Captions (CC3M) [44] containing 3.3M image-text pairs and WebVid-2M [2] containing 2.5M video-text pairs. For CC3M, the image is treated as a one-frame video data during pre-training. Note that due to the limited storage and computation resources, we do not use some much larger datasets like HowTo100M [37] containing 136M video-text pairs as [26, 53]. Also, we do not distill knowledge from CLIP [38], which is pre-trained on 400M image-text pairs, as [35].

We initialize our video encoder by ViT-B/16 [41] and the text encoder by the first six BERT-Base layers [9]. For the cross-modal encoder, the self-attentions in all 3 cross-modal attentions (W2P contains 1 and T2W contains 2) are initialized by the last 6 BERT-Base layers [9]. The model is trained end-to-end during both pre-training and fine-tuning.

In pre-training, the feature dimension is set to 256 when calculating the contrastive losses and the temperature is set to 0.05. For the momentum queue, the momentum value is 0.995 and the size of the queue is 65,536. The above implementation details follow the recent work [25, 24] for a fair comparison. We pre-train the model on CC3M and WebVid-2M for 10 epochs on 8 NVIDIA A100 GPUs where the batch size is 128. We use AdamW [18] optimizer with a weight decay of 0.001 and betas (0.9, 0.98). The learning rate is first warmed-up to 1e-4 and then decays following a linear decay schedule.

During fine-tuning text-to-video retrieval, We sample 20 frames and use HFS to preserve 8 frames to feed into the cross-modal encoder. The model is trained with both VTC and VTM losses, and we obtain similarity scores from the output of the VTM head during inference. For video question answering, we sample 32 frames per video and preserve 16 frames. Since MSRVTT-QA and MSVD-QA [51] are open-ended VQA, in which the answers are in free-form natural language, it is common to convert the task to a classification task by predicting the answer’s label. We input the concatenation of the video and question [CLS] tokens into a two-layer MLP [9] for calculating the cross-entropy loss. All the fine-tuning experiments are conducted on 8 NVIDIA V100 GPUs.

We conduct comprehensive ablation studies to evaluate the effectiveness of the proposed trajectory-to-word (T2W) attention and Hierarchical Frame-Selector (HFS) module.

[L1,L2]: We insert frame-selector layer into the L1-th and L2-th layers. F@T: We sample T frames into the model. F@T1-T2-T3: We use Hierarchical Frame Selector where “F@T1-T2-T3” denotes that T1 frames are originally input into the model and then T2 and T3 frames are respectively remained by the first and second frame-selector layers. All these ablations are implemented using the fine-tuning setting.

We further analyze the impact of the layer settings and frame number on the Hierarchical Frame-Selector module. As shown in Table 2, frame-selector in deep layers (e.g. 6-th and 12-th layers) brings better performances on both retrieval and QA tasks. Moving frame-selector module into shallower layers deteriorates the scores. For example, the accuracy drops 0.9% when the frame-selector module is placed before the third layer. We hypothesize that discarding frames too early would result in a lack of temporal knowledge in subsequent ViT layers. We thus choose to insert the frame-selector module in the 6-th and 12-th layers in the following experiments. In terms of frame number, as shown in Table 3, on the one hand, as the number of frame we sample increases, both retrieval and QA performance obtain improvements. For example, in MSRVTT fine-tuning text-to-video retrieval task, F@20-14-8 outperforms F@8-6-4 by 0.9% on R@1 or on MSVD video question answering task, F@32-24-16 outperforms F@24-18-12 by 0.6%, which suggests that more temporal knowledge benefit our model whether using uniform sampling or our proposed HFS. On the other hand, using HFS gets higher scores than uniform sampling, e.g., F@20-14-8 achieves 38.4% of R@1 score in MSRVTT retrieval task while F@8 only has 38.1%, or on MSVD video question answering task, F@32-24-16 outperforms F@16 by 0.3%, which validates the power of HFS module.

We compare TW-BERT with previous methods on two frequently applied tasks: video-text retrieval (VDTR) and video question answering (VDQA). Table 4 and 5 report the performances of VDTR on MSRVTT [54], DiDeMo [16], LSMDC [42], and ActivityNet Caption [19], respectively, where the former three datasets contain both zero-shot and fine-tuning setups and the last one only has fine-tuning setup. Table 6 reports the VDQA on MSRVTT [54] and MSVD [6]. Among the compared models, MILES [15], BridgeFormer [14], OA-Trans [48], Clipbert [23] and VIOLET [12] are SOTA models proposed in recently 1-2 years. Note that VIOLET [12] and ALPRO [24] distill knowledge from additional large-scale BERTs while we do not. Also, we show the number of pre-training video-text pairs in these tables for more clear comparisons.

From these tables, we can find that when the pre-training data is in the same scale, TW-BERT achieves the best performance compared with all the other models on both VDTR and VDQA. For example, on DiDeMo VDTR, TW-BERT outperforms BridgeFormer by 4.8% on R@1, or on MSVD VDQA, TW-BERT outperforms ALPRO by 2.6%. Moreover, compared with the models trained on much more data, TW-BERT can still achieve the best performances on various tasks, e.g., on LSMDC VDTR, TW-BERT outperforms VIOLET by 8.0% on R@10 or on MSRVTT VDQA, TW-BERT outperforms MERLOT [57] by 0.5%.

Note that the videos in LSMDC are longer than MSRVTT and DiDeMo, which means that the videos in this dataset contain more temporal contexts than the other datasets. Then as shown in Table 4, the improvements of TW-BERT over other SOTAs are larger than the improvements on the other datasets. For example, compared with BridgeFormer, TW-BERT has an average 3.5% improvement in the zero-shot setting, while on DiDeMo dataset, the corresponding average improvement over MILES is only 1.6%. Such comparisons further validate the effectiveness of TW-BERT in exploiting temporal contexts.

Among these SOTAs, only when compared with VIOLET, which uses an additional large-scale model DALL-E [39] and 32 more times pre-training data than ours (180M VS. 5.5M), TW-BERT cannot comprehensively surpass VIOLET on all the tasks. For example, on MSRVTT VDQA, VIOLET achieves 0.3% higher than TW-BERT, while such marginal improvements are got at the cost of much more training resources. Furthermore, TW-BERT still outperforms VIOLET on the other tasks, e.g., on DiDeMo VDTR, TW-BERT outperforms VIOLET by 9.2% on R@1 or on MSVD VDQA, TW-BERT outperforms VIOLET by 0.6%. These comparisons confirm the effectiveness of the proposed TW-BERT.