STACKED CONVOLUTIONAL DEEP ENCODING NETWORK FOR VIDEO-TEXT RETRIEVAL

Given a video, there are a large number of similar frames that contain redundant information. So we uniformly sample a sequence of $N$ frames with a interval of 0.5 second to represent the salient content of a video as follow [3] and [4]. And then a ResNet-152 [7] pretrained on ImageNet is adopted to extract the feature that represents each frame. In the end, each video is represented as $\boldsymbol{V}=\{\boldsymbol{v_{1}},\boldsymbol{v_{2}},\cdots,\boldsymbol{v_{N}}\}$ with $\boldsymbol{v_{i}}\in\mathbb{R}^{2048}$.

To capture the temporal information between video frames, we then feed $\boldsymbol{V}$ into a single-layer bidirectional GRU (bi-GRU [8]) with 512-dimensional hidden states to process the whole video. For each frame, we concatenate the hidden states from the two directional GRU as the global temporal representation:

$$\boldsymbol{\overrightarrow{h_{i}}}=\overrightarrow{GRU}(\boldsymbol{\overrightarrow{h_{i-1}}},\ \boldsymbol{v_{i}}),\quad\boldsymbol{\overleftarrow{h_{i}}}=\overleftarrow{GRU}(\boldsymbol{v_{i}},\ \boldsymbol{\overleftarrow{h_{i-1}}}),$$

(1)

$$\boldsymbol{f_{v_{i}}}=concate(\boldsymbol{\overrightarrow{h_{i}}},\ \boldsymbol{\overleftarrow{h_{i}}}),\ i\in\left\{1,2,\cdots,N\right\},$$

(2)

where $\boldsymbol{h_{i}}$ denotes to the $i$-th hidden state of bi-GRU.

Finally, we obtain a sequence of feature map $\boldsymbol{F_{vg}}=\left\{\boldsymbol{f_{v_{1}}},\boldsymbol{f_{v_{2}}},\cdots,\boldsymbol{f_{v_{N}}}\right\}\in\mathbb{R}^{N\times 1024}$ from the bi-GRU, which refers to the representation of the each frame in the video. We then apply mean pooling operation on $\boldsymbol{F_{vg}}$:

$$\boldsymbol{f_{vg}}=\frac{1}{N}\sum\limits_{i=1}^{N}\boldsymbol{f_{v_{i}}},$$

(3)

where $\boldsymbol{f_{vg}}$ denotes the global representation of the video.

For text embedding, previous works [3, 4] usually embed a word by a traditional word2vec [9] model and then construct the contextual representation relaying on a single RNN. In this way, the textual representations are not discriminative enough. Recently, a great progress has been made in the field of natural language processing (NLP): a language representation model BERT [10] is proposed and has proven to be effective in many NLP tasks. Motivated by this, our textual data is pretrained by BERT. To be specific, the BERT is trained to predict masked words inside a sentence based only on the context in an unsupervised way. In this way, we can capture the discriminative contextual information between words. After pretraining, the word embedding is then fine-tuned by Transformer module [11] in retrieval task. It is more effective and efficient than RNN, because of its superiority in language modeling.

Given a sentence with length $M$, we can obtain the $M$ fixed embeddings from the pretrained BERT. Specifically, we average the outputs of the last four layer from BERT as the embeddings. We enable to achieve the $M$ fixed embeddings $\boldsymbol{T}=\{\boldsymbol{t_{1}},\boldsymbol{t_{2}},\cdots,\boldsymbol{t_{M}}\}$ with $\boldsymbol{t_{i}}\in\mathbb{R}^{768}$. After obtaining the embeddings of a sentence, we send them into Transformer module to extract the global encoding of words, which employ multi-head self-attention to capture the abundant contextual information. We formulate this as:

	$\displaystyle{\boldsymbol{f_{t_{i}}}}^{k}=Transformer^{(k)}({\boldsymbol{f_{t_{i}}}^{k-1}}),$		(4)
	$\displaystyle i\in\{1,2,\cdots,M\},\ k\in\{1,2,\cdots,K\},$

where $K$ is layer number of Transformer, $M$ refers to the length of a sentence and ${\boldsymbol{f_{t_{i}}}}^{0}=\boldsymbol{t_{i}}$.

The global temporal feature map of a sentence is represented as $\boldsymbol{F_{tg}}=\{\boldsymbol{f_{t_{1}}},\boldsymbol{f_{t_{2}}},\cdots\,\boldsymbol{f_{t_{M}}}\}\in\mathbb{R}^{M\times 768}$. Same to video side, global representation of a sentence is produced by:

$$\boldsymbol{f_{tg}}=\frac{1}{M}\sum\limits_{i=1}^{M}\boldsymbol{f_{t_{i}}},$$

(5)

In addition to global information, local cues are proved to be also necessary in the field of computer vision and natural language processing. So we need to design a module to simultaneously capture the global and local cues. In this paper, we propose a Stacked Multi-scale Dilated Convolutional (SMSDC) module which can be inserted into video or text processing model. This module consists of two steps: the first step is designed to capture the local temporal cues among consecutive frames and skipped frames, while the second step is designed to capture the complex long-range relations among these local temporal cues. Our intuition is that diverse and refined information of an instance can be derived in this way.

Spatial convolution is capable of modeling the spatial context within an image, which is widely used in many tasks such as Image Segmentation. Inspired by this, we implement a multi-scale dilated convolution (MSDC) which represents local temporal feature learning and relation learning both in videos and texts.

The MSDC takes global feature map $\boldsymbol{F_{g}}$ (either of $\boldsymbol{F_{vg}}$ and $\boldsymbol{F_{tg}}$) as input and outputs a local feature vector $\boldsymbol{f_{l}}$ ($\boldsymbol{f_{vl}}$ for $\boldsymbol{F_{vg}}$ and $\boldsymbol{f_{tl}}$ for $\boldsymbol{F_{tg}}$). After obtaining this global feature map $\boldsymbol{F_{g}}\in\mathbb{R}^{L\times d}$, we adopt the dilated convolution (DC) operation which is formulated as:

$\displaystyle\boldsymbol{f^{(r,w)}(t)}=\sum_{i=1}^{w}\boldsymbol{F_{g}[t+r\cdot i]}\times\boldsymbol{W_{[i]}^{(r,w)}},\ t=1,2,\cdots,L$

(6)

where $\boldsymbol{W_{[i]}^{(r,w)}}$ denotes the parameters in the the dilated convolution, $w$ is the kernel size and $r$ is the dilated size.

$\boldsymbol{F_{DC}^{(r,w)}}=\{\boldsymbol{f^{(r,w)}(t)}\}\in\mathbb{R}^{L\times d}$ is the collection of output features of DC. The dilated convolution adaptively fuses the local frames or words together for constituting the different local events. These events contain the different local temporal cues.

To capture more different semantic cues, we extend DC to a multi-scale dilated (MSDC) form with different kernel scale and dilation size, which can capture local temporal cues in terms of different receptive field. Defining $w_{n}\in\{2,\cdots,n+1\}$ be the kernel scale set and $r_{m}\in\{1,\cdots,m\}$ be the dilation size set. $\boldsymbol{F_{MSDC}}=concate(\boldsymbol{F_{DC}^{(r_{1},w_{1})}};\boldsymbol{F_{DC}^{(r_{1},w_{2})}};\cdots;\boldsymbol{F_{DC}^{(r_{m},w_{n})}})\in\mathbb{R}^{nm\times L\times d}$ is the output of MSDC. To represent each input to a vector, we apply nonlinear activation $\sigma$ and maxpooling along the time dimension:

$$\boldsymbol{F_{MSDC}}=maxpooling(\sigma(\boldsymbol{F_{MSDC}}))\in\mathbb{R}^{nm\times d},$$

(7)

We then stack another MSDC on $\boldsymbol{F_{MSDC}}$ to model the long-range relations among the $nm$ different local semantic features, producing a discriminative feature representation $\boldsymbol{F_{SMSDC}}\in\mathbb{R}^{nm\times d}$.

To fuse the $nm$ features into a single vector, we simply apply concatenation fusion to get the final local representation:

$\displaystyle\boldsymbol{f_{l}}=concate(\boldsymbol{F_{SMSDC}[0]},\cdots,\boldsymbol{F_{SMSDC}[nm-1]}),$

(8)

where $\boldsymbol{f_{l}}\in\mathbb{R}^{nmd}$ refers to the output of our proposed SMSDC, which is used to calculate the similarity between video and text.

After encoding the video and text, we can obtain the global and local features of them, respectively. Then, we apply a concatenation operation to fuse them in order to provide more semantic cues for matching calculation between video and text.

$$\boldsymbol{f_{v_{gl}}}=concate(\boldsymbol{f_{vg}},\boldsymbol{f_{vl}}),\quad\boldsymbol{f_{t_{gl}}}=concate(\boldsymbol{f_{tg}},\boldsymbol{f_{tl}}),$$

(9)

In order to calculate the similarity between video and text, we embed them into a joint space, in which the embedding of the positive video-text pairs should be close to each other while the negative pairs should be far away.

Following [4], we use a Fully Connected (FC) layer following with a Batch Normalization(BN) layer as the final embedding layer, which can be formulated as:

	$\displaystyle\boldsymbol{f_{v}}=BatchNorm(\boldsymbol{W_{v}}\boldsymbol{f_{v_{gl}}}+b_{v}),$		(10)
	$\displaystyle\boldsymbol{f_{t}}=BatchNorm(\boldsymbol{W_{t}}\boldsymbol{f_{t_{gl}}}+b_{t}),$

where $\boldsymbol{W_{v}}$, $\boldsymbol{W_{t}}$, $b_{v}$, $b_{t}$ are weights and biases.

With the final embedding of video and text as described above, the similarity of $f_{v}$ and $f_{t}$, denoted as $S(v,t)$, can then be computed as the cosine of the angle between them. The similarity should be subject to the model parameters during the optimization. We utilize bidirectional max-margin hard ranking loss to optimize the model, which is the state-of-the art loss and widely used in vision and language matching task. The loss of this optimization problem can then be written as:

	$\displaystyle\mathcal{L}=max(0,\alpha-S(v,t)+S(v,t^{-}))$		(11)
	$\displaystyle+max(0,\alpha-S(v,t)+S(v^{-},t)),$

where $\alpha$ is the margin constant and is set as a hyperparameter. During the training, hard negative sample is utilized within a batch to penalize the model: $t^{-}$ is the negative text for $v$ while $v^{-}$ is the negative video for $t$.

The whole model is trained end-to-end to minimize $\mathcal{L}$ except that the image feature and word embedding are respectively extracted by pretrained CNN and pretrained BERT, which are all fixed.

Datasets. In this paper, we conduct experiments on two benchmark datasets: MSR-VTT [12] and MSVD [13] to evaluate the performance of our proposed framework. MSR-VTT and MSVD are two challenging datasets in the filed of video question answering, video captioning and video-text retrieval. MSR-VTT consists of 10000 video clips , each of which is annotated with 20-sentence descriptions. MSVD is a small dataset, which contains 1970 videos, while each of them has around 40 sentences. Note that, there are two kinds of sentence constructing strategies in the previous work: JMET [14] uses all the ground-truth sentences while LJRV [15] randomly samples 5 ground-truth sentences for each video. We follow the latter strategy. The splits of train, validation and test for the two datasets are consistent to previous work.

Evaluation Metric. Following prior work on vision-text matching task, we use the standard rank-based criteria: R@K (Recall at rank K, K=1, 5, 10), MedR (median Rank), MeanR (mean Rank) and mAP (mean Average Precision), to report our retireval performance. $RSum=R@1+R@5+R@10$ is calculated to compare the overall preformance. Note that, due to the missing result in prior work, we only report mAP for MSR-VTT and MeanR for MSVD.

Implementation Details. We set the margin $\alpha=0.2$ in the rank loss and the dimension of the embedding space in equation $(10)$ is set to 2048. (n, m) pair in SMSDC is empirically set to (4, 2) for video and (3, 2) for text. The Transformer layer number K is set to 3. We keep the batch size to 64. Learning rate is initialized to $5\times 10^{-5}$ and is decreased by a factor of 2 once the performance does not increase in three consecutive epochs. SGD with Adam is adopted as the optimizer. The maximal number of training epoch is set to 30. The best model is chosen by sum of recalls.

Table 1 and Table 2 show the experimental results and comparisons with previous methods on the two datasets respectively. All the results are cited from its original papers if available. We can see that our proposed method performs best and consistently outperforms state-of-the-art methods in both text-to-video and video-to-text retrieval. It verifies the effectiveness of our proposed method. The performance of video-to-text is higher than text-to-video because one video is paired with several sentence and we take the most similar one as the rank. Sum of recalls is increased from 148.6 to 162.7 on MSR-VTT dataset and 226.7 to 253.9 on MSVD dataset. Note that, we also use the source code and original settings of [4] to test on MSVD, the sum of recalls is around 241, which is still much lower than ours. The relative improvement is smaller on MSVD dataset compared with a same method, the reason may be that the length of videos are longer and the number of sentence is more and thus can provide more semantic information.

We also conduct experiments on MSR-VTT dataset to evaluate the effectiveness of different components in the proposed method, including the text side embedding modeling and the proposed convolution module. The experimental results are shown in table 3, in which the baseline is both video feature and text feature encoding by Bi-GRU with word2vec embedding. Then the word embedding and Bi-GRU of text respectively replaced by BERT pretrained embedding and Transformer. Compared with baseline method, BERT pretraining with Transformer fine-tuning is effective. Then we add MSDC on both video and text, we can see the result is improved, which demonstrates the effectiveness of capturing dense temporal cues. SMSDC further improves the performance, which further captures the relation between temporal cues and result in a refined and discriminative feature.