Deeply Interleaved Two-Stream Encoder for Referring Video Segmentation

Early method [7] utilizes concatenation and convolution to fuse linguistic and visual features. Subsequently, Liu et al. [16] and Li et al. [14] adopt recurrent LSTM and convolutional LSTM to gradually combine the concatenated multi-modal features in the decoder, respectively. These methods directly concatenate visual and linguistic features, which dose not explicitly consider the relationship between each pixel and each word.

To solve this problem, some methods exploit attention mechanism to model the cross-modal relationship. For example, Shi et al. [23] use the visual features as a guidance to learn the keywords corresponding to each pixel, which can suppress the noise in the query expression. Ye et al. [27] and Seo et al. [22] employ non-local module to model the relationship between the pixel-word mixed features. Wang et al. [25] propose an asymmetric cross-attention module, which utilizes two affinity matrices to update visual and linguistic features, respectively. Hu et al. [8] build a bidirectional relationship decoder to realize the mutual guidance between language and vision. Hui et al. [11] rely on the dependency parsing tree to re-weight the edges of the word graph, thereby suppressing the useless edges in the initial fully connected graph. Huang et al. [9] first perceive all entities in the image according to the category and appearance information and then employ the relation words to model the relationships among all entities. Ding et al. [4] design a transformer-based query generation module at the decoding end to understand the vision-language context.

Some other methods employ the dynamic filters generated by the sentence to match the visual features, thereby strengthening the response of the language-related visual region. Gavrilyuk et al. [6] directly apply a dynamic filter to weight all channels of feature map and sum them to yield the pixel-wise segmentation map. Similarly, Margffoy-Tuay et al. [18] learn a set of dynamic filters for each word to re-weight the visual features. Wang et al. [24] introduce deformable convolution [2] into the process of dynamic filtering. In addition. McIntosh et al. [19] design a capsule network to perform vision-language coding.

A common feature of all the above methods is that they fuse visual and linguistic features at the decoding end of the network. The interaction between visual feature of each scale and language feature is isolated. Recently, the encoder fusion strategy [20, 5, 10] is applied to referring segmentation task. It achieves the continuous guidance of language to multi-scale visual features. Regardless of the encoder fusion or the decoder fusion, their language encoding processes do not consider the boosting effect of visual information of multiple levels, and ignore the semantic hierarchy of linguistic information. Integrating the same language features with visual features of different levels may weaken the consistency of cross-modal matching. Different from them, we propose a vision-language interleaved two-stream encoder, which can extract hierarchical language information with the help of multi-level visual features and implement the bidirectional cross-modal interaction at different levels.

Modeling temporal information between frames is important for referring video segmentation. A straightforward idea is to use 3D convolution to capture temporal cues of the video. Therefore, some methods [6, 25, 24, 19] directly employ the I3D network [1] to encode video sequences. In the testing phase, the fixed convolution parameters are not easily generalized to all video clips, which limits the capability of temporal modeling. To solve this problem, Seo et al. [22] design a non-local based memory attention to learn the inter-frame correlation. Hui et al. [10] consider the cross-modal property of referring video segmentation task and utilize language feature to build channel attention, which is applied to the weighted fusion of frames. In this paper, we leverage the language-guided spatial-temporal information to learn a set of position-specific and frame-specific dynamic filters, which can fully exploit the inter-modality and inter-frame information interaction to progressively combine the cross-modal aligned features at multiple scales for current frame.

We design a deeply interleaved two-stream encoder to achieve the mutual guidance and embedding between visual and linguistic features. It can not only encode the hierarchical semantic context of the sentence, but also gradually realize the mixing of multi-modal information from the feature extractors at different levels.

Specifically, we take ResNeSt [30] as the visual encoder. It contains five feature blocks, and their features are defined as $\{{\mathrm{V}_{i}}\}_{i=1}^{5}$. The query expression is denoted as $R=\{r_{t}\}^{T}_{t=1}$, where $t$ indicates the $t$-th word and $T$ is the total number of words. We utilize the Bert-embedding [3] to obtain the initial representation $\mathrm{L}_{0}=\{l_{t}^{0}\}^{T}_{t=1}$ of $R$. $\mathrm{L}_{0}$ is fed into three cascaded transformer blocks to encode the linguistic context $\{{\mathrm{L}_{j}}\}_{j=1}^{3}$.

To establish the connection between the two encoders, we propose a vision-language mutual guidance (VLMG) module to interlacedly guide their feature extraction. For the convenience of description, we define the sizes of $\mathrm{V}_{i}$ and $\mathrm{L}_{j}$ as $C\times H\times W$ and $C\times T$, respectively. $H$, $W$ and $C$ denote the height, width and channel number, respectively. The feature $\mathrm{V}_{i}$ is first reshaped into a matrix representation with size $C\times(HW)$. The vision-to-language affinity matrix between $\mathrm{V}_{i}$ and $\mathrm{L}_{j}$ can be calculated as follows:

$$\begin{split}&{\mathrm{A}_{\scriptscriptstyle vl}}^{\prime}=({\mathrm{W}_{v}^{1}\mathrm{V}_{i})}^{\top}({\mathrm{W}_{l}^{1}\mathrm{L}_{j})},\\ &{\mathrm{A}_{\scriptscriptstyle vl}}={\rm softmax}({\mathrm{A}_{\scriptscriptstyle vl}}^{\prime}),\\ \end{split}$$

(1)

where $\mathrm{W}_{v}^{1}$, $\mathrm{W}_{l}^{1}\in\mathbb{R}^{C_{1}\times C}$ are the learnable parameters. ${\mathrm{A}_{\scriptscriptstyle vl}}\in\mathbb{R}^{(HW)\times T}$ describes the similarity between each pixel of $\mathrm{V}_{i}$ and each word of $\mathrm{L}_{j}$. The softmax function is used to normalize the affinity matrix along the first dimension. We utilize ${\mathrm{A}_{\scriptscriptstyle vl}}$ to project the visual feature $\mathrm{V}_{i}$ to the language space as follows:

$$\begin{split}&\widetilde{\mathrm{V}}_{i}=\mathrm{V}_{i}{\mathrm{A}_{\scriptscriptstyle vl}}.\\ \end{split}$$

(2)

Next, the features $\widetilde{\mathrm{V}}_{i}\in\mathbb{R}^{C\times T}$ and $\mathrm{L}_{j}\in\mathbb{R}^{C\times T}$ are fed into the standard transformer structure to learn their co-embedding:

$$\begin{split}&\widehat{\mathrm{L}}_{j}=\mathrm{MSA}(\mathrm{LN}(\widetilde{\mathrm{V}}_{i}+\mathrm{L}_{j}))+(\widetilde{\mathrm{V}}_{i}+\mathrm{L}_{j}),\\ &\mathrm{L}_{j+1}=\mathrm{FFN}(\mathrm{LN}(\widehat{\mathrm{L}}_{j}))+\widehat{\mathrm{L}}_{j},\end{split}$$

(3)

where $\mathrm{MSA}$ denotes the multi-head self-attention module. $\mathrm{LN}$ represents the LayerNorm layer. FFN is the feed forward network. We further use $\mathrm{L}_{j+1}\in\mathbb{R}^{C\times T}$ and $\mathrm{V}_{i}$ to learn the affinity matrix of language-to-vision:

$$\begin{split}&{\mathrm{A}_{\scriptscriptstyle lv}}^{\prime}=({\mathrm{W}_{v}^{2}\mathrm{V}_{i})}^{\top}({\mathrm{W}_{l}^{2}\mathrm{L}_{j+1})},\\ &{\mathrm{A}_{\scriptscriptstyle lv}}={\rm softmax}({{\mathrm{A}_{\scriptscriptstyle lv}}^{\prime}}^{\top}),\\ \end{split}$$

(4)

where $\mathrm{W}_{v}^{2}$, $\mathrm{W}_{l}^{2}$ are the learned parameters. Similarly, the column-wise normalized affinity matrix ${\mathrm{A}_{\scriptscriptstyle lv}}\in\mathbb{R}^{T\times(HW)}$ is utilized to map the mixed multi-modal feature $\mathrm{L}_{j+1}$ to the vision space:

$$\begin{split}&\widetilde{\mathrm{L}}_{j+1}=\mathrm{L}_{j+1}\mathrm{A}_{\scriptscriptstyle lv}.\\ \end{split}$$

(5)

Finally, we adopt concatenation $Cat(\cdot,\cdot)$ and convolution $Conv$ to incorporate the feature $\widetilde{\mathrm{L}}_{j+1}\in\mathbb{R}^{C\times H\times W}$ into the visual encoder:

$$\begin{split}&\overline{\mathrm{V}}_{i}=\mathrm{Norm}(\mathrm{V}_{i})+\mathrm{Norm}(Conv(Cat(\mathrm{V}_{i},\widetilde{\mathrm{L}}_{j+1}))),\\ \end{split}$$

(6)

where the $\mathrm{Norm}$ represents the L2-normalize, and its purpose is to normalize the feature maps to the same order of magnitude to avoid preference bias. The updated multi-modal feature $\overline{\mathrm{V}}_{i}$ is fed to the next convolution block of the visual encoder for further feature extraction. Fig. 3 illustrates the detailed structure of VLMG. By embedding VLMG between the visual CNN encoder and the linguistic transformer encoder, the multi-modal interaction process runs through the entire network. And most of the previous global attention methods [25, 8] need to calculate an affinity matrix with the size of $HW\times HW$, so increasing the size of the input image will lead to a sharp increase in computational cost. The size of the affinity matrix calculated in VLMG is much smaller than $HW\times HW$ (e.g. $(HW)\times T$ in Eq. 1 and Eq. 4, $T\times T$ in Eq. 3). This also makes our encoder more suitable for real applications.

For referring video segmentation, the language expression contains important spatial-temporal information related to the video. Therefore, we can use it to guide the cross-frame feature fusion. Let $\overline{\mathrm{V}}^{r}\in\mathbb{R}^{C\times H\times W}$ and $\overline{\mathrm{V}}^{c}\in\mathbb{R}^{C\times H\times W}$ indicate the features of the reference frame and the current frame, respectively. They are generated by Eq. 6. To capture useful spatial-temporal information from the reference frame and the current frame through the guidance of linguistic features, we firstly calculate the affinity matrix ${\mathrm{A}_{\scriptscriptstyle r}}\in\mathbb{R}^{(HW)\times T}$ between $\overline{\mathrm{V}}^{r}$ and $\mathrm{L}_{0}$, as:

$$\begin{split}&{\mathrm{A}_{\scriptscriptstyle r}}={\rm softmax}(({\mathrm{W}_{v}^{3}\overline{\mathrm{V}}^{r})}^{\top}({\mathrm{W}_{l}^{3}\mathrm{L}_{0})}).\\ \end{split}$$

(7)

The column-wise normalized similarity matrix is used to map the reference feature $\overline{\mathrm{V}}^{r}$ to the language space, as:

$$\begin{split}&\widetilde{\mathrm{V}}^{r}=\overline{\mathrm{V}}^{r}{\mathrm{A}_{\scriptscriptstyle r}}.\\ \end{split}$$

(8)

We further adopt the transformer block to mix $\widetilde{\mathrm{V}}^{r}\in\mathbb{R}^{C\times T}$ and $\mathrm{L}_{0}\in\mathbb{R}^{C\times T}$ and obtain the reference-frame modulated language feature $\mathrm{L}^{r}\in\mathbb{R}^{C\times T}$:

$$\begin{split}&\widehat{\mathrm{L}}^{r}=\mathrm{MSA}(\mathrm{LN}(\widetilde{\mathrm{V}}^{r}+\mathrm{L}_{0}))+(\widetilde{\mathrm{V}}^{r}+\mathrm{L}_{0}),\\ &\mathrm{L}^{r}=\mathrm{MLP}(\mathrm{LN}(\widehat{\mathrm{L}}^{r}))+\widehat{\mathrm{L}}^{r}.\end{split}$$

(9)

Then, we superpose the current frame $\overline{\mathrm{V}}^{c}$ to further compute a multi-frame modulated language feature $\mathrm{L}^{c}\in\mathbb{R}^{C\times T}$ as follows:

$$\begin{split}&\widetilde{\mathrm{V}}^{c}=\overline{\mathrm{V}}^{c}({\rm softmax}(({\mathrm{W}_{v}^{4}\overline{\mathrm{V}}^{c})}^{\top}({\mathrm{W}_{l}^{4}\mathrm{L}^{r}))}),\\ &\widehat{\mathrm{L}}^{c}=\mathrm{MSA}(\mathrm{LN}(\widetilde{\mathrm{V}}^{c}+\mathrm{L}^{r}))+(\widetilde{\mathrm{V}}^{c}+\mathrm{L}^{r}),\\ &\mathrm{L}^{c}=\mathrm{MLP}(\mathrm{LN}(\widehat{\mathrm{L}}^{c}))+\widehat{\mathrm{L}}^{c}.\end{split}$$

(10)

Thus, under the guidance of language, the feature $\mathrm{L}^{c}$ successively fuses the language related spatial-temporal information from the reference frame and the current frame.

In order to generate the position-specific dynamic filters to accurately calibrate the current-frame feature based on the inter-frame and inter-modality information, we in advance learn the position-adaptive guidance feature based on the feature $\mathrm{L}^{c}$:

$$\begin{split}&\mathchoice{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\displaystyle{{\mathrm{V}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\textstyle{{\mathrm{V}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptstyle{{\mathrm{V}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptscriptstyle{{\mathrm{V}}}}^{c}=\mathrm{L}^{c}({\rm softmax}((({\mathrm{W}_{v}^{5}\overline{\mathrm{V}}^{c})}^{\top}({\mathrm{W}_{l}^{5}\mathrm{L}^{c}))^{\top})}),\\ \end{split}$$

(11)

where $\mathchoice{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\displaystyle{{\mathrm{V}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\textstyle{{\mathrm{V}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptstyle{{\mathrm{V}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptscriptstyle{{\mathrm{V}}}}^{c}\in\mathbb{R}^{C\times H\times W}$. $\mathchoice{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\displaystyle{{{v}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\textstyle{{{v}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptstyle{{{v}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptscriptstyle{{{v}}}}^{c}_{i,j}\in\mathbb{R}^{C}$ and ${\overline{v}}^{c}_{i,j}\in\mathbb{R}^{C}$ represent the vectors of position $(i,j)$ in $\mathchoice{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\displaystyle{{\mathrm{V}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\textstyle{{\mathrm{V}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptstyle{{\mathrm{V}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptscriptstyle{{\mathrm{V}}}}^{c}$ and $\overline{\mathrm{V}}^{c}$, respectively. Then $\mathchoice{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\displaystyle{{{v}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\textstyle{{{v}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptstyle{{{v}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptscriptstyle{{{v}}}}^{c}_{i,j}$ is used to generate a set of dynamic kernels to filter the features of the neighborhood around $(i,j)$:

$$\begin{split}&{v_{d}}^{\prime}=\sum\limits_{k=-1}^{1}\sum\limits_{l=-1}^{1}(w_{k,l}\mathchoice{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\displaystyle{{{v}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\textstyle{{{v}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptstyle{{{v}}}}{\overarrow@\arrowfill@\relbar\relbar\rightharpoonup\scriptscriptstyle{{{v}}}}^{c}_{i,j})\odot{\overline{v}}^{c}_{i+k\times d,j+l\times d},\\ \end{split}$$

(12)

where $w_{k,l}\in\mathbb{R}^{C\times C}$ is the learnable parameter. $\odot$ denotes the element-wise multiplication. Actually, Eq.12 can be understood as a 3$\times$3 depth-wise convolution with dilated rate $d$. During the process of filtering, the multi-scale neighborhood is adopted. We then concatenate $\overline{v}^{c}$ and multiple ${v_{d}}^{\prime}$ of different dilated rates and follow a convolution layer to combine them. Finally, the resulted cross-frame multi-modal features from stage3$\sim$stage5 and the visual features from stage1$\sim$stage2 are feed into the decoder (feature pyramid network) to predict the final segmentation result.

In order to illustrate the effectiveness of the proposed method, we conduct extensive experiments on four referring video segmentation datasets, which are A2D Sentences [6], J-HMDB Sentences [6], Refer-DAVIS2017 [13], and Refer-Youtube-VOS [22]. Gavrilyuk et al. [6] extended the original Actor-Action Dataset (A2D) [26] and J-HMDB [12] datasets to the referring video segmentation task by adding additional natural language descriptions. A2D Sentences is composed of 3,782 videos with 6,655 referring expressions. And this videos contain 8 different actions that are preformed by 7 actor classes. Following the previous setting [25, 10], we split A2D into two parts, 3,017 of which are used for training and 737 for testing. Each video contains 3 to 5 frame pixel-level segmentation masks. For J-HMDB Sentences, it consists of 928 videos with 21 different action classes. Each video contains a corresponding language expression. And the main purpose of this dataset is to evaluate the generalization ability of the model. Generally, previous methods directly use the model trained on the A2D dataset to test all the videos in this dataset. Similarly, Refer-DAVIS2017 is based on DAVIS2017 [21]. It contains 90 videos, 60 for training and 30 for testing. Refer-Youtube-VOS is by far the largest dataset, it contains a total of 3,978 videos and about 15,000 language expressions.

We compare the performance of our method and the previous state-of-the-art methods on four different datasets. First of all, for the A2D Sentences dataset, we can find that the proposed method is significantly better than the previous methods. This also reflects the effectiveness of the two-stream encoder and language-guided dynamic filtering. For all the evaluation metrics in Tab. 1, our model achieves an average absolute gain of more than 5.5%. Especially on Prec@0.6, Prec@0.7 and Prec@0.8, our method obtains absolute enhancement of 7.1%, 7.9% and 8.6% compared with the second-best method. For the metric Prec@0.9, our method improves it from 9.8% to 15.1%, which is 1.5 times the previous performance. This also proves that our network can more accurately perceive the boundary of the object.

Following the previous methods [10, 17], we further use the J-HMDB Sentences dataset to verify the generalization ability of the model trained on the A2D Sentences dataset. As shown in Tab. 2, we can find that the proposed model achieves the best performance under all evaluation metrics. Particularly, our network shows a very significant improvement of 19.7% on Prec@0.7. For the Refer-DAVIS2017, it only contains 60 training videos, so the previous methods [13, 22] usually take advantage of RefCOCO [29] for data augmentation. Our model achieves the gain of 6.5% and 5.3% in terms of $\mathcal{J}$ and $\mathcal{F}$. Finally, Tab. 4 illustrates the accuracy on the large-scale dataset Refer-Youtube-VOS. Our model surpasses other methods in all evaluation metrics. We also present some visual cases in Fig. 4. It can be seen that our method can produce accurate segmentation masks, even when the language expression does not contain location information or the length of query is various.

We carefully evaluate the proposed components on the Refer-DAVIS2017 dataset. The quantitative results are shown in Tab. 5 and Tab. 6.