Augmented 2D-TAN: A Two-stage Approach for Human-centric Spatio-Temporal Video Grounding

Review of 2D-TAN. 2D-TAN is a video grounding algorithm which first splits the input video into $N$ clips and extracts visual features¹¹1We use SlowFast Network[2] pretrained on Kinetics-600[[1] and AVA[3] to extract visual features. for each clip. Then dense moment (a sequence of clips) proposals $\mathcal{P}$ are generated as $\mathcal{P}=\{P_{ij}=\left[C_{i},C_{i+1},...,C_{j}\right]|1\leq i\leq j\leq N\}$ where moment $P_{ij}$ contains the $i$-th clip $C_{i}$ to the $j$-th clip $C_{j}$. The feature of each moment proposal is extracted by a moment feature aggregation module $MFA(\cdot)$. Then a moment-level 2D feature map $M\in\mathbb{R}^{N\times N\times c}$ is constructed as:

where $F_{i}$ is the feature of clip $C_{i}$. Then a Temporal Adjacent Network is employed to infer moment-wise matching score between $M$ and the sentence feature. The moment with the largest matching score is outputted as the grounding result. In the original implementation of 2D-TAN, $MFA(\cdot)$ is instantiated as the max-pooling operation which cannot model long-term temporal variation. Hence, the captured moment features are smoothed and the features of adjacent moments can be very similar, i.e., not discriminative. We propose a Bi-LSTM moment feature aggregation module to learn a more discriminative feature representation for each moment. And we further propose a Random Concatenation Augmentation strategy to augment the training data. We will introduce the Bi-LSTM module and the augmentation strategy in the following.

Bi-LSTM Aggregation Module. As shown in Figure 2, the Bi-LSTM Aggregation Module consists of a forward LSTM and a backward LSTM. They are both effective for modeling the temporal variation in each moment. Thus, compared to the original 2D-TAN, the moment features learned with our Bi-LSTM aggegation module are more suitable for calculating the cross-modal matching score with the sentence feature. Note that the parameters of our Bi-LSTM are shared across all moments, so we can implement the inference in parallel to improve efficiency.

Random Concatenation Augmentation. We observe severe overfitting in the original implementation of 2D-TAN and propose to randomly concatenate training samples to augment the training data. Specifically, we randomly select two videos (with similar frame rate) $V,V^{\prime}$ and their corresponding language queries $Q,Q^{\prime}$. We generate the augmented video as $V_{aug}=\{V_{(t)}\}_{t=\tau_{s}-\delta_{1}}^{\tau_{e}+\delta_{2}}||\{V^{\prime}_{(t)}\}_{t=\tau_{s}^{\prime}-\delta^{\prime}_{1}}^{\tau_{e}^{\prime}+\delta^{\prime}_{2}}$, where $\tau_{s}$, $\tau_{e}$ indicate the start frame and end frame index of the ground truth moment, respectively. $\delta_{1},\delta_{2}$ are random offsets and we ensure the number of frames in $V_{aug}$ is similar to that of $V$. $||$ represents concatenation operation along the temporal dimension. We randomly assign $Q$ or $Q^{\prime}$ as the query of $V_{aug}$. During training, we perform this augmentation with a probability of 0.5 at each training step. The proposed augmentation strategy greatly enriches the training data and reduces the risk that the model simply learns to ground the salient segment.