Temporal Contrastive Graph Learning for Video Action Recognition and Retrieval

For video representation learning, a large number of supervised learning methods have been received increasing attention. The methods include traditional methods [23, 20, 49, 50, 35, 39, 27] and deep learning methods [41, 44, 54, 28, 45, 55, 66, 26, 29]. To model and discover temporal knowledge in videos, two-stream CNNs [41] judged the video image and dense optical flow separately, then directly fused the class scores of these two networks to obtain the classification result. C3D [44] processed videos with a three-dimensional convolution kernel. Temporal Segment Networks (TSN) [55] sampled each video into several segments to model the long-range temporal structure of videos. Temporal Relation Network (TRN) [66] introduced an interpretable network to learn and reason about temporal dependencies between video frames at multiple temporal scales. Temporal Shift Module (TSM) [26] shifted part of the channels along the temporal dimension to facilitate information exchanged among neighboring frames. Although these supervised methods achieve promising performance in modeling temporal dependencies, they require large amounts of labeled videos for training an elaborate model, which is time-consuming and labor-intensive.

Although there exists a large amount of videos, it may take a great effort to annotate such massive data. Self-supervised learning generates various pretext tasks to leverage abundant unlabeled data. The learned model from pretext tasks can be directly applied to downstream tasks for feature extraction or fine-tuning. Specific contrastive learning methods have been proposed, such as the NCE [11], MoCo [15], BYOL [10], SimCLR [3]. To better model topologies, contrastive learning methods on graphs [40, 68, 12] have also been attracted increasing attention.

For self-supervised video representation learning, how to effectively explore temporal information is important. Many existing works focus on the discovering of temporal information. Shuffle&Learn [34] randomly shuffled video frames and trained a network to distinguish whether these video frames are in the right order or not. Odd-one-out Network [8] proposed to identify unrelated or odd video clips. Order prediction network (OPN) [25] trained networks to predict the correct order of shuffled frames. VCOP [61] used 3D convolutional networks to predict the orders of shuffled video clips. SpeedNet [1] designed a network to detect whether a video is playing at a normal rate or sped up rate. Video-pace [53] utilized a network to identify the right paces of different video clips. In addition to focusing on the temporal dependency, Mas [52] proposed a self-supervised learning method by regressing both motion and appearance statistics along spatial and temporal dimensions. ST-puzzle [18] used space-time cubic puzzles to design pretext task. IIC [43] introduced intra-negative samples by breaking temporal relations in video clips, and use these samples to build an inter-intra contrastive framework. Though the above works utilize temporal dependency or design specific pretext tasks for video self-supervised learning, the comprehensive temporal diversity and chronological characteristics are not fully explored. In our work, we build a novel inter-intra snippet graph structure to model multi-scale temporal dependencies, and produce self-supervision signals about video snippet orders contrastively.

In this stage, we randomly sample consecutive frames (snippets) from the video to construct video snippet tuples. If we sample $N$ snippets from a video, there are $N!$ possible snippet orders. Since the snippet order prediction is purely a proxy task of the TCGL and our focus is the learning of 3D CNNs, we restrict the number of snippets of a video between $3$ to $4$ to alleviate the complexity of the order prediction task, inspired by the previous works [36, 61, 60]. The snippets are sampled uniformly from the video with the interval of $p$ frames. After sampling, the snippets are shuffled to form the snippet tuples $\mathbf{S}=\langle s_{1},s_{2},\cdots,s_{n}\rangle$. For each snippet $s_{i}$, all the frames within are uniformly divided into $m$ frame-sets with equal length, then we get the frame-set $\mathbf{F}_{i}=\langle f_{1},f_{2},\cdots,f_{m}\rangle$ for snippet $s_{i}$. For snippet tuples, they contain dynamic information and strict chronological relations of a video, which is essentially the long-term temporal dependency of the videos. For the frame-sets within a snippet, the frame-level temporal relation among frames provides us the short-term temporal dependency of the videos. By taking both long-term and short-term temporal dependencies into consideration, we can increase the temporal diversity more comprehensively and precisely.

To extract spatio-temporal features, we choose C3D [44], R3D [45] and R(2+1)D [45] as feature encoders. The same 3D CNNs are used for all snippets and frame-sets, as Figure 2 (b) shows. C3D is an extension from 2D CNNs for spatio-temporal representation learning since it can model the temporal information and dynamics of the videos. C3D network consists of $8$ convolutional layers stacked one by one, with $5$ pooling layers interleaved, and followed by two fully connected layers. The size of all convolutional kernels is $3\times 3\times 3$, which is validated in previous work [44]. R3D is the 3D CNNs with residual connections. R3D block consists of two 3D convolutional layers followed by batch normalization and ReLU layers. The input and output are connected with a residual unit before the ReLU layer. R(2+1)D is similar to R3D. The 3D convolution is decomposed into two separate operations, the one is 2D spatial convolution, and the other is 1D temporal convolution.

Due to the effectiveness of graph convolutional network (GCN) [19, 68, 59] in modeling unregular graph-structured relationship, we use it to explore node interaction within each snippet and frame-set for modeling multi-scale temporal dependencies of videos. After obtaining the feature vectors for snippets and frame-sets, we construct two kinds of temporal contrastive graph structures: inter-snippet and intra-snippet temporal contrastive graphs, to increase the temporal diversity of videos, as shown in Figure 2 (c).

To build intra-snippet and inter-snippet temporal contrastive graphs, we take advantage of prior knowledge about the chronological relation and the corresponding feature vectors. To fix notation, we denote intra-snippet and inter-snippet graphs as $\mathbf{G}_{intra}^{k}=f(\mathbf{X}_{intra}^{k},\mathbf{A}_{intra}^{k})$ and $\mathbf{G}_{inter}=f(\mathbf{X}_{inter},\mathbf{A}_{inter})$, respectively, where $k=1,\cdots,m$, $m$ is the number of frame-sets in a video snippet. As shown in Figure 3, the prior knowledge that the correct order of frames in frame-sets, and the correct order of snippets in snippet tuples are already known because our proxy task is video snippet order prediction. Therefore, we can utilize the prior chronological relationship to determine the edges of graphs. For example, in Figure 3, if we know that snippets (frame-sets) are ranking chronologically $1\rightarrow 2\rightarrow 3\rightarrow 4$, we can connect the temporally related nodes and disconnect the temporally unrelated nodes. Here, we take inter-snippet temporal graph $\mathbf{G}_{inter}$ as an example to clarify our temporal contrastive graph learning method.

For $\mathbf{G}_{inter}$, we randomly remove edges and masking node features to generate two graph views $\mathbf{\widetilde{G}}_{inter}^{1}$ and $\mathbf{\widetilde{G}}_{inter}^{2}$, and the node embeddings of two generated views are denoted as $\mathbf{U}=f(\mathbf{\widetilde{X}}_{inter}^{1},\mathbf{\widetilde{A}}_{inter}^{1})$ and $\mathbf{V}=f(\mathbf{\widetilde{X}}_{inter}^{2},\mathbf{\widetilde{A}}_{inter}^{2})$. Since different graph views provide different contexts for each node, we corrupt the original graph at both structure and attribute levels to achieve contrastive learning between node embeddings from different views. Therefore, we propose two strategies for generating graph views: removing edges and masking node features.

The edges in the original graph are randomly removed using a random masking matrix $\mathbf{\widetilde{R}}\in\{0,1\}^{N\times N}$, where the entry of $\mathbf{\widetilde{R}}$ is drawn from a Bernoulli distribution $\mathbf{\widetilde{R}}_{ij}\sim\mathcal{B}(1-p_{r})$ if $\mathbf{A}_{ij}=1$ for the original graph, and $\mathbf{\widetilde{R}}_{ij}=0$ otherwise. Here $p_{r}$ denotes the probability of each edge being removed. Then, the resulting adjacency matrix can be computed as follows, where $\circ$ is Hadamard product.

$$\mathbf{\widetilde{A}}=\mathbf{A}\circ\mathbf{\widetilde{R}}$$

(1)

In addition, a part of node features is masked with zeros using a random vector $\mathbf{\widetilde{m}}\in\{0,1\}^{F}$, where each dimension of it is drawn from a Bernoulli distribution $\widetilde{m}_{i}\sim\mathcal{B}(1-p_{m}),\forall i$. Then, the generated masked features $\mathbf{\widetilde{X}}$ is calculated as follows:

$$\mathbf{\widetilde{X}}=[\mathbf{x}_{1}\circ\mathbf{\widetilde{m}};\mathbf{x}_{2}\circ\mathbf{\widetilde{m}};\cdots;\mathbf{x}_{N}\circ\mathbf{\widetilde{m}}]^{\top}$$

(2)

where $[\cdot;\cdot]$ is the concatenation operator. We jointly leverage these two strategies to generate graph views.

Inspired by the NCE loss [11], we propose a contrastive learning loss that distinguishes embeddings of the same node from these two distinct views from other node embeddings. Given a positive pair, the negative samples come from all other nodes in the two views (inter-view or intra-view). To compute the relationship of embeddings $\mathbf{u}$, $\mathbf{v}$ from two views, we define the relation function $\phi(\mathbf{u},\mathbf{v})=f(g(\mathbf{u}),g(\mathbf{v}))$, where $f$ is the L2 normalized dot product similarity, and $g$ is a non-linear projection with two-layer multi-layer perception. The pairwise contrastive objective for positive pair $(\mathbf{u}_{i},\mathbf{v}_{i})$ is defined as:

$$\ell(\mathbf{u}_{i},\mathbf{v}_{i})=\frac{e^{\frac{\phi(\mathbf{u_{i}},\mathbf{v_{i}})}{\tau}}}{e^{\frac{\phi(\mathbf{u_{i}},\mathbf{v_{i}})}{\tau}}+\sum\limits_{k=1}^{N}{\mathbf{\mathbb{I}}_{[k\neq i]}e^{\frac{\phi(\mathbf{u_{i}},\mathbf{v_{k}})}{\tau}}}+e^{\frac{\phi(\mathbf{u_{i}},\mathbf{u_{k}})}{\tau}}}$$

(3)

where $\mathbf{\mathbb{I}}_{[k\neq i]}\in\{0,1\}$ is an indicator function that equals to $1$ if $k\neq i$, and $\tau$ is a temperature parameter, which is empirically set to $0.5$. The first term in the denominator represents the positive pairs, the second term represents the inter-view negative pairs, the third term represents the intra-view negative pairs. Since two views are symmetric, the loss for another view is defined similarly for $\ell(\mathbf{v}_{i},\mathbf{u}_{i})$. The overall contrastive loss for $\mathbf{G}_{inter}$ is defined as follows:

$$\mathcal{J}_{inter}=\frac{1}{2N}\sum\limits_{i=1}^{N}[\ell(\mathbf{u}_{i},\mathbf{v}_{i})+\ell(\mathbf{v}_{i},\mathbf{u}_{i})]$$

(4)

The contrastive loss for intra-snippet graphs $\mathbf{G}_{intra}^{k}(k=1,\cdots,m)$ can be computed similarly as $\mathbf{G}_{inter}$.

$$\mathcal{J}_{intra}^{k}=\frac{1}{2N}\sum\limits_{i=1}^{N}[\ell^{k}(\mathbf{u}_{i},\mathbf{v}_{i})+\ell^{k}(\mathbf{v}_{i},\mathbf{u}_{i})]$$

(5)

Then, the overall temporal contrastive graph loss is defined as follows, where $\alpha$ and $\beta$ are the weights for inter-snippet graph and intra-snippet graph, respectively.

$$\mathcal{J}_{g}=\alpha\sum\limits_{k=1}^{m}\mathcal{J}_{intra}^{k}+\beta\mathcal{J}_{inter}$$

(6)

We formulate the order prediction task as a classification task using the learned video snippet features from the temporal contrastive graph as the input and the probability distribution of orders as output. Since the features from different video snippets are correlated, we build an adaptive order prediction module that receives features from different video snippets and learns a global context embedding, then this embedding is used to recalibrate the input features from different snippets, as shown in Figure 4.

To fix notation, we assume that a video is shuffled into $n$ snippets, the snippet features of nodes learned from inter-snippet temporal contrastive graph learning are $\{\mathbf{f}_{1},\cdots,\mathbf{f}_{n}\}$, where $\mathbf{f}_{k}\in R^{c_{k}}(k=1,\cdots,n$). To utilize the correlation among these snippets, we concatenate these feature vectors and obtain joint representations through a fully-connected layer:

$$\mathbf{Z}=\mathbf{W}_{s}[\mathbf{f}_{1},\cdots,\mathbf{f}_{n}]+\mathbf{b}_{s}$$

(7)

where $[\cdot,\cdot]$ denotes the concatenation operation, $\mathbf{Z}\in\mathbb{R}^{c_{con}}$ denotes the joint representation, $\mathbf{W}_{s}$ and $\mathbf{b}_{s}$ are weights and bias of the fully-connected layer. We choose $c_{con}=\frac{\sum_{k=1}^{n}c_{k}}{2n}$ to restrict the model capacity and increase its generalization ability. To make use of the global context information aggregated in the joint representations $\mathbf{Z}_{con}$, we predict excitation signal for it by a fully-connected layer:

$$\mathbf{E}=\mathbf{W}_{e}\mathbf{Z}+\mathbf{b}_{e}$$

(8)

where $\mathbf{W}_{e}$ and $\mathbf{b}_{e}$ are weights and biases of the fully-connected layer. After obtaining the excitation signal $\mathbf{E}\in\mathbb{R}^{c}$, we use it to recalibrate the input feature $f_{k}$ adaptively by a simple gating mechanism:

$$\mathbf{\widetilde{f}}_{k}=\delta(\mathbf{E})\odot\mathbf{f}_{k}$$

(9)

where $\odot$ is channel-wise product operation for each element in the channel dimension, and $\delta(\cdot)$ is the ReLU function. In this way, we can allow the features of one snippet to recalibrate the features of another snippet while concurrently preserving the correlation among different snippets.

Finally, these refined features $\{\mathbf{\widetilde{f}}_{1},\cdots,\mathbf{\widetilde{f}}_{n}\}$ are concatenated and fed into two-layer perception with soft-max to output the snippet order prediction. The cross-entropy loss is used to measure the correctness of the prediction:

$$\mathcal{J}_{o}=-\sum_{i=1}^{C}\mathbf{y}_{i}\textrm{log}(\mathbf{p}_{i})$$

(10)

where $y_{i}$ and $p_{i}$ represent the probability that the sample belongs to the order class $i$ in ground-truth and prediction, respectively. $C$ denotes the number of all possible orders.

The overall self-supervised learning loss for TCGL is obtained by combing Eq (6) and Eq (10), where $\lambda_{g}$ and $\lambda_{o}$ control the contribution of $\mathcal{J}_{g}$ and $\mathcal{J}_{o}$, respectively.

$$\mathcal{J}=\lambda_{g}\mathcal{J}_{g}+\lambda_{o}\mathcal{J}_{o}$$

(11)

Datasets. We evaluate our method on three action recognition datasets, UCF101 [42], HMDB51 [22] and Kinetics-400 [17]. UCF101 is collected from websites containing $101$ action classes with $9.5$k videos for training and $3.5$k videos for testing. HMDB51 is collected from various sources with $51$ action classes and $3.4$k videos for training and $1.4$k videos for testing. Kinetics-400 is a large-scale action recognition dataset, which contains 400 action classes and around 306k videos. In this work , we use the training split (around 240k videos) as the pre-training dataset.

Network Architecture. We use PyTorch [37] to implement the whole framework. For video encoder, C3D, R3D and R(2+1)D are used as backbones, where the kernel size of 3D convolutional layers is set to $3\times 3\times 3$. The R3D network is implemented with no repetitions in conv{2-5}_x, which results in $9$ convolution layers in total. The C3D network is modified by replacing the two fully connected layers with global spatiotemporal pooling layers. The R(2+1)D network has the same architecture as the R3D network with only 3D kernels decomposed. Dropout layers are applied between fully-connected layers with $p=0.1$. Our GCN for both inter-snippet and intra-snippet graphs consist of one graph convolutional layer with $512$ output channels.

Parameters. Following the settings in [61, 62], we set the snippet length of input video as $16$, the interval length is set as $8$, the number of snippets per tuple is $3$, and the number of frame-sets within each snippet is $4$. During training, we randomly split $800$ videos from the training set as the validation set. Video frames are resized to $128\times 171$ and then randomly cropped to $112\times 112$. We set the parameters $\lambda_{g}=\lambda_{o}=1$ to balance the contribution between temporal contrastive graph module and adaptive order prediction module. The weights $\alpha$ and $\beta$ are both set to $1$ according to the ablation study result. To optimize the framework, we use mini-bach stochastic gradient descent with the batchsize $16$, the initial learning rate $0.001$, the momentum $0.9$ and the weight decay $0.0005$. The training process lasts for $300$ epochs and the learning rate is decreased to $0.0001$ after $150$ epochs. To make temporal contrastive graphs sensitive to subtle variance between different graph views, the parameters $p_{r}$ and $p_{m}$ for generating graph view 1 are empirically set to $0.2$ and $0.1$, and $p_{r}=p_{m}=0$ for generating graph view 2. And the values of $p_{r}$ and $p_{m}$ are the same for both inter-snippet and intra-snippet graphs. The model with the lowest validation loss is saved to the best model.

In this subsection, we conduct ablation studies on the first split of UCF101 with R3D as the backbone, to analyze the contribution of each component of our TCGL.

The number of snippets. The results of R3D on the snippet order prediction task with different number of snippets are shown in Table 1. The prediction accuracy decreases when the number of snippets increases because the difficulty of the prediction task grows when the snippets number increase, which makes the model hard to learn. Therefore, we use $3$ snippets per video to make a compromise between task complexity and prediction accuracy.

The number of frame-sets. Since the snippet length is $16$, the number of frame-sets within each snippet can be $1,2,4,8,16$. When the number is $16$, the frame-set only contains static information without temporal information. When the number is $1$ or $2$, it is hard to model the intra-snippet temporal relationship with too few frame-sets. From Table 2, we can observe that more frame-sets within a snippet will make the intra-snippet temporal modeling more difficult, which degrades the order prediction performance. Therefore, we choose $4$ frame-sets per snippet in the experiments for short-term temporal modeling.

The intra-snippet and inter-snippet graphs. To analyze the contribution of intra-snippet and inter-snippet temporal contrastive graphs, we set different values to $\alpha$ and $\beta$, shown in Table 3. To be noticed, removing intra-snippet graph will degrade the performance significantly even with the inter-snippet graph, which verifies the importance of intra-snippet graphs for modeling short-term temporal dependency. Additionally, when setting the weight values of intra-snippet graph and inter-snippet graph to $1$ and $0.1$, the prediction accuracy is $80.2\%$. While exchanging their weight values, the accuracy drops to $54.9\%$, which validates the importance of intra-snippet graphs in modeling short-term temporal dependency. In addition, the performance of TCGL drops significantly when removing either or both of the graphs. When $\alpha=\beta=1$, the prediction accuracy is the best ($83.0\%$). These results validate that both intra-snippet and inter-snippet temporal contrastive graphs are essential for increasing the temporal diversity of features.

The adaptive snippet order prediction. To analyze the contribution of our proposed adaptive snippet order prediction (ASOR) module, we remove this module and merely feed the concatenated features into multi-layer perception with soft-max to output the final snippet order prediction. It can be observed in Table 4 that our TCGL performs better than the TCGL without the ASOR module in both order prediction and action recognition tasks. This verifies that the ASOR module can better utilize relational knowledge among video snippets than simple concatenation.

The methods of introducing noises. To justify our superiority on modeling both feature and topology levels, we make comparisons with three simple graph corruption methods: (1) adding random Gaussian noise; (2) randomly removing edges; (3) randomly removing nodes. The (order prediction, action recognition) accuracies (%) for them are ($56.1,51.8$), ($55.2,53.4$) and ($54.9,52.7$), respectively, while we achieve ($80.2,67.6$), justifying the our superiority on modeling both feature and topology levels.

To verify the effectiveness of our TCGL in action recognition, we initialize the backbones with the model pretrained on the first split of UCF101 or the whole K400 training-set, and fine-tune on UCF101 and HMDB51, the fine-tuning stops after $150$ epochs. The features extracted by the backbones are fed into fully-connected layers to obtain the prediction. For testing, we sample $10$ clips for each video to generate clip predictions, and then average these predictions to obtain the final prediction results. The average classification accuracy over three splits is reported and compared with other self-supervised methods in Table 5. The “Random” means the model is randomly initialized without pre-training. When pre-trained on UCF101, we outperform the current best-performing method PRP [62]. When pre-trained on K400, we outperform the current best-performing method V-pace [53]. In addition, we consistently outperform the other state-of-the-art methods and random initialization method for all evaluation metrics. Furthermore, when pre-trained on UCF101, we achieve better accuracies than some K400 pre-trained methods (Mas [52] and ST-puzzle [18]). These results validate that our TCGL can effectively increase the temporal diversity of videos and learn discriminative spatio-temporal representations.

To further verify the effectiveness of our TCGL in video retrieval, we test our TCGL on the nearest-neighbor video retrieval. Since the video retrieval task is conducted with features extracted by the backbone network without fine-tuning, its performance largely relies upon the representative capacity of self-supervised model. The experiment is conducted on the first split of UCF101 or the whole training-set of K400, following the protocol in [61, 62]. In video retrieval, we extract video features from the backbone pre-trained by TCGL. Each video in the testing set is used to query $k$ nearest videos from the training set using the cosine distance. When the class of a test video appears in the classes of $k$ nearest training videos, it is considered as the correct predicted video. We show top-1, top-5, top-10, top-20, and top-50 retrieval accuracies on UCF101 and HMDB51 datasets, and compare our method with other self-supervised methods, as shown in Table 6 and 7. For all backbones, our TCGL outperforms the state-of-the-art methods on nearly all evaluation metrics by substantial margins. Figure 5 visualizes a query video snippet and its top-3 nearest neighbors from the UCF101 training set using the TCGL embedding. It can be observed that the representation learned by the TCGL has the ability to retrieve videos with the same semantic meaning. To have a better understanding of what TCGL learns, we follow the Class Activation Map [67] to visualize the spatio-temporal regions, as shown in Figure 6. These examples exhibit a strong correlation between highly activated regions and the dominant movement in the scene. This validates that our TCGL can learn discriminative temporal representations for videos.