Enhancing Unsupervised Video Representation Learning
by Decoupling the Scene and the Motion

The most significant characteristic of the video representation learning is the requirement for temporal modeling compared to the image representation learning. Early works first use 2D CNNs to capture appearance features at the frame level and then do average pooling or adopt LSTM over the temporal dimension to learn motion patterns(Wang et al. 2018; Zhou et al. 2018). Another type of method(Simonyan and Zisserman 2014) use a two-way ConvNet to capture spatial appearance features and temporal motion patterns respectively. As a natural evolution, 3D CNNs (Tran et al. 2015; Carreira and Zisserman 2017; Hara, Kataoka, and Satoh 2018) are later used to capture spatio-temporal patterns at the same time. Feichtenhofer et al.(2019) uses two-pathway on the basis of 3D network and achieves good results. However, Li et al.(2018) and Girdhar et al.(2020) point out that the current commonly used video datasets are plagued with implicit biases over scene and object structureand they propose two datasets, which requires a complete understanding of spatio-temporal information for a model to give correct prediction. Choi et al.(2019) and Wang et al.(2018) propose to mitigate scene bias from the perspective of training strategy.

Self-supervised learning has received extensive attention in the field of image classification. One common way is to define a pretext that are related to downstream tasks(Noroozi and Favaro 2016a; Gidaris, Singh, and Komodakis 2018; Jenni, Jin, and Favaro 2020). Another mainstream type is based on metric learning, which aims to minimize the distance between similar samples while pushing away dissimilar samples in the feature space. Contrastive loss(Hadsell, Chopra, and LeCun 2006) proposes to decrease the distance between positive pairs while pushing the negative pairs to a certain margin. Triplet loss(Schroff, Kalenichenko, and Philbin 2015) makes further improvements by introducing triplets, which minimizes the distance between an anchor and a positive sample and maximizes the distance between the anchor and a negative sample. Recent works based on contrastive loss (Wu et al. 2018; He et al. 2020; Chen et al. 2020) have achieved excellent results on multiple visual tasks and narrowed the gap between the supervised learning and the unsupervised learning. The core idea in contrastive learning is to strengthen the invariance of the network to various data augmentations. In this article, we explore video representation learning under the framework of self-supervised learning.

Recently, many self-supervised video representation learning methods have been proposed. Among them, one prominent direction is to design a surrogate signal that can been used as supervision such as sequence order of frames(Misra, Zitnick, and Hebert 2016), space-time cubic puzzles (Kim, Cho, and Kweon 2019), video clip order (Xu et al. 2019; Luo et al. 2020) and video playback rating (Yao et al. 2020; Benaim et al. 2020). Besides, Gan et al.(2018) and Wang et al.(2019a) use the statistics of optical flow as supervision. Another mainstream category is based on contrastive learning, whose core is to construct positive samples and negative samples. TCN(Sermanet et al. 2018) treats the same actions under different cameras as positive samples and different time periods of the same video as negative samples. IIC(Tao, Wang, and Yamasaki 2020) regards multi-modal data as positive samples, and videos with shuffled frame order as negative samples. CVRL(Qian et al. 2020) proposes temporally consistent spatial augmentation with simple operations and treats the generated results as positive samples. In this work, we design two simple but effective strategies to construct positive and negative samples.

The entire framework is shown in Figure 2. Formally, given an unlabeled video dataset $X$ that contains $N$ videos, we sample $T$ frames from the video for each clip and input the clip to the network. Random cropping is performed on each input to generate three clips with different spatial regions but maintaining temporal consistency, denoted as $c_{1}$, $c_{2}$ and $c_{3}$. Afterwards, we apply basic augmentation $b$, spatial warping $s$ and motion disturbance $t$ on these three clips respectively to construct a triplet, i.e., anchor ${a}=b(c_{1})$, positive sample ${p}=s(c_{2})$ and negative sample ${n}=t(c_{3})$. Compared to $a$, $p$ destroys the structural information of scene but maintains temporal semantics. Meanwhile, $n$ disturbs the local motion pattern of the moving subject but retains the spatial information. This triplet is fed into a 3D backbone $f$ to extract spatio-temporal features which are then projected to a D-dimension feature space followed by L2 normalization, and we denote the decoded feature as $z_{a}$, $z_{p}$ and $z_{n}$. In this way, the triplet is projected into a normalized embedding space $(z_{a},z_{p},z_{n})\in\mathcal{R}^{D}$. We then perform spatio-temporal representation learning in the normalized embedding space using two metrics: intra-video triple loss and contrastive loss, which will be introduced in details in the later section.

The core idea of positive sample construction is to break local contexts while keeping motion semantics basically unchanged with data augmentation. Spatial data augmentations, e.g., rotation, color jittering, have been widely used in the image-related task. However, it is underexplored in the video domain. A naive way for video augmentation may be applying existing image spatial augmentations to each frame of the video. However, some of these operation may damage the motion semantic. For example, if different rotation angles are used for consecutive frames, the generated video will looks like suffering a severe camera shake, making the video difficult to recognize. In order to make the temporal abstraction of the entire video remains similar, all consecutive frames of a video should perform the same transformation, and a video data augmentation that meets this requirement is temporally consistent.

Thin-Plate-Spline (TPS) is widely used in the OCR field to rectify distorted text regions(Jaderberg et al. 2015; Shi et al. 2016). In contrast, we aim to damage the statistics of the scene but keep motion pattern unchanged with TPS. Specifically, we select $N$ uniformly distributed destination points on the target video as $\mathcal{D}=\{d_{i}\}_{i=1,...,N}\in\mathbf{R}^{2\times N}$. For each destination point $d_{i}$, we add a small offset $\Delta_{i}=[\Delta x_{i},\Delta y_{i}]^{T}$ to generate a corresponding source point $s_{i}=d_{i}+\Delta_{i}$ on the original video, making up a source point set $\mathcal{S}$. Both the horizontal and vertical offsets are randomly sampled from $[-C,C]$, where $C$ is typically one-tenth of the frame size. Then $s$ and $d$ are used to compute the parameters of TPS. At last, the grid $P$ is computed by the TPS transformation and the final warped video is generated by a bilinear sampler given $P$ and the original video. An illustration of spatial warping is shown in Figure 3. In this way, although spatial local statistics are modified, the global context is maintained. In each training epoch, due to the randomness of the grid, the generated warped videos are different and the pixels in the local area always show huge differences from the original videos. Therefore, the network needs to focus more on motion pattern and pay less attention to spatial changes to extract consistent representations for the original and warped videos. Since all frames in the video perform the same transformation operation, it is a temporally consistent augmentation.

In contrast to ${p}$, the major difference between ${a}$ and ${n}$ is the motion pattern. A straightforward idea may be using other videos directly as ${n}$ as in recent contrastive learning method (Sermanet et al. 2018). However, besides the temporal information, there still exists many artificial cues to distinguish two videos, which are easier to solve for the network (Kim, Cho, and Kweon 2019). Therefore, it is not guaranteed that the network will focus on the motion. To overcome this limitation, we propose to generate ${n}$ with large temporal abstraction differences but similar context from ${a}$ using Temporal Local Disturbance (TLD). TLD comprises two transformations and are described in details as below.

By applying the equation, we can accelerate or decelerate the video motion without changing background pixels too much. In particular, given a scale factor $\phi(t)$, we have

where $\phi(t)$ is randomly sampled from [0, M] over time $t$, and $M$ is a hyperparameter controlling the amplitude. We find that a too big $M$ may result in unnatural videos with wide black boundary, and setting $M$ to 5 gives best result.

Intuitively, the choose of $\tau$ determines the similarity between $x_{i}$ and $\hat{x}_{i}$. When $\tau$ approaches zero, the generated $\hat{x}$ looks very similar with $x$. To differentiate the ${n}$ and ${a}$ apart, the encoder network must focus on global rather local statistics.

We employ two objective functions to optimize the model. One is intra-video triplet learning, which generates negative sample by itself. The other one is the contrastive learning, which takes in other video as negative samples. While contrastive loss has been widely and successfully used in self-supervised methods, we verify that our methods does not rely on such loss design and generalize well with triplet loss.

Datasets. All the experiments are conducted on three video classification benchmarks, UCF101, HMDB51 and Kinetics (Kay et al. 2017). UCF101 consists of 13,320 manually labeled videos in 101 action categories and HMDB51 comprises 6,766 manually labeled clips in 51 categories, both of which are divided into three train/test splits. Kinetics is a large scale action recognition dataset that contains 246k/20k train/val video clips of 400 classes.

In this section, we evaluated DSM on video retrieval task. Following Clip Order and SpeedNet, the network is fixed as a feature extractor after pre-training with DSM on the split 1 of UCF101. Then the videos from both UCF101 and HMDB51 are divided into clips in units of 16 frames. All the clips in the training set constitute a Gallery, and each clip in the test set is used as a query to retrieve the most similar clip in the Gallery with cosine distance. If the category of the query appears in the K-nearest neighbors is retrieved, then it is considered as a hit. It should be noted that in order to keep the scale of representations generated by each 3D architecture consistent, we replaced the original global average pooling with an adaptive max pooling, yielding representations with a fixed scale of $1024\times 2\times 7\times 7$. We show the accuracy when $K=1,5,10,20,50$ and compare with other self-supervised methods on UCF101 and HMDB51 in Table 2 and Table 3 respectively. It can be seen that when using the same backbone C3D, DSM surpasses the mainstream method Clip Order on the UCF101, and surpasses Clip Order and VCP on the HMDB51, which proves that the representations extracted by DSM are more discriminative.

In this section, we explore the effectiveness of each component in the DSM. Results are shown in Table 4, from which we can conclude that all of these components lead to better results, and in the way of generating negative samples, both scaling optical-flow and temporal-shift are effective and the combination of the two can bring further gains.

Visualizing salient regions. In order to analyse which space-time regions our model focus on, we visualize the energy of the extracted representations with CAM(Zhou et al. 2016). For comparison, we pre-train two models using I3D as backbone on the UCF101 under two settings: i. fully supervised, ii. self-supervised using DSM. We select some videos with obvious motion from HMDB51 and use a sliding window to generate multiple clips for each video, and then visualize the corresponding activation maps of these clips in Figure 5. Specifically, assuming that each video has $L$ frames, a sliding window with a scale of 16 and a stride size of 4 slides on the temporal dimension, generating $(L-16)/4$ clips for each video. All clips of each video are input to the above two models and the extracted feature representations of the last 3D layer before global average pooling is of the shape of $1024\times T\times N\times N$, where $T$ and $N$ are the scale of the temporal and spatial dimension. Afterwards, we average over all channels to compress these features into the shape of $T\times N\times N$, then upsample and mask these heatmaps to the original videos. Visualization results under setting i are displayed in the first row, and those under setting ii are shown in the second row. It can be observed that the supervised approach is severely affected by the scene bias and falsely focus on the static background. On the contrary, DSM suffer less from scene bias and correctly focus on moving objects. Moreover, for setting ii, we average and normalize the feature of each clip into a scalar, which is recorded as response value, then we plot the curve of all clip response values over time. We find the curve has a low value when there is a inconspicuous movement, such as a clip that is about to end an action, and the alternate phase of a cyclic action, which is consistent with our original intention to enhance the temporal sensitivity of the model.