Contrastive Learning from Spatio-Temporal Mixed Skeleton Sequences for Self-Supervised Skeleton-Based Action Recognition

Contrastive learning, whose goal is increasing the similarity between positive pairs and decreasing the similarity between negative pairs, has shown promising performance in self-supervised representation learning [23, 24]. In the past few years, there emerged numerous self-supervised representation learning works based on contrastive learning, such as instance discrimination [36], SwAv [37], MoCo [23], SimCLR [24], BYOL [38], contrastive cluster [39], DINO [40], and SimSiam [41]. These methods have achieved advanced results and are easy to transfer to other areas, such as skeleton-based action recognition. In this paper, we follow MoCov2 [42] framework to implement our method.

Most of the top-performing contrastive methods leverage data augmentations, which is crucial in learning useful representations because they modulate the hardness of the self-supervised task via the contrastive pairs. Recently, Mixing [26] and CutMix [27] are widely discussed in self-supervised contrastive learning. These operations are mainly performed between embedding features or between samples. MoCHi [25] proposes a hard negative mixing strategy, which generates hard negative samples by mixing the embedding features to improve the generalization of the learned visual representations. Vi-Mix [43] proposes CMMC to mix the data across different modalities of a video in their intermediate representations. MixCo [44], Un-Mix [45] and i-Mix [46] perform mixing operation between samples and generate corresponding smoothing labels to let the model be aware of the soft degree of similarity between contrastive pairs. The above methods mainly construct contrastive pairs with the mixing features. Different from these methods, we utilize the features of the trimmed and truncated views separated by STMP to provide hard samples for contrastive learning. Our method not only makes better use of the augmented features, but also improves the ability of the model to learn both local and global representations. RegionCL [47] also propose to utilize the separated features, while our method leverages the topological information of the skeleton data to maintain the consistency of action information, and performs mixing operation on both spatial and temporal domain. Moreover, we propose to apply multiple SkeleMix operation to provide a more adequate hard contrastive pairs.

Self-supervised skeleton-based action recognition has emerged as one of the promising direction for action recognition. LongT GAN [17] and P&C [48] rely on regeneration of the skeleton sequences to help the model to learn spatio-temporal representations. Colorization [35] leverages the colorized skeleton point cloud and designs an auto-encoder framework that can effectively learn spatio-temporal features from the artificial color labels of skeleton joints. With the development of contrastive learning, the past few years have witnessed a surge of successful self-supervised contrastive skeleton-based action recognition. AS-CAL [20] exploits eight different skeleton sequence augmentations and their combinations to generate query and key views for contrastive learning. ISC [34] proposes inter-skeleton contrastive learning to enhance the learned features via different input skeleton representations. MS²L [18] and CP-STN [33] combine contrastive learning with multi-pretext tasks such as masked sequences prediction, enabling the model to fully extract discriminative representations with spatio-temporal information. Works like CrosSCLR [22] and AimCLR [21] also try to improve contrastive learning by introducing extra hard contrastive pairs. In CrosSCLR, a cross-view knowledge mining strategy is developed to exam the similarity of samples, and select the most similar pairs as positive ones to boost the positive set in complementary views. In AimCLR, an extreme augmentation strategy is proposed to introduce movement patterns, which forces the model to learn more general representations by providing harder contrastive pairs. In this paper, we propose a unique skeleton sequence augmentation strategy SkeleMix to provide hard contrastive samples, which guides the model to learn better local and global spatio-temporal representations.

As illustrated in the gray box of Figure 3, SkeletonCLR [22] follows the MoCov2 framework [42] to learn skeleton-based action representations. Given a skeleton sequence $S$, two different augmentations $A$ and $A^{\prime}$ are applied to $S$ to obtain query sample $\hat{x}$ and key sample $x$. We denote $x,\hat{x}\in\mathbb{R}^{C\times T\times V}$, where $C$, $T$, and $V$ are the number of channels, frames, and nodes, respectively. A query encoder and a momentum updated key encoder followed by global average pooling (GAP) are used to get query embedding $q$ and key embedding $k$. Then the key embedding $k$ is stored in a first-in-first-out memory queue $\textbf{Q}=\{m_{i}\}^{K}_{i=1}$, where $K$ denotes the queue size. Following the criteria for constructing contrastive pairs in MoCov2, $q$ and $k$ form positive pairs while $q$ and the embeddings in Q form negative pairs. The InfoNCE loss [49] formulated as Eq. (1) is used to train the network, where $\cdot$ is dot product to compute the similarity between two $L2$ normalized embeddings, and $\tau$ is the temperature hyperparameter (set to 0.2 by default). $Z(v)=\sum_{i=1}^{K}\exp(v\cdot m_{i}/\tau)$ denotes the similarity between embeddings of view $v$ and memory queue Q.

The parameters of query encoder $\theta_{q}$ are updated via gradient back propagation while the parameters of key encoder $\theta_{k}$ are updated as a moving-average of the query encoder, which can be formulated as:

where $m\in[0,1)$ is a momentum coefficient and typically close to 1 to maintain consistency of the embeddings in the memory queue.

In this section, we will introduce our SkeleMix augmentation, which is a spatio-temporal mixing augmentation strategy. As illustrated in Figure 2, our SkeleMix follows CutMix to peroform spatio-temporal mixing on skeleton sequences. Specifically, in the cropping step, skeleton joints are first partitioned into five part subsets $\mathcal{P}=\{$left-hand, right-hand, left-leg, right-leg, trunk$\}$ based on the topological information of skeleton data. Second, two discrete uniform distributions $\mathcal{B}_{s}{\sim}U(\mathcal{B}_{sl},\mathcal{B}_{su})$ and $\mathcal{B}_{t}{\sim}U(\mathcal{B}_{tl},\mathcal{B}_{tu})$ are used to determine the number of body parts to crop out and the duration of the cropped skeleton fragments. We sample once in each training iteration and obtain $\mathcal{N}_{s}$ and $\mathcal{N}_{t}$. Then, we randomly choose $\mathcal{N}_{s}$ body parts from $\mathcal{P}$ to get the cropped skeleton joints $\mathcal{S}$, and randomly sample the start frame $t_{s}$ from a valid range that guarantees the completeness of the cropped skeleton fragments. Combining $\mathcal{N}_{t}$ and $t_{s}$, we can obtain the corresponding temporal cropping region $\mathcal{T}$. Finally, we perform spatio-temporal cropping operation on $\hat{x}$ to get the cropped skeleton fragments. It is notable that the cropped regions ($\mathcal{S}$ and $\mathcal{T}$) within a mini-batch of the training skeleton sequences are typically the same to maintain consistency of action information. In combining step, we randomly combine the cropped skeleton fragments (named as the trimmed view) and the remaining skeleton sequences (named as the truncated view) to generate the mixed skeleton sequences. Thus, the mixed skeleton sequences consist of the trimmed view and truncated view. Furthermore, to provide a sufficient hard samples for contrastive learning, we propose to perform multiple SkeleMix augmentation. For convenience, we denote the mixed skeleton sequences generated by the $r_{th}$ SkeleMix augmentation as $\hat{x}^{r}_{m},r\in(1,...,R)$, where $R$ denotes the total number of SkeleMix augmentation performed.

In this section, we introduce our SkeleMixCLR in detail. A weight-sharing encoder with the query view in SkeletonCLR [22] is used to extract features $\hat{f}^{r}_{m}$ from the mixed skeleton sequences $\hat{x}^{r}_{m}$. We denote $\hat{f}^{r}_{m}\in\mathbb{R}^{C^{\prime}\times T^{\prime}\times V}$, where $C^{\prime}$ is the feature dimension and $T^{\prime}=T/\mathcal{R}$ where $\mathcal{R}$ is the temporal downsampling ratio of the model. To separate the embedding of the trimmed view $p^{r}$ and the truncated view $g^{r}$, we utilize a spatio-temporal mask pooling operation (STMP), which can be formulated as:

where $M^{r}\in\mathbb{R}^{T^{\prime}\times V}$ is a [0, 1] mask indicates the corresponding cropped skeleton fragments, while $\bar{M^{r}}=\textbf{I}-M^{r}$ indicates the corresponding remaining skeleton sequences. I is unit matrix with shape $\mathbb{R}^{T^{\prime}\times{V}}$. Then, we extend the contrastive pairs with the trimmed view and the truncated view. Following the criteria of constructing contrastive pairs in MoCov2 [42], for the trimmed embedding $p^{r}$, it is positive with corresponding key embeddings $k_{p^{r}}^{+}$ while negative with $g^{r}$ and the embeddings stored in the memory bank Q. Therefore, the contrastive loss for the trimmed view $p^{r}$ can be written as:

Similarly, the truncated embedding $g^{r}$ is positive with corresponding key embeddings $k_{g^{r}}^{+}$, while negative with $p^{r}$ and the embeddings store in the memory bank Q. Therefore, the contrastive loss for the truncated view $g^{r}$ can be written as:

Since the trimmed view and the truncated view share context information during forward inference, and the action information they contain is incomplete, they provide hard samples for contrastive learning. The expanded contrastive samples provide extensive hard positive pairs such as $p^{r}$ and $k_{p^{r}}^{+}$ (as well as $g^{r}$ and $k_{g}^{+}$) and hard negative pairs such as $p^{r}$ and $g^{r}$, thus helping the model to learn better global and local spatio-temporal representations and improving the performance in downstream tasks.

Considering that the trimmed view and the truncated view are symmetrical, we use the average of $\mathcal{L}_{p^{r}}$ and $\mathcal{L}_{g^{r}}$ as the overall loss of the $r_{th}$ mix branch, which can formulated as:

Finally, the loss used to optimize the encoder can be formulated as:

where $\lambda$ is a hyperparameter to balance the easy contrastive pairs and the hard contrastive pairs.

In order to evaluate the effectiveness of the proposed method, we conduct experiments on three widely used datasets for skeleton-based action recognition.

To perform a fair comparison, we use the same data pre-processing with SkeletonCLR [22] and AimCLR [21] except for that we resize the length of skeleton sequences to 64 frames, rather than 50 frames. This allows our SkeleMix  to be implemented more efficiently, since the temporal downsample ratio $\mathcal{R}$ is 4 of ST-GCN backbone. Thus, the temporal size of the final output feature is 16. The batch size for both pretraining and downstream tasks is set to 128 for by default, except in specific cases.

(1) $Shear$: The shear augmentation is a linear transformation on the spatial dimension. The shape of 3D coordinates of body joints is slanted with a random angle. The transformation matrix is defined as:

where $a_{12}$, $a_{13}$, $a_{21}$, $a_{23}$, $a_{31}$, $a_{32}$ are shear factors that randomly sampled from a uniform distribution $a_{ij}\sim U(-\beta,\beta)$, where $\beta$ is the shear amplitude. Follow SkeletonCLR and AimCLR, we set $\beta=0.5$. The skeleton sequence is multiplied by the transformation matrix $T_{s}$ on the channel dimension.

(2) $Temporal~{}Crop$: For the temporal skeleton sequence, specifically, we symmetrically pad some frames to the sequence and then randomly crop it to the original length, which increases the diversity while maintaining the distinction of original samples. The padding length is defined as $T/\gamma$, where $\gamma$ is the padding ratio and here we set $\gamma=6$.

We compare our method with other methods under a variety of evaluation protocols, including KNN evaluation protocol, linear evaluation protocol, finetune protocol, and semi-supervised evaluation protocol.

In this section, we conduct ablation studies on different datasets with linear evaluation protocol to verify the effectiveness of different components of our method.

Our method can provide abundant hard contrastive pairs in contrastive pretext task. To further verify the effectiveness of our method, we test the influence of each component. A blank control method (denoted as $Cont$) is constructed by naively extending a query view and constructing the corresponding contrastive pairs. The results are shown in Table VII, where $\mathcal{L}_{p}$ and $\mathcal{L}_{g}$ denote whether to use $p$ and $g$ to construct hard contrastive pairs, respectively. $d_{pg}$ denotes whether to use $p$ and $g$ to construct hard negative pairs. Since SimSiam [41] further finds that stop-gradient is critical to prevent from collapsing, we also use this strategy denoted as $detach$ when constructing hard negative pairs between $p$ and $g$. As shown in Table VII, naively extending contrastive pairs based on normal augmentations brings limited returns in terms of performance (0.4% on Xsub benchmark and 0.5% on Xview benchmark), which indicates that more does not mean effective. Compared with the blank control method, we found that the hard positive pairs contribute a lot to the performance and using hard negative pairs also greatly improves the performance on NTU-60 Xview benchmark. The stop-gradient strategy can further improve the performance.

To further analyze our approach, we compare the average cosine similarity of easy contrastive pairs provided by baseline and hard contrastive pairs introduced by our method. As shown in Figure 4, as the training progresses, the cosine similarity between easy positive contrastive pairs (query and key) is very close to 1, and the cosine similarity between easy positive contrastive pairs (query and Queue) is very close to 0, which contribute less and less to the loss [25]. Hard positive contrastive pairs (truncated and key, trimmed and key) and hard negative contrastive pairs (truncated and trimmed) are harder than easy contrastive pairs, which contributes more to optimize the model.

To summarize the above, our method uses SkeleMix augmentation strategy to generate the mixed skeleton sequences and separate the trimmed view and the truncated view at the feature level to provide hard samples for contrastive learning. SkeleMixCLR further utilizes the hard samples to expand the contrastive pairs with hard positive pairs and hard negative pairs, which significantly improves the capacity of the model in learning better complete and discriminative spatio-temporal representations for skeleton-based action recognition.

Based on the above experiments, we conclude that our method achieves state-of-the-art performance for self-supervised skeleton-based action recognition.