Idempotent Unsupervised Representation Learning for Skeleton-Based Action Recognition

Skeleton sequences encode the motion trajectories of human joints, representing rich information about human actions. Thus, skeleton data serves as a suitable modality for human action recognition [68, 25, 66, 42]. Skeleton can be obtained by applying pose estimation algorithms on RGB videos or depth maps [38].

Early studies focused on extracting hand-designed spatial and temporal domain features from skeleton sequences for human movement recognition [49, 50, 51, 10, 27]. In later work, efforts were made to model the positional information and higher-order temporal difference information of the human skeleton [44, 49]. Additionally, graphical models were built by tracking the trajectory of human joints to capture joint information in video sequences [52].

Recently, there has been a surge of interest in using graph structures for learning models [52]. Graph Neural Network (GNN) is one such model capturing intra-graph dependencies through information transfer between nodes. Various approaches have been proposed, such as spatio-domain inference networks, recurrent neural networks (RNNs), and graph convolutional networks (GCNs), to exploit graph structures for human action recognition [40, 36, 59]. These models automatically learn spatio-temporal patterns from skeleton data, facilitating strong action generalization. Moreover, attention mechanisms, multiscale aggregation schemes, and lightweight convolution operations have been integrated into GCN-based models to enhance their effectiveness and reduce computational costs [25, 66, 6, 41].

The self-supervised task aims to extract data features from a large amount of unlabelled data [8]. It can be widely used in semantic segmentation, image classification, action recognition and many other tasks [17, 31]. These tasks are mainly classified into methods based on reconstruction and on contrastive learning.

Reconstruction based approach after masking part of the original data, the network is used to reconstruct the masked part of the data. He et al. [14] proposed Mased Auto-Encoder (MAE) to encode the visible patches and decode the visible and masked patches. This approach has been extended to the video domain and has been used in several studies. These methods typically use a visual Transformer as the backbone network in order to perform the mask reconstruction task. Feichtenhofer et al. [9] extended the image-based masked auto-encoder to use spatio-temporal learning to randomly mask spatio-temporal segments of a video and learn am auto-encoder for reconstruction at the pixel-level reconstruction. Similarly, in MaskFeat, Wei et al. [55] used several video cubes and utilized the model to predict them using the remaining information.

Contrastive learning pushes pairs of positive sample together while pushing pairs of negative sample further apart. To generate negative samples, contrastive learning pairs anchor frames with frames from other videos. There are various ways of generating positive and negative samples, which is the main factor that distinguishes different contrastive methods.

Most of these methods generate positive and negative samples by different ways in order to minimize and maximise the distance between them respectively. In the image domain, positive samples are usually generated by enhancing the image in different ways [47, 56, 1, 62, 16]. These enhancements include rotation, cropping, random greyscale and colour change [2]. Scaling these methods in video can be difficult because each video comparison increases the memory required, especially if multiple enhancements are used for multiple positive samples. Another challenge is incorporating the temporal domain into the enhancement. Some methods simply apply the same enhancement in the image to each frame [48]. Some methods include additional frame alignments that may be based on the temporal domain [26]. Finally, some methods rely on motion and optical flow maps as positive samples [33].

Self-conditional generative modeling [14] is frequently employed as a pre-training task in self-supervised learning. It is generally structured as an auto-encoder. Formally, given the input skeleton data $\mathbf{x}$, the reconstruction loss is:

where $\mathbf{z}=f(\mathcal{T}(\mathbf{x}))$, $f(\cdot)$ is the encoder, and $\mathcal{T}(\cdot)$ is data transformation. $g(\cdot)$ is the generator. $\mathcal{D}(\cdot,\cdot)$ is the distance. $p_{\mathbf{x}}$ is the data distribution and $p_{\mathbf{z}|\mathbf{x}}$ is the feature distribution given $\mathbf{x}$. $H(\cdot|\cdot)$ is the conditional entropy. In the context of MAE, this data transformation represents masked data during training. Conversely, in denoising auto-encoders, this transformation signifies adding noise to the input data.

In the context of mutual information, this loss function is equivalent to optimizing the mutual information $I(\mathbf{z};\mathbf{x})$ between the extracted features $\mathbf{z}$ and the input data $\mathbf{x}$. Based on the relationship between mutual information and entropy, we get

Since the entropy of $\mathbf{x}$ remains constant, decreasing the reconstruction loss is akin to increasing the mutual information. Conversely, as the features $\mathbf{z}$ are deterministically derived from the data $\mathbf{x}$ by an encoder $f(\cdot)$, the entropy $H(\mathbf{z}|\mathbf{x})$ tends towards zero. Hence, maximizing the mutual information $I(\mathbf{z};\mathbf{x})$ is equivalent to maximizing the entropy of the feature space $H(\mathbf{z})$.

Estimating the true distributions $p(\mathbf{z})$ of the representation space is exceedingly challenging. Following works [63, 24], we leverage lossy data coding, a computationally feasible alternative, as a surrogate for the entropy of continuous random variables $H(\mathbf{z})$. This approach involves determining the minimal number of bits required to encode a set of samples $\mathbf{Z}=[\mathbf{z}^{1},\dots,\mathbf{z}^{m}]\in\mathbb{R}^{d\times m}$ subject to a distortion $\varepsilon$, as defined by the coding length function below [47, 72]:

where $\varepsilon$ is the upper bound of the expected decoding error between $\mathbf{z}\in\mathbf{Z}$ and the decoded $\hat{\mathbf{z}}$. $\det(\cdot)$ is the determinant of a matrix. $d$ is the dimension of the feature space. Utilizing the identity $\det(\exp(\mathbf{A}))=\exp(\text{Tr}(\mathbf{A}))$, we derive $L=\text{Tr}\left(\mu\log\left(\mathbf{I}+\lambda\mathbf{Z}^{T}\mathbf{Z}\right)\right)$, where Tr denotes the trace of the matrix and $\mu=\frac{m+d}{2}$, $\lambda=\frac{d}{m\varepsilon^{2}}$. Finally, we apply a Taylor series expansion to the logarithm of the matrix to obtain:

because the features $\mathbf{z}$ are projected into spherical space $\mathbb{S}^{d-1}$, $\text{Tr}\left(\mu\lambda\mathbf{Z}^{T}\mathbf{Z}\right)=m\mu\lambda$. Hence, the first term does not contribute to the reconstruction learning. In essence, self-conditional generation primarily diminishes the inter-data similarity within the feature space:

where $\mathbf{R}=\sum_{n=3}^{\infty}\frac{(-1)^{n}\mu\lambda^{n}}{n}\text{Tr}\left(\left(\mathbf{Z}^{T}\mathbf{Z}\right)^{n}\right)$.

The idempotence of a self-conditional generative model refers to its stability in re-encoding [57]. More precisely, if we denote the original data as $\mathbf{x}$, the encoder as $f(\cdot)$, the encoded feature as $\mathbf{z}=f(\mathbf{x})$, the decoder as $g(\cdot)$, and the reconstruction as $\hat{\mathbf{x}}=g(\mathbf{z})$, then the self-conditional generative model is considered idempotent:

Idempotence is frequently employed in the generative domain to augment the perceptual loss of generated images. The idempotent loss is formulated as:

where $\mathbf{z}^{T}\mathbf{z}=1$ because we normalize the feature space. Therefore, the idempotent generative model maximizes the entropy of the feature space while simultaneously minimizing the feature distance between the data and the generated data. The total loss of the idempotent generative model is expressed as:

where $\mathbf{A}\in\mathbb{R}^{m\times m}$ is the adjacency matrix defined by the data generation and $\mathbf{C}$ is a constant. $\mathbf{F}=\mathbf{Z}\text{diag}(\sqrt{p(\mathbf{x})})$ The weights $\mathbf{A}_{\mathbf{x},\hat{\mathbf{x}}}=\frac{p(\mathbf{x},\hat{\mathbf{x}})}{\sqrt{p(\mathbf{x})p(\hat{\mathbf{x}})}}$. $\mathbf{L}=\mathbf{I}-\mathbf{A}$ is the Laplacian matrix. This demonstrates its equivalence to spectral contrastive learning. And the advantage of our approach over spectral contrastive learning is that we additionally optimise the residual term $\mathbf{R}$ to capture higher order information.

Further, we exploit data idempotence and feature idempotence to enhance representation learning and action generation. The unified generative-perceptual model contains both an encoder $f(\cdot)$ and generator $g(\cdot)$. And our idempotency constraints pay attention to both the data and the feature distributions, which improves both generation and feature learning.

As mentioned in Eq. 2, the generative network without idempotent constraints has a conditional entropy $H(\mathbf{z}|\mathbf{x})$ of 0 because the encoding process is deterministic. Idempotent generative networks, on the other hand, treat features $\mathbf{z}$ as a random variable sampled from the distribution of features across all of the generated data $\hat{\mathbf{x}}$ for the same data $\mathbf{x}$, thus transforming into a non-deterministic process:

where $G(\cdot)$ is the generation process. Therefore, idempotent constraints are essentially about diminishing conditional entropy $H(\mathbf{z}|\mathbf{x})$, which in turn maximizes the mutual information between features and data.

In contrast, methods like MAE implicitly prioritize maximizing feature similarity across masked samples of the same data:

where $M(\cdot)$ is the random masking process. Consequently, features from two distinct data that undergo similar transformed or generated data are clustered into the same class. However, the data obtained through data transformation may not be the real data and thus far from the real data distribution.

Through the analysis of previous work [13, 67, 54, 12, 7] on spectral contrastive learning, the error rate $P_{e}=P[\phi(\mathbf{x})\not=y_{\mathbf{x}}]$ of the downstream linear evaluation $\phi(\cdot)$ can be bounded by the generated adjacency matrix $\mathbf{A}$ and clustering error probabilities $\alpha=P[y_{\mathbf{x}}\not=y_{\hat{\mathbf{x}}}]$:

This theorem illustrates the constraints on accuracy imposed by the purity $1-\alpha$. A large purity and a small number of clusters result in a low error rate. When the diversity in the generated data is insufficient, the sum of small singular values of the adjacency matrix become large, resulting in less tightly clustered groups. Conversely, excessive diversity in the generated data may compromise the preservation of motion information, thereby increasing the error rate in clustering and undermining overall clustering effectiveness.

Therefore, to make the feature space of the idempotent generative model more capable of clustering, it is necessary to increase the diversity of the generated data for a stronger feature consistency constraint. However, a paradox is demonstrated here. Ordinary generative processes result in limited diversity under self-conditional generation due to constraints on the distance between the generated data and the original data. So in order to simultaneously obtain diverse and motion semantics preserving generated data, we propose an idempotent self-conditional generation model based on the diffusion generation model. The diversity of the generated data is provided by the noise sampling process of the diffusion model.

Our model consists of three parts, an encoder $f(\cdot)$, a generator $g(\cdot)$ and an adapter $h(\cdot)$. The encoder $f(\cdot)$ extracts features $\mathbf{z}$ as conditions for the generator $g(\cdot)$ and also as inputs to the downstream task classifier $\phi(\cdot)$. And the generator $g(\cdot)$ reconstructs the skeleton data based on the features. The adapter $h(\cdot)$, in turn, is responsible for projecting and fusing the features extracted by the encoder $f(\cdot)$ into the generator’s feature space to be used as conditions.

We then fuse the features into the generator by replacing LayerNorm (LN) with Adaptive LayerNorm (AdaLN) with the following equation:

where $\mathbf{h}$ represents the hidden representation of the generator, $(\mathbf{t}_{s},\mathbf{t}_{b})$ and $(\hat{\mathbf{z}}_{s},\hat{\mathbf{z}}_{b})$ are obtained from linear projection of timestep embedding $t$ and high-frequency condition $\hat{\mathbf{z}}$, respectively. Through AdaLN layers, the condition $\hat{\mathbf{z}}$ guides the denoising process by scaling and shifting the normalized hidden representation.

For enhancing network training, all skeleton sequences undergo temporal downsampling to 120 frames. The encoder $f(\cdot)$ and generator $g(\cdot)$ are built using the Transformer architecture [59], employing hidden channels configured to a dimension of 256. To assess performance, we employ a fully connected layer $\phi(\cdot)$.

To refine our network, we employ the Adam optimizer [30]. Training is executed on a single NVIDIA GeForce RTX 4090, employing a batch size of 128, and the network undergoes training for 400 epochs.

For a comprehensive assessment, we conduct comparative analysis of our approach with other methodologies across diverse scenarios.

Here’s the modified text for the ablation experiments: