Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition

Spatial-Temporal Micro-prototypes Construction. Decoupling spatial structures and temporal variations is vital in distinguishing similar actions. As described in [38], temporal variations help differentiate ”put things into a bag” and ”take things out of a bag”, while spatial structures are significant for distinguishing them from ”reach into pockets”. Thus, we separate spatial and temporal representations for alignment independently. First, we extract the skeleton representations $\mathcal{F}=\Phi(x)\in\mathbb{R}^{\widehat{T}\times\widehat{V}\times C}$ by skeleton encoder $\Phi(\cdot)$, where $\widehat{T}\times\widehat{V}$ represents the spatial-temporal dimension after transformations, $C$ is the feature dimension. Then, we pool the temporal and spatial dimensions respectively to obtain the spatial feature $\mathcal{F}_{s}\in\mathbb{R}^{\widehat{V}\times C}$ and the temporal feature $\mathcal{F}_{t}\in\mathbb{R}^{\widehat{T}\times C}$. To effectively align the skeleton and semantic spaces, we design the learnable spatial micro-prototype $\mathcal{P}_{s}=\{p_{k}^{s}\in\mathbb{R}^{C\times 1}\}_{k=1}^{N_{s}}$ to capture the patterns of skeleton topology structure and construct the learnable temporal micro-prototype $\mathcal{P}_{t}=\{p_{k}^{t}\in\mathbb{R}^{C\times 1}\}_{k=1}^{N_{t}}$ to record the temporal motion variations of the skeleton, where $p_{k}^{s}$ and $p_{k}^{t}$ are randomly initialized attributes of skeleton representation (e.g., amplitude and velocity [6]), $N_{s}$ and $N_{t}$ are the number of primitives. Guided by semantics $\mathcal{Z}$, $\mathcal{P}_{s}$ and $\mathcal{P}_{t}$ progressively learn useful skeleton representations, similar to the growth process of neurons.

Spatial Compression Mechanism. For the given spatial micro-prototype $\mathcal{P}_{s}$, we aim to enable it to progressively and controllably absorb structure-related spatial patterns from skeletons across three phases, enhancing the cross-modal correspondence at the spatial level from coarse-grained to fine-grained. Specifically, in the first phase, we treat the initialized $\mathcal{P}_{s}^{0}$ as the query matrix to calculate similarity scores with spatial skeleton representations $\mathcal{F}^{s}$:

$$\mathcal{H}^{s_{0}}=softmax(\mathcal{F}_{s}\mathcal{P}_{s}^{0}),$$

(1)

where $h_{j,k}^{s_{0}}\in\mathcal{H}^{s_{0}}$ denotes the similarity between the $j$-th joint feature and the $k$-th spatial micro-prototype attribute at the coarse-grained level. Normally, a straightforward approach is to aggregate the joint features to obtain the updated micro-prototype for alignment with semantics:

$$\mathcal{P}_{s}^{1}=\varphi_{s}(\mathcal{F}_{s}^{T}\mathcal{H}^{s_{0}}),$$

(2)

where $\varphi_{s}(\cdot)$ is the refinement function composed of MLPs. However, not all the micro-prototype attributes are relevant for each joint across different samples; aggregating all attributes may inadvertently overemphasize those unrelated to the action itself (e.g., camera views). To mitigate this issue, we selectively drop joint-unrelated attributes, thus avoiding shortcut feature learning. We sort the matrix $\mathcal{H}^{s_{0}}$ row-wise and partition it into two subsets with the hyper-parameter $\alpha\in[0,1]$. Then, we maintain the top-$\alpha$ scores and replace others with zeros. Using the modified matrix $\mathcal{\hat{H}}^{s_{0}}$ in Eq 2, we can get the updated micro-prototype $\mathcal{P}_{s}^{1}$. Considering coarse-grained exploration alone is insufficient, we repeat the above operations at the mid-grained and fine-grained levels further. Meanwhile, $\alpha$ is incrementally increased in each phase to allow for more precise attribute selection. Through these controllable compression processes, the spatial micro-prototype is growing to $\mathcal{P}_{s}^{3}$, capturing more general and concise spatial pattern representations.

Temporal Memory Mechanism. Compared to the spatial micro-prototype $\mathcal{P}_{s}$ growing through the refinement of hierarchical structure, the temporal micro-prototype $\mathcal{P}_{t}$ develops alongside the evolution of resolution. Unlike the complete spatial feature $\mathcal{F}_{s}$, we split the temporal feature into three phases to obtain segment features $\mathcal{F}_{t}=\{\mathcal{F}_{t}^{e}\in\mathbb{R}^{\frac{\hat{T}}{3}\times C}\}_{e=1}^{N_{e}}$ for progressive alignment. Initially, we calculate the similarity scores between the initialized $P_{t}^{0}$ and 1-st temporal feature $F_{t}^{1}$, and then aggregate all frame features to update the temporal micro-prototype as follows:

	$\displaystyle\mathcal{H}^{t_{0}}$	$\displaystyle=softmax(\mathcal{F}_{t}^{0}\mathcal{P}_{t}^{0}),$		(3)
	$\displaystyle\mathcal{\hat{P}}_{t}^{0}$	$\displaystyle=(\mathcal{F}_{t}^{0})^{T}\mathcal{H}^{t_{0}},$		(4)

We do not adopt the compression mechanism here because the sequential temporal features in each phase can constrain the learning process not stuck into the short features. However, the updated temporal micro-prototype in the $e$-th phase faces the challenge of knowledge oblivion that is learned in the $(e-1)$-th phase, as shown in the first row of Fig. 5 (a) - 5 (c). To address this, in each phase, we introduce the memory mechanism to remember new information while selectively recalling the knowledge from the previous phase:

$$\mathcal{P}_{t}^{1}=\underbrace{\sigma(\rho_{r}(\mathcal{\hat{P}}_{t}^{0}))\cdot\mathcal{P}_{t}^{0}}_{recall}+\underbrace{\sigma(\rho_{m}(\mathcal{\hat{P}}_{t}^{0}))\cdot\varphi_{t}(\mathcal{\hat{P}}_{t}^{0})}_{remember},$$

(5)

where $\sigma(\cdot)$ is the sigmoid function to quantize the remember and recall probability, $\rho_{r}(\cdot)$ and $\rho_{m}(\cdot)$ are MLPs for projection, $\varphi_{t}(\cdot)$ is the refine functions. Through the evolution from the first to the third phase, the micro-prototype $\mathcal{P}_{t}^{0}$ progressively captures the regularity-dependent temporal clues, controllable forming to the $\mathcal{P}_{t}^{3}$ at the macro level.

NTU RGB+D 60 [32]. It contains 56880 skeleton sequences with 60 action categories performed by 40 subjects captured from 3 distinct views. Generally, two benchmarks are commonly utilized for evaluation: cross-subject (Xsub) and cross-view (Xview). The Xsub splits all sequences based on subject indices, with 20 subjects allocated for training and the remaining 20 subjects for testing. The Xview divides data based on view variations, using views 2 and 3 for training while reserving view 1 for testing.

NTU RGB+D 120 [22]. As an extended version of the NTU RGB+D 60 dataset, it encompasses 114,480 skeleton sequences with 120 action categories. Similarly, it also provides the two official benchmarks: cross-subject (Xsub) and cross-setup (Xset). The Xsub task requires 53 subjects for training, with the remaining subjects designated for testing. Meanwhile, the Xset task utilizes data captured from even camera IDs for training and others for testing.

PKU-MMD [9]. Almost 20000 skeleton sequences with 51 categories are collected in this dataset, which can be split into two phases with increasing challenges. It offers two benchmarks: cross-subject (Xsub) and cross-view (Xview). For the Xsub task, 57 subjects are designated for training, while 9 subjects are reserved for testing. For Xview, the middle and right views data is included in the training set, with the left view used for testing. In this work, we utilize the first phase for experiments following the [39, 6].

Protocols. We follow the previous studies [6] to conduct the experiments using predefined seen/unseen protocols and tasks in the ZSL and GZSL settings. For NTU 60, we adopt the 55/5 and 48/12 protocols. For NTU 120, they are extended to 110/10 and 96/24 protocols. For PKU-MMD I, we take the 46/5 and 39/12 split strategies.

Metrics. We evaluate the performance of our method using Top-1 classification accuracy $\mathcal{A}cc=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}[y_{i}\in\hat{y}^{*}]$. In the ZSL setting, we calculate accuracy ($\mathcal{A}cc$) on the $\mathcal{D}_{te}^{u}$. For the GZSL setting, we first compute the seen accuracy ($\mathcal{S}$) on the $\mathcal{D}_{te}^{s}$ and unseen accuracy ($\mathcal{U}$) on the $\mathcal{D}_{te}^{u}$, respectively. Then, the harmonic mean accuracy ($\mathcal{H}$) can be calculated by $\mathcal{H}=(2\times\mathcal{S}\times\mathcal{U})/(\mathcal{S}+\mathcal{U})$.

Zero-Shot Learning. We evaluate the performance of our method, Neuron, against state-of-the-art approaches in the ZSL setting, as shown in Table 2 - 4. Across three distinct datasets representing varied scenarios, our proposed method achieves superior performance. Furthermore, the performance of our method is robust across all tasks, demonstrating its ability to learn core spatial-temporal patterns from skeletons that enhance cross-modal correspondences, thereby facilitating better cross-modal transferability.

Generalized Zero-Shot Learning. As shown in Table 2 - 4, our method outperforms previous studies in both seen and unseen categories. Additionally, our method achieves an excellent balance between the seen and unseen categories, i.e., mitigating the tendency of the model towards the seen categories (domain bias), improving a considerable margin over others on the value of harmonic mean metrics. Thus, our synergistic framework actually helps to avoid shortcut representation learning, emphasizing spatial structure-related and temporal regularity-dependent feature learning.

Influence of Seen-Unseen Category Settings. We evaluate the robustness of our method under the different seen-unseen categories settings. To this end, we re-partition all skeleton categories into three different seen-unseen combinations that are non-overlap, keeping the same as the previous studies [39, 6], and then compute the average results of three settings to minimize variance. We report the ZSL and GZSL results in Table 5. It can be seen that our method consistently achieves state-of-the-art results in different settings, showing excellent stability and robustness.

Influence of Components. We evaluate the efficiency of different components of our method in Table 6. It is noted that the performance of Neuron drops significantly without the temporal memory mechanism, around 5.2%/1.5%, showing this technique can mitigate the situation of knowledge oblivion effectively. Additionally, the dual guidance on spatial granularity updating and temporal phase evolving is crucial in progressively controlling the process of cross-modal alignment. Meanwhile, the spatial compression mechanism can improve generalization, increasing the GZSL from 67.1% to 69.9%.

Influence of $\mathcal{N}_{s}$ and $\mathcal{N}_{t}$ in Micro-prototypes. As illustrated in Fig. 3 (a) - 3 (b), we observe that the accuracy initially improves and then decreases with the attribute number increases. The best performance of our method can be obtained when the $\mathcal{N}_{s}$ is around 100 and the $\mathcal{N}_{t}$ is approximately 50. With further increases in attribute values, accuracy stabilizes and remains consistently high, highlighting the effectiveness of our progressive learning approach.

Visualization of Spatial Compression. Here, we plot the t-SNE visualization of unseen skeleton representation distribution at each phase during the spatial compression process, as shown in Fig. 4 (a) - 4 (c). Moving from the coarse-grained to the mid-grained phase, we can see that the compactness of features in ”writing” becomes better. Furthermore, the inter-class separability between the ”reading” and ”put on a hat/cup” actions can also be improved in transitioning from the mid-grained to fine-grained steps. To quantify intra-class compactness, we calculate the compactness of features based on mainstream clustering evaluation metrics, including the silhouette score [29], Adjusted Rand Index (ARI) [16], Fowlkes-Mallows Index (FMI) [12], and Calinski-Harabasz Index (CH) [3], as shown in Fig. 4 (d). The intra-class compactness continually improves, underscoring Neuron’s effective core feature learning.

Visualization of Temporal Memory. As shown in Fig. 5 (a) - 5 (c), we draw the updating process of temporal micro-prototypes over multiple steps, comparing results with and without the temporal memory mechanism. In the first row, the updated temporal micro-prototype without the memory mechanism guidance will be chaotic as frames increase, leading to knowledge oblivion. In contrast, the second row exhibits a more effective learning process, where past knowledge is retained while new information is integrated. We also compute the intra-class compactness with the memory mechanism based on the above metrics [29, 16, 12, 3], all of them achieve better with the learning process ongoing.

Fine-Grained Action Recognition. To compare the performance of our method with the fine-grained method STAR [6] more intuitive, we calculate the absolute difference value of recognition accuracy between Neuron and STAR in each unseen category. As shown in Fig. 6, we find the accuracy of 9 action categories recognized by our method over the STAR a lot. Specifically, the performance improved for ”walking apart” and ”walking towards” actions as we paid isolated attention to temporal cues during the training process. Meanwhile, some spatial-oriented actions, e.g., ”clapping” and ”falling down” actions, are improving significantly. However, some failure cases, such as the ”nausea/vomiting” action, are incorrectly classified into the ”sneeze/cough” action with highly similar skeletons. Meanwhile, the homogeneous situation still exists between their contextual side information. So, further exploration is still needed for those extremely similar and abstract actions.