AFE-CNN: 3D Skeleton-based Action Recognition with Action Feature Enhancement

To recognize an action, it generally requires spatio-temporal information in a 3D skeleton sequence. Therefore, RNN and LSTM network can demonstrate their advantages on spatio-temporal information processing. Liao et al. [16] encoded the spatio-temporal information from streamed 3D skeleton data, wherein a novel hybrid RNN architecture decode them to recognize actions. To reduce the effect on irrelevant joints, Liu et al. [24] designed a global context-aware attention LSTM to selectively learn the spatio-temporal information from most relevant joints. In [25], Si et al. novelty combined a spatial reasoning network and a temporal stack learning network to learn the spatial structural information and temporal information of skeleton sequences. With the similar idea of combining networks, Zhang et al. [26] combined two streams LSTM networks, one is to decode spatio-temporal information in skeleton sequence, another is to transform viewpoints to address various of camera views.

From the perspective of 3D skeleton structure, the key joints are connected with each other by bone links, which contain rich spatial information. Hence, the GCN-based model shows another way of thinking to represent spatio-temporal information in a 3D skeleton sequence. Li et al. [27] adopted a ”two stream” strategy and proposed an action-structural graph convolution network to capture actional links and structural links, which can learn both spatial and temporal features for action recognition. To capture variation of spatio-temporal information in 3D skeleton data, Gao et al. [28] utilized spatio-temporal modeling of 3D skeleton data and applied optimization on consecutive frames for efficient spatio-temporal data representation. In [29], Papadopoulos et al. adopted two novel GCN-based models to capture vertex features and short/long-term temporal features for action recognition. However, such GCN-based methods generally have a low computational efficiency due to their complex network structures.

Generally, a 3D skeleton sequence can be easily encoded into one image and further decoded by a CNN-based model, thus the recent CNN-based works mostly make effort on exploring features in 3D skeleton data and represent them by images. Kim et al. [19] directly transformed frame-wise 3D coordinates of joints to images and decoded them by using a residual temporal CNN model. In [20], Li et al. manually transform raw 3D skeleton data to images by utilizing joint reference and projection, and employed a pre-trained VGG-19 model for end to end training. Despite the success on utilizing high level features, representing low-level features (e.g., joint angles, motion direction and magnitude) are also considered as an effective approach to improve the accuracy of 3D skeleton-based action recognition. Kim et al. [10] represented the features by using the spatial correlations of joints and temporal dynamics of 3D skeleton, and achieved remarkable performance. To eliminate the effect of camera view variations, Liu et al. [23] presented an skeleton feature enhanced method for view invariant action recognition, to further improve the performance, the skeleton motion enhancement methods are also applied. In [18], Li et al. separated the skeleton into several body parts and mapped them to images via a scale invariant transformation approach to eliminate the effect of different body sizes. Combining GCN and CNN is also a novel approach to explore spatio-temporal information in 3D skeleton data. Zhang et al. [30] developed an simple yet effective GCN-based model to enhance the joint feature by introducing semantics of joints and hierarchically exploit their relationship, and used CNN model to explore correlation across frames.

Generally speaking, the spatio-temporal information in 3D skeleton data can be easily encoded in image format and effectively decoded by CNN-based models. However, the existing approaches using handcrafted action feature images are difficult to handle challenging actions and generally require a large scale CNN model to decode spatio-temporal information in image patterns. Therefore, it is highly desirable to develop a more efficient method to encode action features that can be decoded by a light-weight CNN model for reducing computational burden.

As shown in Fig 1, our AFE-CNN contains two main models: Action Feature Enhance model and Action Recognition model. In the Action Feature Enhance model, there are four blocks: 1) Skeleton Sequence Image Transform block; 2) Multi-Frame enhance block; 3) Skeleton Motion Velocity Image Transform block and 4) Temporal Embedding block.

In Block 1, the input is 3D skeleton sequences and the outputs contain two parts. The first part has three images, namely Multi-Frame Attention Map (MFAM), Key-Joints Feature Enhance Image (KJEI) and Bone-Vectors Feature Enhance Image (BVEI). Specifically, MFAM is calculated from skeleton frames through the Multi-Frame Attention Module. KJEI specifies key joints of 3D skeleton sequences which is produced by the Key-Joints Feature Enhance Module. BVEI specifies bone vectors of 3D skeleton sequences through a Bone-Vectors Feature Enhance Module. The second part contains two matrices, namely $\mathbf{M}$ and $\mathbf{N}$, where $\mathbf{M}$ represents the enhanced key joints by the Key-Joints Feature Enhance Module, and $\mathbf{N}$ represents the enhanced bone vectors from the skeleton sequence by the Bone-Vectors Feature Enhance Module. In Block 2, the inputs are MFAM, KJEI and BVEI from block 1 and the outputs are two new images, namely the frame-enhanced KJEI (F-KJEI) and the frame-enhanced BVEI (F-BVEI). In Block 3, the inputs are $\mathbf{M}$ and $\mathbf{N}$. The outputs are two images, namely Key-Joints Motion Velocity Image (KJVI) and Bone-Vectors Motion Velocity Image (BVVI), which are generated by the joint velocity transform model. In Block 4, the Temporal Embedding module takes four images from blocks 2 and 3 as the inputs and generates four feature enhanced action-transformed images, namely temporal frame-enhanced KJEI (TF-KJEI), temporal frame-enhanced BVEI (TF-BVEI), temporal-enhanced KJVI (T-KJVI), and temporal-enhanced BVVI (T-BVVI), which include the temporal information of the skeleton sequence. The action recognition model consists of four light-weight convolutional neural networks and several fully connected layers. The inputs are the four enhanced action-transformed images, i.e., TF-KJEI, TF-BVEI, T-KJVI, and T-BVVI from Block 4 and the output is the action label.

In this section, we will discuss the details about five modules in the action feature enhance model.

In this section, we will discuss the details of CNN models used in our action recognition model and our training strategy.

NTU RGB+D: This dataset is recorded by Kinect v2 sensors and each skeleton is depicted by 3D locations of 25 body joints. In details, it contains 56,880 3D skeleton sequences with 40 different human subjects and covers 60 daily action categories, which including single person actions, human-objective interactions, and human-human interactions. It is a challenge dataset due to its large scale, diverse action categories and various of camera views. We evaluate our method on this dataset by using two official evaluation metrics: cross-subject (subjects with 20 specific IDs are for training and the remaining for testing) and cross-view (samples from camera 1 for testing while the samples from camera 2 and 3 for training).

NTU RGB+D 120: This dataset is the extended version of the NTU RGB+D by adding another 60 challenging action categories. Compared with NTU RGB+D, this dataset is more challenge due to more subjects and increased variations of view points. In details, it contains totally 114,480 3D skeleton sequences with 106 subjects which performers in a wide range of age distribution. These samples are captured from 155 different camera views and recorded in 32 different scenes. This dataset provides two evaluation metrics: cross-subject (subjects with 53 specific IDs are for training and remaining for testing) and cross-setup (samples with even collection setup IDs for training while remaining of odd setup IDs for testing).

UTKinect-Action3D: Compared with NTU RGB+D and NTU RGB+D 120, this dataset is significantly smaller. It has 200 3D skeleton sequences of 10 daily human-object interactive action categories. Each skeleton is recorded as 20 body joints. Due to its small-scale, we utilize the cross-subject (half of the subjects for training and the remaining subjects for testing) as the evaluation metric to prevent potential risk of model overfitting.

Here, we compare our AFE-CNN with several SOTA 3D skeleton action recognition methods on NTU RGB+D, NTU RGB+D 120 and UTKinect-Action3D respectively.

From the results shown in Table 1, our AFE-CNN achieves the highest accuracy on both cross-subject (86.2%) and cross-view (92.2%) evaluation metrics. Accordingly, the corresponding confusion matrix of NTU-RGB+D results under cross-subject metric and cross-view metric are depicted in Fig 5 and Fig 4 respectively. It can be observed that there is a large gap of accuracy between deep learning-based methods (e.g.,MTLN [41],ResTCN [19]) and traditional machine learning-based methods (e.g., TSRJI [11] and PAM+PTF [8]). Because the traditional machine learning-based methods are hard to handle large-scale action dataset, therefore, the most of 3D skeleton-based action recognition methods are developed with deep learning-based architectures (e.g., CNN, LSTM, GCN).

In these deep learning-based approaches, the LSTM-based and GCN-based architecture achieve applaud performance (e.g., ST-GCN [17], VA-LSTM [42]) due to they are expert on spatio-temporal information processing. However, compared with CNN-based methods, the LSTM-based and GCN-based methods consume more computing time due to their complex network structure.

For CNN-based architecture, few of methods directly feed raw coordinate data to a CNN model (e.g., TCN [19]), and the most of methods first transform different levels of action features (e.g., high-level action features [20, 15], low-level action features [11, 10, 18, 41], motion features [22, 21, 23]) to image format as inputs for CNN models. To further improve action recognition accuracy, several methods [18, 10, 21] employee large scale CNN model (e.g., ResNet, Inception-v3) to extract more spatio-temporal information in large size action feature images. Although these approaches significantly improve the performance thanks to the benefits of large scale CNNs feature extraction in image classification, they load more computation burden which largely increase computing time. Moreover, the hand-crafted action features are difficult to handle challenging data with various of camera views, body sizes, and they have become the bottleneck to further improve the performance. In contrast to these methods, our AFE-CNN outperforms all handcrafted action feature image approaches and only requires light-weight CNN models.

As shown in Table 2, our AFE-CNN outperforms than other methods on both cross-subject (80.2%) and cross-setup (81.6%). In this dataset, some methods utilize RNN-based[16] and LSTM-based[24, 25] methods and achieve promising results, and the best LSTM-based [25] method reaches 79.9% accuracy on cross-setup metric. In addition, the GCN-based [27, 29, 30] action recognition approaches achieve applaud performance due to GCNs are specialize on finding spatio-temporal information in raw 3D skeleton sequence. But they still suffer from a high computational burden. Compared with GCN and LSTM-based methods, our AFE-CNN achieves high accuracy of action recognition while ensuring a low computational burden.

Moreover, it can be seen that there is a large gap of accuracy between CNN-based methods and GCN-based methods (e.g., Gimme Signals [43] and GVFE+DH-TCN [29]). This verifies the CNN-based model is hard to handle more challenging dataset due to the limitations of hand-craft features transformed images (e.g., more various of camera views, body sizes and marginally different actions). However, our AFE-CNN can effectively enhance the action features of challenging actions that enables a light-weight CNN model to achieve outstanding performance.

From the results shown in Table 3, our AFE-CNN achieves the accuracy of 99.0% in this dataset and several methods [24, 15, 44] also achieve the same result. Accordingly, the corresponding confusion matrix of this dataset as shown in Fig 6, where some walk and wave action samples are misclassified to each other. In this small-scale dataset, some traditional machine learning-based methods [8, 9] even achieve comparable performance compared with other deep learning-based methods. The CNN-based methods [10, 15, 44] generally have an applaud performance while the GCN-based [13, 28] methods have a lower accuracy. In these CNN-based methods, only PoT2I+Inception-v3 [10] adopts a single CNN architecture, which cannot decode more action features and cause the action recognition accuracy less 0.5% than other CNN-based methods. Although our AFE-CNN utilize a light-weight CNN model, our four-streams CNN architecture ensure that more action features are encoded and decoded, therefore, our AFE-CNN also can achieve a promising performance in small-scale dataset.

In this section, we carry out several ablation studies on NTU-RGB+D dataset to validate the contribution of key modules of AFE-CNN.

In this section, we analysis the complexity of our AFE-CNN by measuring its computing time, floating-point operations per second (Flops) and further compare with several representative methods. It is noted that a comparison of processing time cannot be done fairly due to the diversity in use of frameworks (e.g., TensorFlow, PyTorch) and computing platforms (e.g., single GPU and multi-GPUs) are very diverse.

As shown in Table 7, our AFE-CNN only cost 3.5ms for one forward inference and only consumes 1.4GFlops, which are significantly lower than other methods. Compared with other methods [10, 21, 23] using handcrafted action feature images, our AFE-CNN can be fully executed on GPU. Since we adopt a light-weight CNN model as the back bone, our method can minimize the computing time. Although the GCN-based model has applaudable performance on 3D skeleton-based action recognition task, it costs more computing time due to its complex computational structure.

To illustrate our action feature enhance mechanism, we visualize the action feature enhanced images and multi-frame attention maps of drink water and jump up actions. We further compare them with the action feature images without utilizing any proposed feature enhance module.

As shown in Fig 8, we can observe that the strips in action feature enhanced images are much more distinguishable than the images without being enhanced by our proposed feature enhance modules. Moreover, it is obviously that the TF-KJEI, TF-BVEI, T-KJVI, and T-BVVI encode different information in an action. Thus the CNN model can lead to outstanding performance thanks to the rich features in these action feature enhanced images. For MFAM, we can find that it successfully finds the key frames in an action sequence, where the yellow color indicates a stronger attention.