A Temporal Densely Connected Recurrent Network for Event-based Human Pose Estimation

Human pose estimation from still images in early works usually starts from building the parts-based graphic models or pictorial structure models [24, 25, 26, 27] to learn spatial relationships between articulated body parts. Recently, the performance of these earlier works have been surpassed largely thanks to the great success of deep convolutional networks [28, 18, 3, 1, 4, 29, 2, 30] that provide dominant solutions nowadays. Apart from image-based pose estimation, many efforts [18, 17, 31, 32] has also been made to exploit temporal and motion information for human pose estimation from videos by using optical flow or 3D CNNs, which are related to our work since that we consider to accumulate event signals into a sequence of event frames. However, these methods have the limited ability to extract temporal contexts explicitly. More recently, recurrent architectures [18, 21, 20, 23, 22, 33] is normally integrated with an encoder-decoder CNN framework to model the temporal dependency across frames to refine pose predictions, which have been pioneering frameworks for video-base human pose estimation. Such kind of frameworks share a general structure, where CNNs are normally used to encode and decode every frame sequentially, and a recurrent mechanism is then introduced to temporally link the encoder-decoder streams along time steps to propagate temporal dynamics between neighboring frames. However, they all just model the temporal dependency between two consecutive frames, which are not always effective for human pose estimation from event signals, because in an event video we also need to consider a long-range geometric consistency across a set of frames in a time window. DCPose [33] leverages the temporal cues between past, current, and next frames to facilitate keypoint prediction. However, it still models a short-range temporal dependency, and meanwhile it has to depends on future frames. Similarly, most recent work FAMI-Pose [liu2022temporal] leverages a hierarchical alignment framework to update a short-range of neighboring frames (e.g., 2 previous and 2 future frames) to align with the current frame at feature level.

Different from existing methods, our model introduces dense connections across a set of consecutive frames rather than one single temporal connection between the current and last frames to help recover human poses in event streams.

Most existing methods and algorithms in vision research are based on the conventional frame-based cameras, where the cameras output a sequence of synchronous frames at a fixed rate. The frame contains the entire information in scene, thereby producing redundant vision signals especially when there are only very little changes in the scene between consecutive frames. In contrary, event cameras [6, 5], a new type of bio-inspired vision sensors that have emerged in the last few years, output a signed ‘event’ signal $e_{i}$ at the pixel location of $(u_{i},v_{i})$ only when the detected changes in the log brightness exceeds a predefined threshold $C$ $(\left|\Delta log(I(u_{i},v_{i},t_{i}))\right|>C)$ at time instance $t_{i}$ with the polarity $\rho_{i}$ $(\rho\in\{-1,+1\})$. Then, an event signal can be represented as $e_{i}=(u_{i},v_{i},t_{i},\rho_{i})$, where positive events $(\rho_{i}=+1)$ indicates the brightness increase $(\Delta log(I)>C)$ and negative events $(\rho_{i}=-1)$ indicates the brightness decrease $(\Delta log(I)<-C)$, as shown in Figure 2.

Thus, unlike conventional cameras, event cameras produce a sequence of asynchronous events because they asynchronously sample light of each pixel independently. As a result, the events are spatially much sparser in comparison with conventional frame-based cameras, where each frame is generated by densely sampling the entire pixels at the same time. Hence, event camera is able to best capture local motions in the scene as a stream of sparse and asynchronous events. Event cameras have unique advantages of very high temporal resolution, high dynamic range, low latency, and low power consumption. Hence, event cameras have stimulated a variety of research activities and applications in computer vision [5], including visual SLAM [10], optical flow estimation [13], object tracking [8], object recognition [14, 15, 34], gait recognition [12], and high-speed maneuvers [35], among others.

For human pose estimation, there has been little investigations into applying event cameras to capture and estimate human poses. DHP19 [7], EventCap [36], and EventHPE [19] are perhaps most related to our work. In DHP19 [7], a simple encoder-decoder CNN is designed to estimate human poses from single integrated event frames. The method normally fails in those frames that contain only few visible body parts. Other two related works EventCap and EventHPE are mainly dedicated to address 3D human motion capture and shape recovery rather than pose estimation from event streams. In EventCap [36], it aims to achieve markerless human motion capture, which however requires a necessary additional input of gray-scale images apart from event signals. EventHPE [19] can achieve human pose and shape estimation simultaneously with the solo event source. However, it needs to fuse an event sequence and its corresponding optical flows, where optical flows require an extra FlowNet to infer. Moreover, a beginning body posture at the first frame of gray-scale images is necessary for this method. Compared with them, our approach only takes in event signals as the sole input source to address the human pose estimation with a light model.

There are already many RGB-based datasets for human pose estimation, including images and videos, such as COCO [37], MPII [38], LSP [39], PoseTrack [40], etc., which have largely facilitated the evaluations of RGB-based algorithms. However, event-based human pose datasets have been rarely curated. There are only two existing datasets, DHP19 [7] and MMHPSD [19], which are commonly used to evaluate the pose estimation. DHP19 contains total of 33 action categories, but most of them are simple slow motions such as hand waving and leg swinging so that a very simple CNN can even saturate on this dataset. The MMHPSD dataset is a multimodal dataset that contains more diverse actions with fast, medium, and slow motions. However, the 2D joint annotations are roughly obtained by using OpenPose [4] as MMHPSD dataset was initially created for 3D human shape estimation. Moreover, both datasets were acquired indoors under a fixed environment with clean backgrounds. Therefore, we curated a large-scale multimodal event-based dataset CDEHP that are captured in outdoor environments with uncontrolled lighting conditions.

We follow the widely-used pipelines [28, 1, 42, 21] to predict the body keypoints by using an encoder-decoder architecture, which usually consists of a series of convolutional layers followed by a couple of upsampling and deconvolutional layers to generate high-resolution spatial feature maps. A few of regression layers take the feature maps as input to finally estimate the heatmaps for all keypoints. We adopt the convolutional network in [1], i.e., SimpleBaseline, as our base network to extract the feature maps for each frame to generate heatmaps, as the SimpleBaseline [1] is a very simple but effective convolutional network for human pose estimation from still images.

SimpleBaseline simply uses the ResNet [43] as the backbone to extract high-level image feature and then typically adds three transposed convolution layers (deconvolution) to recover the high-resolution spatial feature maps. Specifically, there are $256$ filters with a $4\times 4$ kernel and a stride of 2 in each deconvolutional layer. An $1\times 1$ convolutional layer is finally adopted at last to predict the heatmaps of the body keypoints based on the feature maps. In our architecture, we use the abbreviation ConvNet2 to indicate such an $1\times 1$ convolutional layer, while we use the abbreviation ConvNet1 to indicate the convolutional block composed of the ResNet and three deconvolution layers.

To build our recurrent network, we use an LSTM module to link the convolutional base network in temporal dimension, as illustrated in Figure 3, where we choose to insert the LSTM module between ConvNet1 and ConvNet2. By such recurrent style model, the parameters of ConvNet1 and ConvNet2 are shared across different time steps. Since the LSTM module take the convolutional feature maps generated by the ConvNet1 as input, we leverage two convolutional LSTM layers [44] without peephole connections as our LSTM module, where the operations inside it are defined as:

Unlike the standard LSTM, ’*’ here denotes a convolutional operation similar with [44]. As a result, all the ’+’ in Eq. 1 represent the element-wise addition. $X_{t}$ denotes the input feature, while ${\bf{W}}_{**}$ denotes the weights of the LSTM module. $\epsilon_{*}$ denote the bias terms. $\delta$ and $\varphi$ represent the activation functions $sigmoid(\cdot)$ and $tanh(\cdot)$, respectively. $i_{t}$, $f_{t}$, $o_{t}$, and $h_{t}$ represents the input gate, forget gate, output gate, and hidden state, respectively. Such convolutional design of LSTM gates allows for more attentions on the regional context rather than global information, and it can effectively capture the local changes of joints.

We adopt two such convolutional LSTM layers with $3\times 3$ kernels as our LSTM module, referred to as the function $\mathpzc{L\left(\cdot\right)}$. Successive event frames in an event video are sent into the feature extractor ConvNet1 to produce the feature maps of size $W^{\prime}\times H^{\prime}\times M$ as the input of the LSTM module across each time step. Lastly, the outputs of the LSTM module are sent into the heatmap generator ConvNet2 with an $1\times 1$ convolutional layer to predict the heatmaps for each frame. Mathematically, we denote the computation of ConvNet1 and ConvNet2 consistently used in all time steps as the functions $\mathpzc{F}\left(\cdot\right)$ and $\mathpzc{g}\left(\cdot\right)$, respectively, and $I_{t}\in\mathbb{R}^{W\times H}$ is the original input event frame at time step $t$. Denote $\mathbf{b}_{t}\in\mathbb{R}^{W^{\prime}\times H^{\prime}\times K}$ as the heatmaps in time step $t$. The LSTM-based recurrent network can be formulated mathematically as following,

The key strategy in this paper is that we propose a novel densely connectivity pattern as shown in Figure 3 in the recurrent network to pass all the preceding heatmaps to the next time step for explicitly building sequential and non-sequential geometric consistency in an event video. We introduce direct connections from the heatmaps in every time step to the inputs of LSTM modules in all subsequent time steps. In this way, the $t$-th LSTM module $\mathpzc{L\left(\cdot\right)}$ receives the heatmaps in all preceding time steps. All these past heatmaps are simply accumulated as one heatmap, which is then concatenated (indicated by the operation $\oplus$) with the feature map extracted by ConvNet1 $\mathpzc{F}\left(\cdot\right)$ in the $t-th$ step as input to the LSTM module. Mathematically, a new recurrent architecture with the proposed temporal dense connections can be formulated as:

The preceding frames may play different important roles in predicting poses at current frame. Meanwhile, the visible body parts in the previous frames also may make different contributions on completing the lost body parts in the current frames. Therefore, we introduce an attention weight matrix into the fusion of the preceding heatmaps during the propagation. In addition, it should be noted that there are not prior heatmaps when $t=0$. Thus, we have to employ the base network (ConvNet1 + ConvNet2) to generate the heatmap at the first frame to provide initialized prior input to LSTM module for the first frame self, as illustrated in Figure 3. Consequently, we have:

where $\mathbf{b}_{t}\in\mathbb{R}^{W^{\prime}\times H^{\prime}\times K}$, and $\mathbf{W}^{t}_{\tau}$ is the attention matrix to weight the heatmap of the $\tau$-th frame to fuse with the feature maps at $t$-th frame. $\odot$ indicates the element-wise multiplication that means the spatial attention on different body parts in the corresponding frame is also applied. Such spatio-temporal attention mechanism is introduced into the dense connections to pay different importance to the preceding frames and their spatial joints when fusing their information to the current frame. Because of such temporal connectivity, we refer to this recurrent architecture as Temporal Densely Connected Recurrent Network (tDenseRNN). This new architecture allows for both the sequential and non-sequential temporal dependency modeling thanks to such skipped dense connections rather than only sequential connections between two consecutive frames in [18, 20, 22].

For the ground truth heatmap for $k$-th joint, we transform the label in Cartesian coordinates for the $k$-th joint into a heatmap by placing a Gaussian peak at the center of the joint locations. We have $T$ time steps in the recurrent network, where $T$ is the number of consecutive frames in each training sequence. The total loss is accumulated at the end of each time step to supervise the network training. In the training, we aim to minimize such loss with the $l_{2}$ distance between the predicted heatmaps and ground truth heatmaps for all joints and all frames jointly,

where $\mathbf{b}_{t}(k)$ and $\mathbf{b}_{t}^{*}(k)$ are the predicted heatmaps and ground truth heatmaps for $k$-th joint at $t$-th time step, respectively.

A multimodal human pose dataset captured in outdoor scenes, CDEHP, has been curated to evaluate our approach, because there is not a very challenging dataset captured in different environments to facilitate the evaluation of our approach which easily saturates on the existing datasets with simple actions captured in a fixed indoor environment.

Data acquisition. The CDEHP dataset was collected from different outdoor environments under varying light conditions by using a multiple camera system. The multiple camera system contains 2 different imaging sensors, including one event camera CeleX-V [8] and one RGB-D camera Intel RealSense D435i, which could output a sequence of event streams, RGB color frames, and depth frames simultaneously. Specifically, the resolution of RGB-D camera (Intel RealSense D435i) is set to be 840*480 with the frame rate of 60, while the resolution of the event camera (Celex-V) is set to be 1280*800. The samples in the CDEHP dataset are captured in four different outdoor environments by 20 subjects, with 15 being male and 5 being female. Each subject performs 25 different actions with varying speeds (slow/medium/fast) for $3\sim 4$ times, as listed in TABLE I. Finally, this amounts to 500 video samples collected in total, where each video sample contains RGB video sequences, depth video sequences, and event streams. Overall, this amounts to be a total of 82K frames in the dataset in terms of the number of RGB-D frames. Some action samples with RGB and event modalities are shown in a figure of Appendix.

Camera calibration. In our multiple camera system, we use three imaging cameras, including event camera, depth, and RGB cameras, where depth and RGB cameras are integrated into one single RGB-D camera D435i. We need to calibrate the intrinsic and extrinsic parameters of these cameras in order to obtain the transformations between camera spaces. First, we calibrate the intrinsic parameters of the event camera, where particularly gray-scale images rather than event streams are used during the calibration. For the RGB-D camera, we directly use the intrinsic and extrinsic parameters provided by the manufacture for RGB and depth cameras. Lastly, we calibrate the relative mapping matrix of the event camera space to the RGB-D camera space while the intrinsic and extrinsic parameters of RGB-D cameras are fixed during calibration. In this way, we can directly project the joint locations in the 3D space of RGB-D camera to the 2D space of event camera.

Annotation. Currently, there are few event-based datasets for human pose estimation. One main reason is that it is not easy to annotate on event streams when dataset is captured outdoors. Most of the existing datasets are collected indoors using a set of motion capture sensors to assist annotation, which limits the diversity of human actions. Therefore, we use a simple but effective multiple camera system that are easily to annotate the joint locations. In our multiple camera system, we need to calibrate the intrinsic and extrinsic parameters of these cameras in order to obtain the transformations between camera spaces. With the calibration, we first manually annotate the ground truth of 2D joints on the RGB frames. The depth of each 2D joint is then obtained by warping its corresponding depth image to the RGB frame. Accordingly, we can have 3D joint locations in the 3D space of RGB-D camera by using the intrinsic parameters of depth camera. Lastly, it is straightforward to obtain the 2D joint locations in event camera space by projecting the annotations in RGB-D camera space based on the calibrated matrix. It should be noted that since that we integrate the event stream to a sequence of successive frames in a fixed time interval, we temporally align the RGB-based annotations to the event frames according to their frame rates as in [7]. To implement the annotation, we developed a interactive GUI tool as illustrated in a figure of Appendix. In addition, some exemplar annotated action samples can be found in the Appendix.

Data comparison. We compare our dataset with two other existing datasets. As shown in TABLE II, although our data is not the largest one in terms of the number of frames, we have a more diversity in action classes, as listed in TABLE I. Moreover, our dataset is acquired in multiple outdoor scenes under varying environments rather than a fixed environment. Our dataset also can offer the possibility of 3D pose estimation in future, because we have the annotations of 3D joint locations by using the depth camera.

Event frame and video generation. In order to leverage the frame-based deep learning techniques, we need to convert asynchronous events into frames, referred as event frames. To do that, we follow the strategy in [9] to simply accumulate the events at the fixed time interval of 8.333 $ms$ in a pixel-wise manner to obtain 2D histograms of events as an event frame. According to this procedure, around 82K event frames are generated in ours CDEHP dataset. A set of consecutive frames generated from each event stream compose an event video. Using these videos, we segment them into a number of successive short video clips of the length $T$ same with the temporal length of our recurrent network. These short video clips are then used to train and test our model.

Datasets. We evaluate our method on three datasets, DHP19[7], MMHPSD[19], and CDEHP. DHP19 is the first dataset collected for event-based human pose estimation. It contains a total of 33 actions, where each action is performed 10 times by 17 subjects (12 female and 5 male). The DHP 19 dataset was recorded from four event cameras located in different four views. We only adopt the event data from a single camera view (camera view 3 as illustrated in [7]) to evaluate our approach, while our paper focuses on the 2D pose estimation from event data. Following the previous work [7], we split 17 subjects into 12 subjects for training and 5 subjects for testing for evaluation. MMHPSD is a latest event-based human pose estimation dataset which was initially created for 3D shape estimation, where 4 different types of visual sensors, including one event camera, one polarization camera, and five RGB-D cameras, are used to capture the actions. We utilize 2D joints in the pre-processed data, issued publicly by the authors of [19], as the pose annotations, which are obtained by projecting the fine-tuned 3D shapes constructed from the annotations of multi-view RGB frames [19]. In MMHPSD, there are 15 subjects performing 3 groups of actions (21 actions in total) for 4 times, where each group includes actions with slow/medium/fast speed respectively. The detailed action classes of the MMHPSD dataset can be found in [19]. Following the work [19], we split 15 subjects into 12 subjects for training and 3 subjects for testing. For our CDEHP dataset, its details has been introduced in Section III-E. For evaluation, we split it a training set from 15 subjects and a testing set from the remained 5 subjects.

Implementation. In the experiments, all event frames are cropped to a fixed size of $256\times 256$ with the human bodies set at center. Random rotation is simply used for the data augmentation. During training, the temporal length of our recurrent network is set to be 16 (i.e., $T=16$), which is found to be large enough to obtain sufficient information for completing lost events, achieving the best performance. Adam optimizer with the learning rate of $5e$-5 and the weight decay of $1e$-4 is used to optimize the learning process. If the training loss does not continue to decrease in $5$ epochs, the learning rate is then reduced by half until the learning rate is less than $1e$-6 and then the training ends. The model is carried out on four NVIDIA 2080Ti GPUs.

Evaluation metrics. The commonly used standard evaluation metric is the object keypoint similarity (OKS) [3], which is calculated based on the Euclidean distance between a detected joint and its ground truth. We report a set of standard average precision scores: $\text{AP}^{50}$ (AP at OKS = 0.50), $\text{AP}^{75}$ (AP at OKS = 0.75), and AP (AP at $\text{OKS}=0.50,0.55,...,0.90,0.95$). In addition, we also adopt the percentage of correct keypoints (PCK) metric, which reports the percentage of correct keypoint detection. A detected joint is considered as correct if it falls within a normalized distance to the ground truth. Nevertheless, on the DHP19 dataset, the PCK and AP tend to saturate on this dataset by using even a very simple network, since that they both can reach almost 100% when every model listed in TABLE III has just started the training for a few epochs. Therefore, we employ the mean per joint position error (MPJPE) as the evaluation metric, which represents the average Euclidean distance between ground truth and prediction, which is usually measured with pixel distances in 2D image space, i.e., $\frac{1}{K}\sum_{k}^{K}\left\|p_{i}-p^{*}_{i}\right\|$. $p_{k}$ and $p^{*}_{k}$ denote the ground truth and predicted position of the $k$-th joint in image space.

Effect of the temporal length. Temporal lengths $T$ in Eq. 5 is a hyper-parameter that refers to the number of successive frames input to the recurrent network for the training. We experiment with the varying temporal lengths, i.e., $T=2,4,8,16$, and report the results in terms of the PCK metric on the three datasets, as recorded in Figure 5. The figure shows the effects of different temporal lengths on the model performance with increasing epochs in the training process. We can observe the best performance on CDEHP dataset is achieved with $T=8,16$, while on other two datasets the best results are achieved with $T=16$. The results with the shorter temporal lengths decrease significantly especially when $T=2,4$. This study reveals that with the proposed densely connections in a long range are effective to pass previous useful information to help predict the pose at the current frame. But, it doesn’t mean that the longer the temporal length is, the better performance is, while on CDEHP and DHP 19 datasets we can see the performance with $T=8$ is close to the one with $T=16$. It exhibits that the information of the preceding frames that are very far away from the current frame does not contain effective information as useful as the near frames. Meanwhile, a longer temporal length could bring a higher computational cost accordingly. Therefore, we do not try to set up a higher temporal length over the current setting $T=16$, as we set $T=16$ in this paper.

Contribution of each component. We conduct extensive experiments to study the effect of each component in our model. First, we insert a convolutional LSTM module between ConvNet1 and ConvNet2 to simply create a standard recurrent neural network for modeling the temporal dependency across frames, as stated in Eq. 2. We consider this model as our baseline, denoted as RNN. Next, we incorporate one single connection between every two adjacent frames in the baseline model, where the output heatmap at the last frame are passed to the current frame to concatenate with the features as newly input to the LSTM module. We denote this model as tThinRNN. Instead of one single connection, the dense connection is introduced to the baseline model, as described in Eq. 3, which is denoted as tDenseRNN-w/o-AT. Lastly, the attention mechanism is furthermore incorporated into the tDenseRNN-w/o-AT model to form our full model, as stated in Eq. 5. We use tDenseRNN to denote our full model. For all above models, we fix the backbone of the encoder in ConvNet1 as ResNet-18.

From the results on all datasets, as listed in TABLE III, we can see using RNN already serves as a simple and strong baseline, achieving 5.36 MPJPE on DHP19, 84.15 AP on MMHPSD, and 77.97 AP on CDEHP dataset. Especially on MMHPSD and DHP19 dataset, our all models can achieve a set of similar performance with marginal differences, because actions in MMHPSD and DHP19 are not very challenging so that our models are easy to saturate on these two datasets. Therefore, in the following experimental study, we only analyze the results on our CDEHP dataset. For tThinRNN, we can find that it does not improve much in performance and even its AP drops a little (-0.30 AP) in comparison with the RNN model. By adding dense connections, tDenseRNN-w/o-AT can outperform tThinRNN by a distinct margin of +1.63 AP. Furthermore, tDenseRNN with the attention mechanism achieve the best performance, where AP is improved from 79.30 to 80.24. In terms of PCK, we even improve the performance by a large margin of +1.30 PCK compared to tDenseRNN-w/o-AT. These results verify the strength of our dense connections with the attention mechanism for the pose estimation from event video.

Pose augmentation. Motivated by [45], we introduce a simple approach to augment the training poses of our dataset towards a greater diversity and thus improve the generalization of our model. Specifically, we augment the pose events $\{u_{i},v_{i}\}$ in an event frame $I_{t}$ by rotating those events by the joints of knees and elbows, generating new pose events, as illustrated in Figure 6,

where $(u_{i}^{a},v_{i}^{a})$ is the new locations of events in the augmented poses and $(u_{c},v_{c})$ is the center point of rotation. $\theta\in\left(-90,90\right)$ is set randomly.

Our model trained with the additional augmented training poses, denoted as tDenseRNN+PoseAug, can achieve significant improvements over the model tDenseRNN on CDEHP and HDP19 dataset, which demonstrates the diversity of training data is quite useful to obtain a greater generalization of the model. It should be noted that we do not augment the data for MMHPSD dataset, because the raw data of MMHPSD dataset have not been available publicly so far.

Results on DHP19. TABLE III summarizes the results on the DHP19 datasets, where we can see our baselines and the full model tDenseRNN achieve the best performance compared to other competitors. However, on DHP19 all methods achieve nearly same level of performance since that there are only pixel errors of $1\sim 2$ in terms of the used metric of 2D MPJPE, while they can produce almost the result of 100% in terms of PCK and AP metrics on DHP19. Therefore, obviously the DHP19 cannot be used to evaluate our method, because the included actions in this dataset are too simple so that it has not be a challenging dataset as the evaluation benchmark for pose estimation.

Results on CDEHP. TABLE III shows the comparisons of our model with state-of-the-arts on the CDEHP dataset, in terms of PCK and AP evaluation metrics. We can see that our tDenseRNN model achieves the best performance. Among those competitors, HigherHRNet [3] ,Hourglass [28] and SimpleBaseline [1] are three representative methods in image-based pose estimation, which achieve promising results of 75.60 AP, 75.87 AP and and 77.51 AP, respectively. DKD [20], DCPose[33], and FAMI-Pose [liu2022temporal] are three pioneering methods for video-based pose estimation, and they outperform those image-based methods by noticeable margins (+5.23 PCK, +3.24 PCK, and 5.34 PCK compared to HigherHRNet, respectively). This result implies that the temporal information can provide useful motion clues for event-based pose estimation. LSTM-CPM [18] is also one of the strong baselines for video-based pose estimation method, which however does not produce an expected promising result (40.99 AP). This is may because it inherits the convolutional architecture from CPM [46] using a multi-stage refining scheme, which focuses on designing large receptive fields in multi-stage CNNs to capture long-range spatial dependencies that are not applicable for event frames with the absence of some body parts. Similar to DKD, DCPose and FAMI-Pose, RNN and tThinRNN introduce the recurrent module to model the temporal dependency achieving comparable results with them. By comparing them with our models tDenseRNN-w/o-AT and tDenseRNN, we can find our models always maintain high performance across all evaluation metrics, especially $\text{AP}^{75}$, thanks to the dense connections that are not considered in DKD, DCPose and FAMI-Pose, which only leverage a couple of adjacent frames.

Results on MMHPSD. As shown in TABLE III, we have the similar observation with the results on CDEHP. video-based methods including ours can achieve promising results thanks to the temporal information considered, in comparison with image-based methods [28, 1, 3]. Different to the results on CDEHP, the overall better performance results can be obtained on MMHPSD across all methods, because the actions are simpler than CDEHP dataset and captured in a fixed environment. Meanwhile, our methods outperform the competitive methods including DKD, DCpose and FAMI-Pose [20, 33, liu2022temporal] by a very noticeable margin, showing the strength of the temporal dense connection across frames.

Action-wise result comparison on CDEHP dataset. To analyze the results on various types of actions, we report the action-wise results on CDEHP dataset, as list in TABLE IV. We can observe that most of the challenges come from those slow/medium actions and complex actions, since that the former actions could produce more sparse event signals and the latter actions can bring severe self-occlusion of body parts in some cases. Therefore, a better performance can be obtained on those simple actions with fast speeds, as shown in TABLE IV. From the results, we can see that DKD [20] and FAMI-Pose [liu2022temporal] are two methods that achieve promising results compared with our methods, while they can achieve the second/third best and even the best in some cases. In addition, almost all the methods can obtain the similar good performance on those simple actions, for example, squat jump, jumping jack and big jump, while our methods (tDenseRNN) can particularly outperform other competitive methods by a large margin on those complex and slow actions, such as crawling, sit up jumping, burpee, and cartwheel. Moreover, for those actions that a few body part is moving, our method can also achieve superior results with a large performance improvement over other methods, such as throwing and mopping. Moreover, we can obverse the performance improvements between our two baselines tDenseRNN and tThinRNN on those slow actions specifically, where the performance of TDenseRNN can usually gain noticeable improvements over tThinRNN thanks to the dense connections, such as walking, crawling, sweeping. Overall, with our method the best or second results are obtained in all types of actions. This action-wise result demonstrates that our method, to some extent, can effectively alleviate the problem of the sparse and incomplete observations from event cameras. Meanwhile, it is observed that pose estimation from complex actions can also be addressed by our method.

Result Visualization. Some qualitative results are shown in Figure 7 to visualize the effectiveness of our model for event-based human pose estimation on CDEHP dataset compared to the baseline model RNN. We can observe that tDenseRNN can still accurately estimate the body joints when there are invisible body parts in event frames, as shown in Figure  7 (a)(d). In addition, for those complex actions, tDenseRNN can effectively handle the self-occlusions by encouraging the geometric consistency in the presence of fast and large-degree motion variations, as shown in Figure 7 (b)(c)(e). These visualized results further verify the strength of our model tDenseRNN.

Attention Visualization. We also visualize the attention maps on two example sequences to understand the key frames where information is densely propagated, as illustrated in Figure 8. In the first example as shown in Figure 8a, one of the foots are not complete or even disappeared at some frames. On one hand, the attention regions on the first frame are mainly distributed in those key joints, where especially the information of the foot joint can be propagated to help complete the foot information in the subsequent frames. On the other hand, each frame can receive attentions from a set of preceding frames, as shown in the bottom row of Figure 8a, where all the 7 preceding frames are combined to predict the human poses at 8th frame, since that body moving is continuous and temporally dependent. In the second example as shown in Figure 8b, the event signals at the 7th and 9th frames are very sparse. In this case, the preceding frames are very useful to alleviate the problem of incomplete event information for the human body. Especially, at the 1st frame one of the feet is visible, whereas at 5th and 6th frames that foot is not visible. Therefore, the attention regions at that foot area of the 1st frame to the 5th and 6th frames are much brighter to pass more information of that foot to the 5th and 6th frames. The visualization results show the effectiveness of the attention mechanism used in temporally dense connections.

Through extensive experiments above, we demonstrate the effectiveness of our method in addressing the challenges of event-based human pose estimation. As our approach converts event streams into a set of event frames, approaches for normal video-based human pose estimation, such as DKD, DCPose, and FAMI-Pose [20, 33, liu2022temporal], are directly employed to event frames as comparisons, achieving competitive results as shown in TABLE III and TABLE IV. Even that, we do not adopt those complex schemes used in the video-based approaches to fuse and align neighboring frames with the current frame. Instead, we build a simple densely connected RNN architecture to achieve the best performance. Hence, we have demonstrated that leveraging a longer range of neighboring frames is a simple yet effective way to alleviate the problem of incomplete information in event-based human pose estimation. Furthermore, taking future frames into count, such done in DCPose and FAMI-Pose, is not a natural way to perform online tasks, since one of the advantages of using event cameras is their ability to operate in real-time manner.

This paper presents a temporal densely connected recurrent network for human pose estimation from event data. There are some potential limitations in our method. First, our method cannot work directly on raw event signals, which have to be converted into frames first. Second, for those unnormal actions like cartwheel, where the human body is upside down rather than usually upright, we did not collect enough samples as many as the normal actions to train the model on CDEHP dataset. For future work, we will focus on estimating human poses from raw event signals by leveraging point-based methods, and try to extend the action samples to make the dataset more diverse. Finally, as shown in the results, and to the best of our knowledge, CDEHP is still the most challenging dataset in human pose estimation from event data. There is still plenty of room for improvement and problems remained, even though our method can help alleviate the problem brought by event signals in event-based human pose estimation, as the event signals are too sparse and irregular in spatio-temporal space when we try to encode complex actions. In the future, we will try to distill knowledge from other modalities, such as RGB and depth cameras, to help extract and encode event signals.