TAPE: Temporal Attention-based Probabilistic human pose and shape Estimation

Most related to our work are methods for human pose and shape prediction leveraging the temporal aspect of a video. Kanazawa et al. [13] proposed a regression-based method to learn human motion kinematics by predicting past and future frames with a temporal encoder. Kocabas et al. [15] proposed VIBE, a temporal encoder with Gated Recurrent Units (GRUs) to capture the motion between the static features and a motion discriminator trained with the AMASS [25] dataset for adversarial training. TCMR [4] reduced drastically the acceleration error of VIBE by using a temporal encoder which consists of 3 GRUs, one for the past, one for the future and one for the current frame and then integrating the features to produce a smooth video result with better pose and acceleration estimation than previous works. Recently, MPS-Net [34] proposed a motion continuity attention (MoCA) module which captures the continuity between frames and a hierarchical attentive feature integration (HAFI) module to effectively combine adjacent past and future feature representations to strengthen temporal correlation and refine the feature representation of the current frame. MPS-Net achieves temporally coherent pose estimates without penalizing the accuracy on pose prediction. Our method adopts the MoCA and the HAFI modules from MPS-Net as well as the motion discriminator in VIBE.

Biggs et al. [1] extend HMR [12] with $N$ prediction heads. This leads to a discrete set of hypotheses, instead of a full probability of poses as we do. In a concurrent work, Sengupta et al. [32] use a Gaussian posterior to model the uncertainty in the parameter prediction. Recently, ProHMR [19] used Normalizing Flows to predict a distribution of 3D poses conditioned on the provided 2D input. The probabilistic modeling of ProHMR is efficient at computing the most likely pose comparable to the SOTA methods and also outperforms previous work on optimization tasks such as 2D keypoint fitting and multi-view refinement. Our method uses the probabilistic model of ProHMR for video input.

We use a pretrained ResNet-50 [8] backbone from ProHMR that was trained on standard human pose and shape estimation datasets. The backbone extracts static features for each frame of the video. We consider $T=16$ frames at a time and, following Wei et al. [34], the features are sent to the Motion Continuity Attention module that outputs $T$ temporal features. The $T$ temporal features are integrated to a single feature context vector $c$ by the Hierarchical Feature Integration module proposed by Wei et al. [34].

Following Kolotouros et al. [19], we model $p(\theta|I)$ using Conditional Normalizing Flows. We learn a mapping $f:\mathbb{R}^{d}\times\mathbb{R}^{c}\to\mathbb{R}^{d}$ that is bijective in the latent variable $z$ and the pose parameters $\theta$, and is parametrized by the image features $g=c(I)$. More specifically, $\theta=f(z;c)$ and $z=f^{-1}(\theta;c)$. The choice of this particular model class enables to perform both fast likelihood computation and sampling from the distribution. Another useful property is that, as shown in Kolotouros et al. [19], the mode of the output distribution is the transformation on $z=0$, i.e.,

which means that in the absence of additional evidence, the probabilistic model can be used to make predictions by choosing the sample with the maximum probability.

Camera parameters $\pi$ and SMPL shape parameters $\beta$ are regressed by a small pretrained MLP as in ProHMR. The MLP takes as input the context vector $c$ and outputs matrices of shape and camera parameters per frame.

We train a motion discriminator as in Kocabas et al. [15] using the AMASS dataset [25]. The motion discriminator enforces the generator, i.e. our network, to produce plausible human motions and shapes. The discriminator takes as input the poses $\Theta$ that the generator predicted and outputs a value $\in[0,1]$ representing the probability that $\hat{\Theta}$ belongs to the manifold of plausible human motions. The motion discriminator consists of a GRU with 2 layers and hidden size 1024. The aggregation of hidden states is done by a self-attention mechanism. The objective function for training the Motion Discriminator is:

where $p_{R}$ is a real motion sequence from the AMASS dataset, while $p_{G}$ is a generated motion sequence. Since $D_{M}$ is trained on ground-truth poses, it also learns plausible body pose configurations. Therefore, the pose discriminator that was used in ProHMR is not needed in our method.

Due to lack of ground truth SMPL annotations in some of the datasets, we use mixed training following ProHMR. When SMPL data are available, we minimize the negative log-likelihood of the ground truth examples $\theta_{gt}$

In every dataset, we minimize the reprojection loss jointly with an adversarial motion prior based on the motion discriminator described in Section 3.4. In notation,

where

Following ProHMR, we use 6D representation [37] to model rotations and the $L_{orth}$ loss to force the 6D representations of the samples drawn from the distribution to be close to the orthonormal 6D representation.

The final training objective becomes:

The loss weights are defined as: $\lambda_{nll}=0.001$, $\lambda_{exp,2D}=0.001$, $\lambda_{exp,adv}=\lambda_{mode,adv}=0.01$, $\lambda_{mode,2D}=0.01$, $\lambda_{mode,3D}=0.05$, $\lambda_{mode,\theta}=0.001$, $\lambda_{mode,\beta}=0.0005$ and $\lambda_{orth}=0.1$.

Extending the model fitting procedure in ProHMR to the temporal domain, we use a video-based pose prior that models the likelihood of the pose at a specific frame conditioned on the video evidence:

As initialization for the fitting, we use the mode $\theta^{*}_{I}$ of the conditional distribution calculated from the regression step.

Following SMPLify [2], $E_{J}$ penalizes the weighted 2D distance between the projected model joints and the detected joints and $E_{\beta}$ is a quadratic penalty on the shape coefficients.

The sequence length during training is $T=16$ frames with a mini batch size of 32. Each frame gets augmented as described in Section 3.1 and then static features are computed using ResNet-50. We initialize our network with pretrained versions of the ResNet-50 network as well as the Normalizing Flows based on the ProHMR checkpoint that is publicly available. We keep the ResNet-50 features fixed, but continue training the Normalizing Flows. During training, we draw 2 samples from the distribution and not only the mode. The motion discriminator takes as input the predicted parameters $\Theta$ with the ground-truth data from AMASS and is trained to predict a single fake/real probability for each sample. We train our network as well as the motion discriminator using Adam optimizers with learning rate equal to $5e-5$. Training requires at least $7$ epochs and takes about $1$ hour on a single NVidia GTX1080Ti GPU.

We train and evaluate our network for human prediction using the following datasets:

1. 1.

   MPI-INF-3DHP [27]: This dataset contains videos of human motion, mostly indoors, at 30fps. MPI-INF-3DHP is augmented with SMPL parameters every 10 frames, derived from SPIN.

2. 2.

   Human3.6M [9, 3]: Human3.6M is a large-scale dataset captured indoors with an optical motion capture system and the training set consists of 8 different subjects performing various actions. Human3.6M was captured at 50fps and we subsample it to 25fps to match MPI-INF-3DHP. We optionally use SMPL annotations acquired with Mosh [24].

3. 3.

   3DPW [26]: 3DPW was generated using IMU sensors combined with a 2D pose detector to compute ground truth SMPL parameters. We keep the initial resolution of 3DPW at 30fps. 3DPW is the only dataset we use that consists only of outdoor videos.

We report Procrustes-Aligned Mean Per Joint Position Error (PA-MPJPE) to evaluate the accuracy of the obtained 3D poses. Mean Per Joint Position Error (MPJPE) takes also into consideration the global translation/orientation of the human and the predicted camera parameters. Mean Per Vertex Error (MPVE) is only available in 3DPW and it helps to measure the accuracy in shape prediction. We compare TAPE with state-of-the-art single-image and temporal methods. Acceleration error (ACCEL-ERR measured in $mm/s^{2}$), calculated as the difference in acceleration between the ground-truth and predicted 3D joints can be important in video methods since it evaluates how temporal coherent the prediction between consecutive frames is. Given though that existing datasets contain limited variation in acceleration, we rely more heavily in our evaluation on the rest of the error metrics.

We consider two scenarios for assessing the performance of the proposed method. In the first scenario, we train our network using MPI-INF-3DHP and Human3.6M augmented with SMPL annotations. We evaluate our method on all datasets (MPI-INF-3DHP, Human3.6M, 3DPW). Evaluating on 3DPW shows how our method generalizes to unknown data and how it estimates human shape and pose in videos with outdoor activities. In the second scenario, we, additionally, use the 3DPW [26] dataset with outdoor scenes during training, but ignore the SMPL annotations in Human3.6M. We evaluate our method again on all datasets. AMASS [25] is used for adversarial training to obtain real samples of 3D human motion in both scenarios.

In Table 2, we see the results for the second training scenario. Ignoring the SMPL annotations in Human3.6M during training drops the performance slightly for all methods. However, our method outperforms all other methods in the PA-MPJPE, MPJPE and MPVPE metrics for all datasets, which is also facilitated by the fact that 3DPW has been added in the training set. These metrics show that our method predicts better poses, body orientation and human shape.

In both scenarios (Tables 1, 2), the acceleration error of our method is comparable to MPS-NET.