CameraPose: Weakly-Supervised Monocular 3D Human Pose Estimation
by Leveraging In-the-wild 2D Annotations

Instead of refining on the original noisy 2D keypoints, we utilize the confidence score combined with the 2D $(x,y)$ coordinates as input to the refinement network. We first normalize the coordinates of keypoints to $(-1,1)$ with respect to the input image height and width. We also normalized the confidence scores to a comparable scale by Eq. 3:

The neural network architecture of our Refinement Network is a standard residual block consisting of fully connected layers with a hidden dimension of $512$. The refinement loss $\mathcal{L}_{ref}$ is formulated as:

In this paper, the 2D-3D pose pairs are calculated in the camera coordinate system, so the camera parameters can be simplified to be the intrinsic matrix $\textbf{M}^{int}$ in Eq. 5 and a 3D offset $\textbf{t}_{3D}$. For intrinsic matrix $M_{int}$ we are essentially predicting a $4$-dimensional vector, namely $f_{x},f_{y},c_{x},c_{y}$, the focal lengths $f_{x}$, $f_{y}$, and principal center offsets $c_{x}$, $c_{y}$ along the $x$ and $y$ direction respectively.

The camera parameter branch consists of 2 residual blocks with a hidden dimension of $512$, which can be plugged in to any standard 3D pose estimators. There are three losses that can be involved depending on the annotations. The 2D reprojection loss $\mathcal{L}_{2D}$ as shown in Eq. 7 calculates the Euclidean distance between the reprojected 2D poses and ground truth. The mean-square error (MSE) is used in loss calculation for both the camera parameter loss and 3D inference loss as shown in Eqs. 8 and  9 respectively.

Since CameraPose can work on 2D-3D pose pairs as well as 2D alone pose estimations, the loss design can be different according to the availability of labels. In the case of all annotations are available during the training stage, the camera loss can be calculated as:

In the case of 2D annotation alone training step, the loss calculation will be from 2D reprojection error:

Similar to the framework in  [11], we utilized both generator and discriminator to further improve the diversity in training poses. As shown in Figure 5, the generator is plugged in to the 2D pose generation stage, and the discriminator is applied on both the 2D and 3D pose inference.

The generator is actually formed by 3 simple multi-layer perceptions that generated different parameters for 3 different augmentation operations respectively: (1) changing the bone angle $\textbf{X}_{ba}$, (2) changing the bone length $\textbf{X}_{bl}$ and (3) changing the camera view and position of the input 3D pose $\mathbf{R}\cdot\textbf{X}_{bl}+\mathbf{t}$.

The discriminator part of the framework can be divided into 2 portions, the $\mathcal{D}_{2D}$ and $\mathcal{D}_{3D}$ as we want to make sure that both the augmented $\textbf{X}^{*}$ and $x^{*}$ formed plausible human poses in both image coordinate and camera coordinate. But in our work we not only want to ensure the goodness of the augmented poses from the generator, we also want to utilized the discriminator to regulated our reprojected 2D poses for those 2D annotations only dataset cases. The discriminators also adapt the part-aware Kinematic Chain Space (KCS) proposed in [11], they are fully connected networks with a structure similar to the pose regression network using the KCS representation [36] of 2D or 3D poses as input. Here we use the LS-GAN loss:

The overall framework is made differentiable and can be trained in the end-to-end fashion. We update different modules alternatively by minimizing loss in Eq. 4, Eq. 10, Eq.  11 as well as generators and discriminators with some preassigned hyper-parameters $\lambda$.

Then we interactively train the entire model and update the weights of 3D lifting network using the losses:

For the 2D-3D paired annotations, we utilize the most popular datasets 3D HPE dataset Human3.6M [15], 3DHP [24] and 3DPW [34]. Both Human3.6M and 3DHP were collected indoor in some laboratory environment through the MoCap (motion capture) system [26] with multiple calibrated cameras. The 3DPW is a more challenging dataset collected in outdoor environment using IMU (inertial measurement unit) sensors with mobile phone lens.

For 2D annotations only datasets, we used MPII [1] which contains a variety of in-the-wild everyday human activities. Another popular 2D dataset MS-COCO [21] is also used for qualitative analysis purposes. Although the 2D annotation dataset such as MPII is much less than Human3.6M or 3DHP in terms of sample size, these 2D annotation datasets contain more challenging human poses with different activities. Note that both Human3.6M and 3DHP are video based datasets, so that the total number of images is much larger than MPII and MSCOCO. We summarized the datasets utilized in our experiments in Table 3.

Different datasets have distinctive annotations on joints, which make the model training difficult. In this paper, we used the Human3.6M format as standard one, and interpreted missing joints by labeling nearby joints for other datasets. All the joints that are not included in Human3.6M format will be discarded.

Many existing 3D HPE algorithms use the groundtruth as model input for evaluation. However, groundtruth is not available in real use cases. To evaluate the model performance on the real-world applications, we also used existing 2D detector HRNet to extract the 2D keypoints as model input and rerun results on different datasets.

Due to the various labeling schemes or joint formats difference, we preprocess other schemes into the Human3.6M format by simple interpolation of some related joints and removal of the unused joints. For example, there is no pelvis; we simply create such joint by computing the mid-point of the left and right hip of any given label. Even though such interpolations are not always perfect due to the nature of each dataset, this preprocessing procedure allows us to have a better idea and comparison on cross-dataset scenarios.

CameraPose network is trained on 2 datasets: Human3.6M (2D + 3D) and MPII (2D). For the former, we followed most 3D human pose estimation training protocols using the subjects S1, S5, S6, S7, S8 from Human3.6M as our 2D-3D training data, and subjects S9, S11 for evaluation purposes. For the latter, we filtered and selected around 10k training samples by checking the joints annotations. For evaluation, MPI-INF-3DHP and 3DPW were used to get quantitative results in terms of MPJPE (mean-per-joint-position-error) and PA-MPJPE (aligned with ground-truths by rigid transformation).

The model training can be divided into 3 steps. The refinement network was trained as the first step for 100 epochs with learning rate being 0.0001 and weight decay at epochs 30, 60 and 90, respectively. Next step, the 3D lifting network along with the pose generator and discriminator was trained using Human3.6M dataset for 10 epochs with a learning rate of 0.0001. This step is for warm-up and GAN tuning which can make the following model training more stable. Finally, the model was trained in an end-to-end fashion using both 2D-3D pairs annotations as well as 2D alone annotations. In each iteration, we first updated the weights of generator and discriminator to make the generators more stable. Then the 3D lifting network was updated based on the augmented poses plus the 2D-3D annotated dataset. After that, 2D only annotations were utilized to tune the camera parameter branch. The model was trained for 75 epochs with a learning rate of 0.0005 and weight decay at 30, 60, respectively. And the weighting for loss we choose $\lambda_{cam}=0.01$, $\lambda_{2D,\boldsymbol{\phi}}=0.5$, $\lambda_{2D,\boldsymbol{\phi}^{\prime}}=0.2$, and $\lambda_{3D}=1.0$.

As shown in Table 4, our method outperforms the SOTA on the most challenging dataset 3DPW by a noticeable margin ($3$mm and $13$mm). It also has significantly higher accuracy than other weakly-supervised methods like [29] and [37]. Our model also achieves the highest accuracy on the Human3.6M dataset. Experimental results clearly show the strong generalization capability of our proposed method. Adding the camera parameter branch can help the model to learn from in-the-wild datasets with 2D annotations, which is very effective for hard examples.

The results on the 3DHP are slightly lower than SOTA methods, and we claim it is due to the fact that the 2D annotations we added from MPII are more helpful for challenging cases such as the 3DPW dataset. The best accuracy on the 3DHP dataset can be 75.54 MPJPE using our model, which outperforms the current SOTA if we select a specific model for the 3DHP dataset.

We used HRNet as 2D detectors to extract the 2D keypoints on all the training and evaluation datasets. We added the refinement network to both the SOTA method [11] and our proposed model. By adding the refinement network, both PoseAug and our model have improved accuracy on both Human3.6M and 3DHP. In addition, our model outperforms the SOTA on both testing datasets. Therefore, both our proposed camera parameter network and the refinement network are useful for 3D HPE.