Deep3DPose: Realtime Reconstruction of Arbitrarily Posed Human Bodies from Single RGB Images

Fig. 3 illustrates the structure of our multi-task network. It is inspired by architectures like [42, 43, 44], which refine the predictions recurrently. An image is first encoded by a convolutional network, generating a set of image features $F$, which are then passed over to the first estimation for each individual task at stage 2. We get coarse predictions for joint confidence maps ${H^{1}}$, IUV maps (body partition ${S^{1}}$ and $uv$ coordinates ${U^{1}}$) in stage 1. In the successive stages, the network takes as input the image feature $F$, the results of previous stages of the same type. We formulates the procedure as follows:

where $2\leq t\leq 5$ is the stage index, and $\delta(\cdot)$ is the mapping for Branch *, as defined as four Conv3$\times$3-BN-ReLU blocks and one Conv1$\times$1 task-specified regressor. $Cat(\cdot)$ is the concatenation operation. In stage 6, the joint confidence map and IUV map from the previous stage is concatenated and treated as input to predict not only the joint and IUV, but also two additional terms: the mask $M$ and the part orientation maps $L_{3d}$.

Loss Term. To guide the training of the multi-task network, we apply losses for predictions at each stage, specifically $L_{2}$ losses for the confidence maps $H$, POFs and $UV$ maps. Note for the $UV$ map, we only take into account a body part if the pixel is located inside it. When training part partition, a standard multi-class cross-entropy loss is used. Note that due to the difference of human part areas, we balance the supervision for part segmentation classification by the weight ${w_{c}}$ for each human part $c$, so that the network would not over-fit body parts of large area. The balance weight ${w_{c}}$ is inversely proportional to the part area, as in [45]. Our IUV (body partition and $UV$ maps) ground truth for hands is inaccurate, because we fit a statistic model into a skeleton without finger joints, so we just ignore the IUV loss for hands. The segmentation mask is trained by binary cross-entropy loss.

Implementation. The training of our multi-task network consists of three phases. (1) First, we pre-train our network for the 2D joint detection task with in-the-wild image dataset for stage $1\sim 5$, ignoring other tasks, which gives better generalization performance. Our 2D joint detection task is trained with an initial learning rate of $10^{-3}$ and is reduced every 200,000 iters by a factor $\gamma=0.333$, as [46] does. (2) Second, we combine our newly collected dataset with 2D joint dataset, and apply a mix-training strategy for other tasks while freezing the weights of feature extractor and 2D joint detector for 100,000 iters by a learning rate of $5\times 10^{-4}$. Note that our newly collected data has all desired ground truth for every task. Fig. 4 shows a few images in it, including the original captures and the augmented ones through background replacement. Data augmentation with background replacement greatly increases the generalization of in-the-wild images. (3) Finally, we unfreeze the weights of well-trained 2D task and feature extractors, but apply a smaller learning rate (multiplied by 0.1) for these weights. Our full-task training takes 1,000,000 iterations with a learning rate of $5\times 10^{-4}$ reduced every 200,000 iters by a factor $\gamma=0.333$. We employ a rotation augmentation ($\pm 30^{\circ}$), a scaling augmentation (0.75-1.25) and left-right flipping (only for in-the-wild dataset) for training. We use the Caffe framework [47] for network training, and use the Adadelta solver [48]. The performance of each task is strongly dependent on the relative weighting between the loss of each task [49]. And in our experiment, we set loss weights as follows: $w=0.5$ for $UV$, $w=0.05$ for body partition, $w=0.5$ for heatmap, $w=1.0$ for foreground mask and $w=1.0$ for POFs. To balance accuracy and efficiency, we use MobileNet-V2 [50] as our feature extractor. Note that we removed the downsampling operations in last two blocks (by replacing stride=2 in downsampling convolution with stride=1), and maintained the size of the final feature map to be $1/8$ of the input image, which is 224-by-224.

Similar to SMPL, we approximate the human full-body geometry with a skinned mesh that is driven by an articulated skeleton model using Linear Blend Skinning (LBS). Our skeleton has 45 degree of freedoms (DOFs), 6 of which are for global position and orientation, and 39 are for joint angles (note that a joint may be of 1, 2 or 3 DOFs). We built a female mesh model of 28,109 vertices (or 56,142 triangular faces), carrying more geometric details than the SMPL model of $6,890$ vertices. The female model is elaborately rigged and parameterized such that the shape can be easily controlled. ¹¹1Yet currently we do not have a decently parameterized male model on par with the female model. For scenes with a male subject, we either use motion-retargeting to drive a male model mesh but without body dimension adjustment (e.g. Fig. 17(b), or just blindly use the female model (e.g. one case in the accompanying video).

Following a relatively mature process we build a parametric human full-body geometry model (Fig. 5). The model is controllable in two aspects: (1) skeleton scales, which encode coarse-level variation like the overall and the per-bone scales, (2) mesh vertex offsets, which encode fine-level shape variation, such as thickness of a limb. The parametric human full-body model can be represented as:

where $\textbf{H}_{i}(\cdot)$ is the coordinate of the $i$-th vertex of the mesh model, $\hat{\textbf{v}}_{ij}^{\prime}$ is the result after applying the scaling and offsetting to $\hat{\textbf{v}}_{ij}$ (which is the $i$-th vertex represented in the local coordinate frame of the $j$-th bone), $\textbf{Q}(\bm{\beta})\oplus$ describes the vertex offsetting, $\textbf{S}(\bm{\alpha})\otimes$ describes the bone scaling, the shape parameters ${\bm{\alpha}}\in R^{8}$ and ${\bm{\beta}}\in R^{26}$ provide a low-dimensional representation of human bone scale variances and vertex offset variances across individuals respectively, ${\bm{\theta}}$ is the pose for deformation, $T_{j}({\bm{\theta}})$ is the transformation of the $j$-th bone for pose ${\bm{\theta}}$, $w_{ij}$ is a sparse weight map for deformation, $n$ is the number of bones.

Given the subject-specific full-body mesh model (obtained in § 5.3) and the network observations (2D pose from 2D joints probability maps, 3D part orientations from the 3D limb orientation fields, the human mask, the IUV map for frame image $I_{i}$), our goal is to estimate a human pose ${\bm{\theta}}$ which best matches the network observations. We estimate the human pose ${\bm{\theta}}$ by minimizing the following objective function:

where $E_{data}$ is the data term penalizing the registration error between the synthesized human model and the observation. $E_{prior}$ is the prior term that penalizes invalid human pose configuration, and $E_{temporal}$ is the pose smoothness term that penalizes the jerkiness in the motion, which is only used for video application. While searching for the solution in an iterative manner, it is possible (and recommended) to use the pose ${\bm{\theta}}_{prev}$ from the previous frame as the initial guess. We will describe each term in detail in the following subsections.

Now we describe how to reconstruct a full-body mesh model for a subject using the network observations. Note in the case that the input is a video stream, the shape reconstruction is done only once, for the first frame.

Our laboratory setup is shown in Fig. 6, where data is captured by four digital video cameras. The designated laboratory area is about $4m\times 4m$, and within it we obtain effective capture court of approximately $2.5m\times 2.5m$, where each subject is fully visible to all cameras. Four cameras are placed at four corners of the court. The floor and the wall are mantled with green curtains, making it easy for automatic segmentation of the foreground body. The total number of actors screened are over $300$, covering a broad range of ages, body shapes and pose extensions. Our dataset has much more subjects than in any of existing 3D datasets. Each person is asked to do certain daily life motions as well as sport motions, for about three minutes. To eliminate redundancy between consecutive video frames, frame images are further filtered, and only 500 to 1,000 images are sampled for each actor. The sampling is achieved by clustering frame images according to the similarity of human poses.

To build a highly accurate parametric body model for each actor, we take some body measurements while an actor is standing still in A-pose. A set of measurements $\mathbf{M}\in R^{44}$ (including but not limited to lengths of limbs, shoulder and back, girths of chest, wrist, hip and stomach, etc.), are tailoring measured with a ruler. Our parametric model has $8+26=34$ shape parameters, more sophisticated than the SMPL model, which has only 10 shape parameters. With the measurements, parameters ${\bm{\alpha}}$ and ${\bm{\beta}}$ are computed by minimization an energy function

where $E_{geoDist}=\sum_{i=1}^{44}||f_{i}({\bm{\alpha}},{\bm{\beta}})-M_{i}||_{2}$ assesses the error of geodesic distance on the parameterized human body, $E_{shape\_prior}$ determines the shape prior error, with respect to mixed Gaussian distribution. The above energy is minimized with the Particle Swarm Optimization method [51, 52].

Parameter values. We set $w_{1}=8.0$ for $E_{geoDist}$ and $w_{2}=0.8$ for $E_{shape\_prior}$ in Eq. 20.

With four multi-view images for each frame, a 3D pose is reconstructed, according to the following steps.

1. 1.

   Estimate 18 joints on each image from each camera, with a 2D joint estimation method [43].

2. 2.

   Validate the consistency of the estimated 2D joints from multi-views (see the following explanation).

3. 3.

   Segment foreground body from background automatically (green curtains setting).

4. 4.

   Solve the human motion by extending Eq. 5 into multi-views, while dropping off the IUV constraints and the 3D constraints from Eq. 6 (As both constraints are not available when constructing our datasets).

5. 5.

   Recover the 3D body mesh with the shape from measurements and the pose from multi-view images, shown in Fig. 7.

In Step 2, it is very common that in a single-view image some joints could be invisible due to occlusion. Therefore we propose a cross validation scheme based on multiple views. For a certain joint, we choose any pair of cameras and use the two 2D estimations to build the 3D joint position. The 3D joint is then re-projected with respect to the view-ports of the other two cameras, creating two projected joints. Each projected joint is compared against the estimated 2D joint under the same view-port, and their Eucleadian distance is calculated. Only if the distances in two view-ports are both below a certain threshold (18 pixels in our experiment), this set of four 2D estimations for this joint is considered to be consistent and reliable. If four view-ports fail to reach consistency, we check if any three of them do. If that happens, the 2D joint estimation in the three view-ports are treated as reliable. If such three view-ports can not be found, we have to ask for help from the previous frame. The 3D joint from the previous frame is compared with the reconstructed 3D joint from any pair of cameras, and the distances are calculated. The minimal distance designates the pair of cameras and their estimations are treated as reliable.

The image-to-surface correspondence (IUV map) proposed in Densepose[1] for human body is an essential mechanism for mapping a 2D image into a 3D geometry. We improved this idea with a more solid implementation, and it works well for persons with loose clothes. In Densepose certain pixels are manually sampled on each human part, and their corresponding points on the meshed surface are manually marked as well. Annotators are asked to determine the body silhouette if it is covered by clothes, and mark around 100 points for each human. In this situation the burden is heavy and errors prone to happen, especially when a human instance wears a large/loose skirt.

We adopt a quite different scheme for computing the part label and the $(u,v)$ coordinate for each pixel, requiring much less human intervention. The reconstructed 3D mesh by measurements is re-projected according to the view-ports of the cameras, creating four images. As each mesh vertex has a $(u,v)$ coordinate and a part label, it is trivial to compute $(u,v)$ and label for each pixel of these images. Due to the topological complexity of human meshes, we follow Densepose and segment a human mesh into 24 parts, and define a $uv$-field on each part, as shown in Fig. 8.

Our system reconstructs 3D human poses and full-body geometry models from single images in realtime, and the reconstructed 3D poses is retargeted to animate a character in realtime (See Fig. 10). Our technology has potentials in applications such as game character control, embodied VR, sport motion analysis and reconstruction of community video.

We also test our method on various in-the-wild videos, showing its robustness to different actors with different body shapes and clothes, even under significant occlusion, lighting and background changes. Fig. 11 shows some excerpted frames. Besides, we also present other-view overlay results in Fig. 9 by using our multi-view test dataset, which reflects the accuracy of human reconstruction achieved by our method. Please refer to the accompanying video for more vivid results.

We evaluate the performance of our method on two popular test benchmarks: 3DPW [53] (outdoor scenes) and Human3.6M [54] (indoor scenes). Following the standard protocol for 3D pose estimation [22] in Human3.6M, we use 5 subjects (S1, S5, S6, S7 and S8) for training, and the rest 2 subjects (S9 and S11) for testing. As for Human3.6M, we get ground truth parameters for training images using MoSh [56] from the raw 3D Mocap markers like [30]. As for 3DPW, we only use its testing dataset for evaluation as previous works [2]. Note that Human3.6M and 3DPW have different skeleton configurations from ours, we therefore learn a linear regressor for a mapping which maps our mesh vertices to 17 joints defined in Human3.6M, as did in [30]. For evaluation, we adopt averaged skeleton dimensions computed from the training set to rescale our reconstruction human, as did in [22].

The results are shown in Table II. Our single image based method is even competitive to the video-based VIBE [5] on Human3.6M and 3DPW. The results also show that our newly collected data improves the performance further, especially on 3DPW (wild), with improved wild generalization.

To show the efficiency of our method, we compare against two state-of-the-art regression-based methods, one is single frame method, SPIN [2], the other is video-based method, VIBE [5]. Fig. 12 shows the result of a side-by-side comparison. It is obvious that our method achieves better image-model alignments than SPIN and VIBE. Regression-based methods usually achieves global image-model alignments quickly, but at the cost of low quality. This type of nonlinear prediction is also uneasy to control, due to the mutual effect between human pose and shape. We decouple them, and since 2D joint positions and dense image-to-surface correspondence (IUV map) offer better image-model alignments, and 3D part orientation helps to avoid depth ambiguity, our method produces more accurate body reconstruction than SPIN and VIBE.

Furthermore, SPIN does not guarantee temporal stability because it regresses different body shapes from different images in a sequence, while VIBE fixes it, but it is not in realtime. Please refer to Fig. 12 and the accompanying video for the comparison results.

Our method is also compared against a recent realtime system, VNect [3], though it captures poses only. We compare not only the raw network outputs, but also the final fitting results (see Fig. 13). It is obvious that our method achieves better pose reconstruction than VNect. Two reasons account for this. Firstly, our multi-task network produces more accurate 3D joint positions, as shown in Fig. 13(c). Secondly, VNect initializes bone lengths for forearm and upper arm to an improper ratio, as seen in Fig. 13(b), while our initialization matches with person in image very well. VNect initializes skeleton by averaging 3D joint positions from the CNN output at the beginning, which is thus very sensitive to single CNN outputs (3D joint positions). We instead utilize more image features, including 2D joints, 3D part orientation and IUV maps, to reconstruct human bodies more accurately and robustly. Please refer to the accompanying video for more comparison results.

SMPLify [4] is a somewhat hybrid method: it fits the SMPL model by optimizing regressed 2D joints without user intervention. Fig. 14 shows the results given by SMPLify and our method. At least three issues about SMPLify can be interpreted from the figure. First, SMPLify is more vulnerable to depth ambiguity than our method, as can be seen from images in the first row. This is because SMPLify relies on 2D joint reprojection alone for model fitting, and this is insufficient to robustly establish a 3D pose. Second, there is no constraints for foot orientation in SMPLify. Last, it is easy for SMPLify to fall into a local minima, as shown in second row. Our method has hardly convergence problem. For a single image, the initial shape from the average of our human model. We use it and predicted 2D joint positions and 3D bone directions to roughly estimate the root position and joint angles as initial pose. The optimization problem is over-constrained.

We have designed a realtime multi-task network that regresses more features than any other deep learning based methods. We believe that more features offer more visual cues that facilitate pose and shape reconstruction. In this section we justify this belief with experiments, evaluating the role of each regressed features in both the CNN regression and the body reconstruction process.

Quantitative analysis for multi-task network To evaluate the importance of our multi-task design, we modify the network in Fig. 3 into 4 structures with different configurations: a) 2D joint detection only, b) 2D joint detection + IUV branch, c) 2D joint detection + POFs, d) 2D joint detection + IUV + POFs. All these networks are trained with the same training dataset, and tested on our validation dataset, which contains 11 different subjects not in the training dataset. For metrics of 2D joint positions, we report PCKh@0.5 [7] (the higher the better). For 3D part orientation, we scale the predicted 3D part orientation by the ground-truth limb length to obtain the 3D joint positions, then align the root joint position and compute the MPJPE (the lower the better). The results are reported in Table III, which shows the power of mutual promotion of multi-task. We found that IUV information improves the accuracy of 3D POF orientation. The reason is that IUV maps provide the part occlusion relationship which conveys some 3D information. IUV maps are usually more abstract and more powerful than 2D landmarks in representing a human, and it is used by [59] as input.

Quantitative analysis for optimization Table IV shows the quantitative performance, which reveals the importance of each energy term. We compare results under 5 different energy term settings: a) 2D position only; b) 2D position + mask; c) 2D position + mask + 3D part orientation; d) 2D position + mask + 3D part orientation + IUV; e) 2D position + mask + 3D part orientation + IUV + temporal. We report MPJPE on Human3.6M and 3DPW. The results shows that every energy term in our optimization is beneficial for human pose reconstruction. The result with temporal term shows higher errors, because it ensures temporal smooth rather than the consistency with the network output cues.

The mask term in optimization. We evaluate the importance of the foreground segmentation mask by comparing the reconstructed bodies with and without this term. Fig. 15 clearly shows the role of the mask when predicted 2D joints and 3D orientation are inaccurate, especially when some joints are missing.

The 3D part Orientation term in optimization. Fig. 16 shows an example with and without the 3D orientation term. The use of the 3D orientation term significantly reduces the reconstruction ambiguity of 3D poses.

The IUV term in optimization. An IUV map plays important roles in both 3D body geometry reconstruction and pose estimation. We evaluate the importance of IUV by dropping off this term in shape reconstruction and pose estimation, respectively. Fig. 17(a) shows a side-by-side comparison while reconstructing an over-weighted lady. Using IUV term gives more accurate body model, because IUV terms impose detailed geometry model constraints from dense correspondences.

For pose reconstruction, Fig. 17(b) shows that reconstructed examples with and without IUV term. It is obvious that IUV term helps recover more accurate result, especially for body orientation.

With no exceptions, our method suffers from several limitations. First, we observed failure cases when a significant part of the target person is either occluded by other objects or out of image boundary. Occlusion is the biggest issue and it imposes more challenge for RGB camera than depth camera based methods. Second, our method also fails for complicated or uncommon poses, particularly those in sports videos, such as gymnastics and skydiving. The main reason is that such data is not adequate in training dataset. Third, our system does not have specific hand pose detector (as did in [60]) and each hand is associated with only one joint, therefore the reconstructed hands are sometimes incorrectly oriented. Finally, our CNN does not handle multiple bodies at this moment, but can easily extended to support this. Solving the above mentioned problems points to interesting future directions.