MUC: Mixture of Uncalibrated Cameras for
Robust 3D Human Body Reconstruction

To exploit the benefits of extensive datasets in single-view 3D human body reconstruction, we adopt a pre-trained encoder from SMPLer-X, a generalist foundation model specialized for monocular tasks. The encoder is portrayed in the top of Figure 1.

We introduce how the single-view encoder works briefly, which is important to subsequent multi-view fusion. First, a human body image is split into a sequence of fixed-size non-overlapping patches, which are linearly projected to image tokens $T_{img}$. They are then concatenated with learnable task tokens $T_{task}$, which are transformed by ViT to $T^{\prime}_{img}$ and $T^{\prime}_{task}$. A BboxNet module predicts bounding boxes to localize face and hands in the feature map. Subsequently, MLPs are employed to regress a set of parameters $\hat{\mathcal{P}}=\{\hat{\mathbf{P}}_{body},\hat{\mathbf{P}}_{hand},\hat{\mathbf{P}}_{shape},\hat{\mathbf{P}}_{face},\hat{\mathbf{P}}_{camera}\}$. Notably, $\hat{\mathbf{P}}_{body},\hat{\mathbf{P}}_{hand}$ are joints parameter in rotation $\theta$, $\hat{\mathbf{P}}_{shape},\hat{\mathbf{P}}_{face}$ are surface parameter in PCA top-10 components to control the curvature of different parts of the mesh. This encoder is tailored to minimize the difference between the estimated parameters $\hat{\mathcal{P}}$ and the ground-truth $\mathcal{P}$, focusing particularly on areas rich in details such as the hands and face. Further, a SMPL-X layer, as described in (Pavlakos et al. 2019), can be the decoder to derive the 3D mesh of human body from $\hat{\mathcal{P}}$.

In the last section, we brief the single-view encoder. In this section, we introduce how the proposed Joint Reweighting Network (JRN) can mix the joint parameters from multiple uncalibrated cameras.

Network architecture The JRN employs two MLPs to mix joint landmark $\hat{\mathbf{P}}_{body,c}$ and hand landmark $\hat{\mathbf{P}}_{hand,c}$ based on camera position $\hat{\mathbf{P}}_{camera,c}$ for the $c$-th camera.

• •

  The first MLP_body gets task token $T^{\prime}_{task}$ and camera parameter $\hat{\mathbf{P}}_{camera,c}$, and outputs the score $s_{c}^{body}=\text{MLP}_{body}(T^{\prime}_{task,c},\hat{\mathbf{P}}_{camera,c})$. And $\hat{\mathbf{P}}_{body}$ is mixed as

  $$\hat{\mathbf{P}}_{body}=\frac{\sum_{c=1}^{N}\hat{\mathbf{P}}_{body,c}\cdot s_{c}^{body}}{\sum_{c=1}^{N}s_{c}^{body}}.$$

  (1)

• •

  The second MLP_hand gets hand token $T_{hand,c}$ and camera parameter $\hat{\mathbf{P}}_{camera,c}$. The hand token is localized and cropped from image token by BboxNet as shown in Figure 1: $s_{c}^{hand}=\text{MLP}_{hand}(T_{hand,c},\hat{\mathbf{P}}_{camera,c})$. And $\hat{\mathbf{P}}_{hand}$ is mixed as

  $$\hat{\mathbf{P}}_{hand}=\frac{\sum_{c=1}^{N}\hat{\mathbf{P}}_{hand,c}\cdot s_{c}^{hand}}{\sum_{c=1}^{N}s_{c}^{hand}}.$$

  (2)

Optimization Objective Previous studies using single-camera setups have indicated that reconstruction inaccuracies primarily stem from incorrect limb assessments, with errors increasing as the distance from the human body to the camera grows due to self-occlusion. To address this, we have integrated the joint distance distribution loss as a spatial prior to refining JRN’s learning process.

The optimization process is illustrated in Figure 2. Joint distance distribution loss is applied to 21 body joints, but for clarity we draw only 4 joint landmarks in Figure 2. Initially, 3D joints from view point $c$ are projected onto a 2D distance map $D_{c}\in\mathbb{R}^{21\times 3}$ in camera $c$’s image coordinate system. We subtract the minimal distance value from each distance map and normalize these values to a range from 0 to 1. Through this process, we obtain an approximate representation of the self-occlusion probability distribution.

The distance lists are captured as $l\in\mathbb{N}^{N\times 21}$. We collect reweighting scores for these joints from $N$ different cameras ($s_{c}^{body}\in\mathbb{R}^{21}$) into $s\in\mathbb{R}^{N\times 21}$, aiming for the Joint Reconstruction Network (JRN) to assign higher scores to joints with a lower probability of self-occlusion across views. We then use the Kullback-Leibler divergence to quantify the difference between $s$ and $l$. Below is the definition of the joint distance distribution loss function:

In the last section, we have introduced joint reweighting network, which can mix joint landmarks from multiple cameras. This section introduces the Surface Reweighting Network (SRN), which amalgamates the body surface attached to joint from multiple cameras. However, the direct fusion of 10,475 vertices of a 3D body presents a significant challenge for current hardware capabilities and poses difficulties in optimizing the reweighting network. To address this, we propose a novel surface reweighting method that concentrates on the continuous representation of these parameters. This approach facilitates a more refined understanding of the 3D human form. The overview of SRN is illustrated in Figure 3.

As shown in Figure 3 (a), initially, the mixed $\hat{\mathbf{P}}_{body}$ and $\hat{\mathbf{P}}_{hand}$ parameters, obtained from the JRN, are inputted into a SMPL-X layer alongside the $\hat{\mathbf{P}}_{shape,c}$ and $\hat{\mathbf{P}}_{face,c}$ parameters encoded from view $c$. This will generate a reconstructed human body mesh, from which vertex normals are projected onto a downsampled UV map. This projection yields the vertex normal map $VN_{shape,c}\in\mathbb{R}^{U\times V\times 3}$. Similarly, the face vertex normal map $VN_{face,c}\in\mathbb{R}^{U^{\prime}\times V^{\prime}\times 3}$ can be cropped from $VN_{shape,c}$.

We employed a conditional U-Net, which is a simplified version akin to the latent diffusion model described by Rombach et al. (Rombach et al. 2022). This U-Net is specifically informed by the predicted camera position $\hat{\mathbf{P}}_{camera,c}$. The architecture of the model is shown in Figure 3 (b). Specifically, it contains two Downsample Blocks and two Upsample Blocks with skip connections. For each block, we use $\hat{\mathbf{P}}_{camera,c}$ as K and V, and features from vertex normal map as Q to compute the cross-attention. This network predicts the weight maps $W_{shape,c}\in\mathbb{R}^{U\times V\times 3}$ and $W_{face,c}\in\mathbb{R}^{U\times V\times 3}$. These weight maps are essential for reweighting the body and facial surface features for each camera view, focusing on capturing the continuous variations in body shape and facial expressions. The final $VN_{shape}$ and $VN_{face}$ can be obtained by weighted normalization and can be supervised by the real vertex normal map generated by ground truth.

Following this, the predicted weight maps $W_{shape,c}$ and $W_{face,c}$ are converted into weight vectors $w_{shape,c}\in\mathbb{R}^{10}$ and $w_{face,c}\in\mathbb{R}^{10}$ through MLPs which bring more reasonable $\hat{\mathbf{P}}_{shape,c}$ and $\hat{\mathbf{P}}_{face,c}$ reweighting. The final $\hat{\mathbf{P}}_{shape}$ and $\hat{\mathbf{P}}_{face}$ can be obtained by weighted normalization as well.

We train our model end-to-end by minimizing a several loss functions: $L_{smplx}$, $L_{joint2D}$, $L_{JRN}$, $L_{surface}$. Each component of the loss function serves a specific purpose in the learning process. The $L_{smplx}$ loss is computed as the L1 distance between the ground truth and predicted SMPL-X parameters, providing explicit supervision for whole body joints, body shape, facial expressions, and camera position. The $L_{joint2D}$ loss is a regression loss for the projected 2D landmarks of the whole body, ensuring the accuracy of landmark localization in two-dimensional space. The $L_{JRN}$ loss, as defined in Eq. 3, is used for training JRN to learn the importance of different body joints across various camera views. Finally, the $L_{surface}$ loss is calculated as the L1 distance between the ground truth and the predicted fused surface feature maps, offering direct supervision for SRN.

All model are trained using single A100 with pytorch. Adam optimizer with an initial learning rate of $3\times 10^{-5}$ for 20 epochs. The initialization weight of encoder from SMPLer-X (Cai et al. 2023).

Two datasets are used to train and evaluate our method.

• •

  Human3.6M (Ionescu et al. 2013) is a large-scale multi-view dataset with ground-truth 3D human pose annotation. We follow the standard training/testing split: using subjects S1, S5, S6, S7 and S8 for training, and subjects S9 and S11 for testing. The SMPL/SMPL-X pseudo-GTs are obtained from NeuralAnnot (Moon, Choi, and Lee 2022). This dataset is widely used in multiple human body reconstruction tasks and helps us to compare with other state-of-the-art methods.

• •

  RICH (Huang et al. 2022) is an in-the-wild and indoor multi-view dataset annotated with ground-truth 2D keypoints and 3D body mesh. We adopt the intrinsic training/testing split in the dataset. The SMPL-X pseudo-GTs are obtained from SMPLify-X. This dataset contains human body joints, shape, hand joints and facial expression annotations, which can be used to prove the effectiveness of our multi-view fusion method. The distribution of camera numbers is shown in Table 1. For each sample in the training and validation sets, we randomly split it into two samples, each with a camera count equal to 4.

Both of these datasets are based on video capture, where the positions of individuals relative to the cameras are constantly changing, hence it can be considered suitable for assessing the generalization ability of our method to camera positions.

For 3D whole-body mesh reconstruction, we utilize the mean per-vertex position error (MPVPE) and mean per-joint position error (MPJPE) as our primary metrics. In addition, we apply procrustes analysis (PA) to the recovered mesh, and report PA-MPVPE and PA-MPJPE after rigid alignment. PA-MPJPE primarily evaluates the pose estimation accuracy of the human body model, whereas PA-MPVPE evaluates the accuracy of the reconstructed 3D model. PA-MPJPE and PA-MPVPE are reported in millimeter (mm).

In this subsection, we present a comprehensive comparison of our proposed calibration-free multi-view fusion model with current SOTA methods in the domain of 3D human body reconstruction. We employ both multi-view and single-view reconstruction methods on the Human3.6M dataset to gauge the performance of our method against an array of established techniques. The results are presented in Table 2.

Our comparison includes 3D human body reconstruction methods from 2016 to 2023. Most of them are in the category of single view, while a few are in the multi-view category. Meanwhile, many methods adopt the SMPL model, which can only represent human body pose and body shape. A more sophisticated SMPL-X, as adopted by our method, can reconstruct human body pose, hand pose, body shape and facial expression jointly. Specifically, we can treat SMPLer-X (Cai et al. 2023) as SOTA method for single-view fusion method and PaFF (Jia et al. 2023) as SOTA method for multi-view fusion method. To eliminate differences between using different templates, we not only evaluated using the SMPL-X template but also utilized the conversion tool provided by SMPL-X (Pavlakos et al. 2019) to transform our prediction results into the SMPL template for evaluation.

From Table 2, we can observe the following.

• •

  Our Method’s Edge: Our approach outperforms other methods in multi-view reconstruction, achieving a PA-MPJPE of 27.1mm, which surpasses the previous best multi-view method by 1.1mm. It also performs better than models with larger parameter counts, such as SMPLer-X-large. Additionally, it maintains a leading position in both PA-MPJPE and PA-MPVPE metrics in single-view scenarios.

• •

  Model Bias: Under different model types (SMPL and SMPL-X), our approach demonstrates significant performance improvements in multi-view reconstruction compared to single view. Moreover, it achieves a notable enhancement in joints accuracy compared to previous methods. Since our model utilizes the SMPL-X model for both training and prediction, and the vertex count of the SMPL-X model is roughly twice that of the SMPL model, forcibly converting results to the SMPL model introduces some errors in the overall vertex alignment. Therefore, we only present results in terms of PA-MPJPE.

From Table 3, we can observe that our method outperforms SMPLer-X on the RICH dataset, which features more complex scenes and diverse camera settings. Our method’s visualization results are depicted in Figure 4. Compared to single-view models, our approach better compensates for the information loss caused by different viewpoints, resulting in superior reconstruction outcomes.

To evaluate the effectiveness of our calibration-free multi-view fusion method, we first present a frame-based analysis for a video of three cameras. As illustrated in Figure 5, our multi-view method effectively compensates for the target’s movement in the wild, maintaining a stable PA-MPVPE fluctuation.

Our method is scalable to an arbitrary number of cameras out of the box, so we conduct quantitative validation to assess the impact of adding more cameras for each case. The goal is to determine how the number of cameras influences the reconstruction precision of different human body parts.

The results, shown in Tables 4, underscore our method’s effectiveness in utilizing additional camera angles. For body reconstruction, there is a marked decrease in PA-MPJPE when increasing the camera count from one to four. Similar enhancements are observed in the hand and face reconstructions. Notably, the transition from one to two cameras yields the most significant improvement, highlighting the value of multi-view geometry. While the incremental gains diminish with each additional camera, the consistent improvements across all metrics reinforce our approach’s effectiveness in leveraging multiple uncalibrated views for 3D human body reconstruction in various settings. The same conclusions are also demonstrated on the Human3.6M dataset, with results presented in Table 5.

Our qualitative evaluation focuses on the influence of the JRN and SRN on the 3D reconstruction of human body poses and shape. Figure 6 visually demonstrates the effect of reweighting across different camera views.

In Figure 6, the size of the red circles superimposed on the model’s joints indicates the importance level that JRN assigns to each joint. Larger circles indicate a higher score, suggesting that these joints exert a more significant influence during the multi-view joint reweighting process. It is evident that with the aid of the joint distance distribution loss, the JRN assigns lower weights to joints with a higher probability of self-occlusion. This strategy effectively harnesses the more accurate parts of predictions from each single viewpoint for fusion. Similarly, predictions using the SRN feature more precise estimates of body shape.

We further perform two quantitative ablation studies experiments to objectively measure the effectiveness of the JRN and SRN: one focusing on whole body reconstruction and the other on detailed hand and face reconstruction.

Whole Body Reconstruction We utilized the Human3.6M and RICH datasets to evaluate the performance improvements. The results are compiled in Table 6. The inclusion of JRN alone, and in combination with SRN, shows a consistent decrease in PA-MPJPE and PA-MPVPE across both datasets, highlighting the benefits of the reweighting mechanisms.

Hand and Face Reconstruction The results on RICH are presented in Table 7. The quantitative analysis clearly demonstrates that the combined use of JRN and SRN yields the best performance, as indicated by the bold values for PA-MPJPE and PA-MPVPE, underscoring their importance in enhancing the precision of 3D human pose reconstruction.