Pixel2ISDF: Implicit Signed Distance Fields based Human Body Model from Multi-view and Multi-pose Images

Our feature extraction networks utilizes multi-view and multi-pose images $\{{I_{k}}\}_{k=1}^{K}$ as input and outputs the geometric feature maps that help to predict the 3D geometry in the canonical space. Specifically, we first adopt the normal network $\mathcal{F}_{normal}$ to extract the normal maps and then utilize the geometry network $\mathcal{F}_{geo}$ to generate geometric feature maps $\{{F_{k}}\}_{k=1}^{K}$ that will be further utilized to extract the pixel-aligned features for the vertices on the posed mesh.

In particular, we adopt the pretrained image-to-image translation network from PIFuHD [21] as our normal network, and use the pretrained ResUNet34 [8] backbone as our geometry network.

After obtaining the geometric feature maps $\{{F_{k}}\}_{k=1}^{K}$, we extract the pixel-aligned features for each vertex $\mathbf{v}_{k}^{i}$ on the posed mesh $M(\theta_{k},\beta_{k})$. For each vertex $\mathbf{v}_{k}^{i}$, we first project it to the image space by utilizing the weak-perspective projection $\Phi$ according to camera parameters scale $s_{k}$ and translation $t_{k}$, then adopt the bilinear interpolation operation to extract the pixel-aligned features $\bm{f}_{k}^{i}$.

where $\Pi$ is the orthogonal projection, $\mathbf{x}$ is the point in 3D space and $\hat{\mathbf{x}}$ is the projected points in 2D image space.

To integrate the feature of the $i$-th vertex in the canonical space from multiple views/poses, we use a fusion network that takes $\{\bm{f}_{k}^{i}\}_{k=1}^{K}$ as the input and outputs the integrated feature $\bm{l}^{i}$, which is illustrated in Figure 1. Specifically, the mean $\bm{\mu}^{i}$ and variance $\bm{\sigma}^{i}$ of features $\{\bm{f}_{k}^{i}\}_{k=1}^{K}$ is calculated and then concatenated with $\bm{f}_{k}^{i}$ to serve as the input of a MLP. The MLP predicts the new feature vector and weight $\{w_{k}^{i}\}_{k=1}^{K}$ for each feature, which generates a weighted sum of features from multiple inputs. The weighted feature is finally forwarded to an MLP for feature integration, ${\bm{l}^{i}}$ which serves as the structured latent code for the vertex $\mathbf{v}^{i}$.

Different from the latent code in NeuralBody [19] which is initialized randomly for optimizing specific humans, our latent code is the feature vector learned by a network with the normal map as the input, which can generalize to humans unseen from training ones.

The above-mentioned latent codes are generated based on the posed mesh. The posed mesh and the canonical mesh share the same latent codes because of their topology correspondence which forms the set of latent codes by $\mathcal{Q}=\{\bm{l}^{1},\bm{l}^{2},\cdots,\bm{l}^{N_{V}}\},\bm{l}^{i}\in\mathbb{R}^{d}$. Here $N_{V}$ represents the number of vertices.

The learned latent codes are anchored to a human body model (SMPLX [15]) in the canonical space. SMPLX is parameterized by shape and pose parameters with $N_{V}=10,475$ vertices and $N_{J}=54$ joints. The locations of the latent codes $\mathcal{Q}=\{\bm{l}^{1},\bm{l}^{2},\cdots,\bm{l}^{N_{V}}\}$ are transformed for learning the implicit representation by forwarding the latent codes into a neural network

To query the latent code at continuous 3D locations, trilinear interpolation is adopted for each point. However, the latent codes are relatively sparse in the 3D space, and directly calculating the latent codes using trilinear interpolation will generate zero vectors for most points. To overcome this challenge, we use a SparseConvNet [19] to form a latent feature volume $\mathcal{V}\in\mathbb{R}^{H\times H\times H\times d}$ which diffuses the codes defined on the mesh surface to the nearby 3D space.

Specifically, to obtain the latent code for each 3D point, trilinear interpolation is employed to query the code at continuous 3D locations.

Here the latent code will be forwarded into a neural network $\varphi$ to predict the SDF $\mathcal{F}_{sdf}(\mathbf{x})$ for 3D point $\mathbf{x}$.

During training, we sample $N_{S}$ spatial points $\mathcal{X}$ surrounding the ground truth canonical mesh. To train the implicit SDF, we deploy a mixed-sampling strategy: 20% for uniform sampling on the whole space and 80% for sampling near the surface. We adopt the mixed-sampling strategy because of the following two reasons. First, sampling uniformly in the 3D space will put more weight on the points outside the mesh during network training, which results in overfitting when sampling around the iso-surface. Second, sampling points far away from the reconstructed surface contribute little to geometry reconstruction, which increases the pressure of network training.

Overall, we enforce 3D geometric loss ${L}_{sdf}$ and normal constraint loss ${L}_{normal}$.

Here $\mathbb{C}(\cdot,\delta)=max(-\delta,min(\cdot,\delta))$.

We use the WCPA [1] dataset as training and testing datasets, which consists of 200 subjects for training and 50 subjects for testing. Each subject contains 15 actions, and each action contains 8 RGB images from different angles (0, 45, 90, 135, 180, 225, 270, 315). Each image is an 1280$\times$720 jpeg file. The ground truth of the canonical pose is a high-resolution 3D mesh with detailed information about clothes, faces, etc. For the training phase, we randomly select 4 RGB images with different views and poses as inputs to learn 3D human models.

The mask is then refined using the MatteFormer [17], which can generate better boundary details. Following [23], the trimap adopted in [17] is generated based on the mask using the erosion and dilation operation.

It is very challenging to get accurate SMPLX from 2D RGB images due to the inherent depth ambiguity. First, we use Openpose[3] to detect the 2D keypoints of the person in the image and then use ExPose[5] to estimate the SMPLX parameters and camera parameters. However, due to extreme illumination conditions and complex backgrounds, the SMPLX obtained in this way is not sufficiently accurate, and we need to refine the SMPLX parameters further. We utilize the 2D keypoints and masks to optimize SMPLX parameters $\theta,\beta$. For the 2D keypoints loss, given the SMPLX 3D joints location $J(\theta,\beta)$, we project them to the 2D image using the weak-perspective camera parameters $s,t$. For the mask loss, we utilize PyTorch3D[20] to render the 2D mask given the posed mesh $M(\theta,\beta)$. Then the mask loss is calculated based on the rendered mask and the pseudo ground truth mask $\mathcal{M}_{gt}$.

During training, we randomly choose $K=4$ images from total of $8*15$ images as input to extract the 2D feature map. Taking into account the memory limitation, we set the latent feature volume $\mathcal{V}\in\mathbb{R}^{224\times 224\times 224\times 64}$, where each latent code $\bm{l}^{i}\in\mathbb{R}^{64}$. During training, we randomly sample $N_{t}=10,000$ points around the complete mesh. To stably train our network, we initialize the SDF to approximate a unit sphere [25]. We adopt the Adam optimizer [12] and set the learning rate $lr=5e-4$, and it spends about 40 hours on 2 Nvidia GeForce RTX 3090 24GB GPUs. For inference, the surface mesh is extracted by the zero-level set of SDF by running marching cubes on a $256^{3}$ grid.

Our method achieved very good results in the challenge, demonstrating the superiority of our method for 3D human reconstruction from multi-view and multi-pose images. To further analyze the effectiveness of our method, we performed different ablation experiments. According to the results as shown in the table 1 and the figure 3, removing the background allows the model to better reconstruct the human body. The two-stage approach of predicting the normal vector map as input has a stronger generalization line on unseen data compared to directly using the image as input. What is more important is that the precision of SMPLX has a significant impact on the reconstruction performance, demonstrating the necessity of SMPLX optimization.