LaplacianFusion: Detailed 3D Clothed-Human Body Reconstruction

PCA-based parametric models have been proposed for handling human body and pose variations: SMPL (Loper et al., 2015; Romero et al., 2017; Pavlakos et al., 2019), GHUM (Xu et al., 2020), and Frank model (Joo et al., 2018). These models can handle body shape variations and pose-dependent shape deformations that cannot be modeled with Linear Blend Skinning (LBS) (Lewis et al., 2000). The models are suitable for expressing human shapes with coarse meshes, but they alone are not enough for containing rich details.

Several approaches represented clothed humans by extending parametric human models. SMPL (Loper et al., 2015) has been extended to express clothed deformations by directly adding a displacement vector to each vertex (Alldieck et al., 2019, 2018a, 2018b; Bhatnagar et al., 2020). CAPE (Ma et al., 2020) proposed an extended model by adding a cloth style term to SMPL. Other approaches (De Aguiar et al., 2010; Guan et al., 2012; Pons-Moll et al., 2017; Bhatnagar et al., 2019; Tiwari et al., 2020; Xiang et al., 2020) used additional parametric models for representing clothes on top of a parametric human model. Additional approaches for expressing surface details include GAN-based normal map generation (Lahner et al., 2018), RNN-based regression (Santesteban et al., 2019), and style-shape specific MLP functions (Patel et al., 2020). However, these approaches are limited to several pre-defined clothes and cannot recover detailed human shapes with arbitrary clothes from input scans.

Truncated signed distance function (TSDF) is a classical implicit representation for reconstruction. TSDF-based approaches that warp and fuse the input depth sequence onto the canonical volume have been proposed for recovering dynamic objects (Newcombe et al., 2015; Innmann et al., 2016). This volume fusion mechanism is extended to be conditioned with the human body prior for representing a clothed human (Yu et al., 2017; Tao et al., 2018). Optimized approaches for real-time performance capture have also been proposed (Dou et al., 2016; Dou et al., 2017; Habermann et al., 2020; Yu et al., 2021).

MLP-based neural implicit representation has been actively investigated for object reconstruction (Park et al., 2019; Mescheder et al., 2019; Chen and Zhang, 2019). PIFu (Saito et al., 2019; Saito et al., 2020) firstly adopted this representation for reconstructing static clothed human from a single image. DoubleField (Shao et al., 2022) uses multi-view RGB cameras and improves visual quality by sharing the learning space for geometry and texture. With depth image or point cloud input, multi-scale features (Chibane et al., 2020; Bhatnagar et al., 2020) and human part classifiers (Bhatnagar et al., 2020) have been used for reconstructing 3D human shapes. Li et al. (2021b) proposed implicit surface fusion from a depth stream and enabled detailed reconstruction even for invisible regions.

Neural parametric models have also been proposed for modeling shape and pose deformations. NASA (Deng et al., 2020) learns pose-dependent deformations using part-separate implicit functions. LEAP (Mihajlovic et al., 2021) and imGHUM (Alldieck et al., 2021) learn parametric models that can recover shape and pose parameters for SMPL (Loper et al., 2015) and GHUM (Xu et al., 2020), respectively. NPMs (Palafox et al., 2021) encodes shape and pose variations into two disentangled latent spaces using auto-decoders (Park et al., 2019). SPAMs (Palafox et al., 2022) introduces a part-based disentangled representation of the latent space.

For subject-specific clothed human reconstruction from scans, recent implicit methods define shape details in a canonical shape and use linear blend skinning to achieve both detail-preservation and controllability. Neural-GIF (Tiwari et al., 2021) learns a backward mapping network for mapping points to the canonical space and a displacement network working in the canonical space. In contrast, SNARF (Chen et al., 2021) proposed a forward skinning network model to better handle unseen poses. SCANimate (Saito et al., 2021) learns forward and backward skinning networks with a cycle loss to reconstruct disentangled surface shape and pose-dependent deformations. MetaAvatar (Wang et al., 2021) proposed an efficient pipeline for subject-specific fine-tuning using meta-learning.

These implicit representations are topology-free and can handle vastly changing shapes, such as loose clothes. However, they individually perform shape reconstruction at every frame and cannot provide temporally consistent mesh topology needed for animation. In addition, this approach needs dense point sampling in a 3D volume to train implicit functions and is computationally heavy.

SCALE (Ma et al., 2021a) recovers controllable clothed human shapes by representing local details with a set of points sampled on surface patches. To avoid artifacts of the patch-based approach at patch boundaries, PoP (Ma et al., 2021b) represents local details using point samples on a global 2D map. Point cloud representation has a flexible topology and can cover more geometric details. However, this representation does not provide an explicit output mesh.

For subject-specific human reconstruction with a depth sequence, DSFN (Burov et al., 2021) represents surface details using vertex offsets on a finer resolution mesh obtained by subdividing the SMPL template mesh. This approach is the closest to ours. The main difference is that our method represents surface details using Laplacian coordinates, instead of vertex offsets. Experimental results show that Laplacian coordinates are more effective for recovering surface details than vertex offsets (Section 8).

Table 1 compares our method with related ones in terms of possible inputs and desirable properties.

In the inputs of functions $f_{d}$ and $f_{l}$, we use the concept of query point to neutralize shape variations of the base meshes for different subjects and at different frames. We define a query point $Q(\cdot)$ as a 3D point on the T-posed canonical neutral SMPL model $\mathcal{M}_{C}$. Consider two base meshes $\mathcal{B}_{t1}$ and $\mathcal{B}_{t2}$ for the same subject at different frames and their $k$-th vertices $\mathbf{v}_{t1,k}$ and $\mathbf{v}_{t2,k}$, respectively. 3D positions of $\mathbf{v}_{t1,k}$ and $\mathbf{v}_{t2,k}$ may differ, but their query points $Q(\mathbf{v}_{t1,k})$ and $Q(\mathbf{v}_{t2,k})$ are defined to be same as they share the same vertex index (Figure 4a). Similarly, $Q(\cdot)$ is defined to be the same for the vertices of base meshes representing different subjects if the vertices have been deformed from the same vertex of $\mathcal{M}_{C}$. In addition, $Q(\cdot)$ can be defined for any point on a base mesh other than vertices by using the barycentric coordinates in the mesh triangles. Once determined, a query point is converted to a high-dimensional vector via a positional encoding $\gamma$ (Mildenhall et al., 2020), and we choose the dimension of ten in our experiments.

For a query point, our two MLPs $f_{d}$ and $f_{l}$ should estimate pose-dependent deformation and Laplacian coordinates, respectively. To provide the pose-dependency, we could simply include the pose parameter $\boldsymbol{\theta}$ in the input of the MLPs. However, a query point is not affected by every joint but strongly associated with nearby joints. For example, the joint angle at the shoulder is irrelevant to local details on a leg. To exploit the correlations between query points and joint angles in a pose parameter $\boldsymbol{\theta}$, we convert $\boldsymbol{\theta}$ to a per-point pose feature $\overline{\boldsymbol{\theta}}(\mathbf{v})$ that retains only relevant joint angles for a query point $\mathbf{v}$. Inspired by Pose Map (Saito et al., 2021), we apply the joint association weight map $\mathbf{W}\in\mathbb{R}^{J\times J}$ and skinning weights $\mathbf{w}(\mathbf{v})\in\mathbb{R}^{J}$ of $\mathbf{v}$ to the original pose parameter $\boldsymbol{\theta}\in\mathbb{R}^{J\times 3}$. Our pose feature used as the input of MLPs is defined by

where $diag(\cdot)$ converts an input vector to a diagonal matrix, and $\lceil\cdot\rceil$ is an element-wise ceiling operation. We manually define the weight map $\mathbf{W}$ to reflect our setting. For example, the details of the head are not correlated with any joint in our reconstructed model, and the details of a leg are affected by all nearby joints together. Then, for the head joint, we set zero for the association weight of any joint in $\mathbf{W}$. For a joint around a leg, we set higher association weights for all nearby joints (Figure 4b).

As discussed in Section 3.2, we use an approximation method (Liang et al., 2012) for calculating Laplacian coordinates from scan points that do not have connectivity. We first locally fit a degree two polynomial surface for each point in the moving least squares manner. In our experiments, we use 20-30 neighbor points for the local surface fitting. Figure 5 shows illustration. Our approximate Laplacian coordinates are defined on a continuous domain, unlike in conventional mesh editing methods (Lipman et al., 2004; Sorkine et al., 2004) that formulate Laplacian coordinates in a discrete domain using mesh vertices. To apply our Laplacian coordinates to a discrete mesh, we take advantage of a pose-dependent base mesh (Section 7).

Although the base mesh $\mathcal{B}_{t}$ nearly fits the input point cloud, points may not reside exactly on the mesh. To obtain the corresponding query point, we project each point $\mathbf{p}$ in the input point cloud $\mathbf{P}_{t}$ onto the base mesh $\mathcal{B}_{t}$: $\overline{\mathbf{p}}=\Pi\left(\mathcal{B}_{t},\mathbf{p}\right)$, where $\Pi$ denotes the projection operation from a point to a pose-dependent base mesh (Figure 3b). The position of $\,\overline{\mathbf{p}}_{t,i}$ is determined by the barycentric coordinates in a triangle of $\mathcal{B}_{t}$, so we can easily compute the query point, skinning weights, and pose feature using the barycentric weights.

For a given surface point $\overline{\mathbf{p}}$ and pose $\boldsymbol{\theta}$, the MLP $f_{l}$ estimates Laplacian coordinates:

The estimation is conducted in the canonical space and transformed into the posed space. Working in the canonical space is essential because Laplacian coordinates are not invariant to rotation. In Eq. (8), we discard the translation part in Eq. (3) as Laplacian coordinates are differential quantity. The estimated $\boldsymbol{\delta}^{\prime}$ is non-uniform Laplacian coordinates, as described in Section 3.2.

We train the MLP $f_{l}$ by formulating a per-point energy function:

where $\boldsymbol{\delta}^{\prime}_{t,i}$ is the Laplacian coordinates of $\overline{\mathbf{p}}_{t,i}$ predicted by $f_{l}$ using Eq. (8), $\boldsymbol{\delta}_{t,i}$ is the GT approximate Laplacian coordinates of $\mathbf{p}_{t,i}$, and $\mu_{t,i}$ is the weight used in Eq. (5). During training, we construct the total energy by summing per-point energies of randomly sampled input points, and optimize the total energy using Adam optimizer (Kingma and Ba, 2015).

Pose-dependent local deformation function $f_{d}$ and neural surface Laplacian function $f_{l}$ are represented as 5- and 3-layer MLPs with ReLU activation, and 600 and 800 feature channels are used per intermediate layer, respectively. In our experiments, we set $\lambda_{r}=0.1\sim 2$ and $\lambda_{a}=2$. We optimize $f_{d}$ and $f_{l}$ with a learning rate of $1.0\times 10^{-3}$, batch sizes of $10$ frames and 5000 points, and 300 and 100 epochs, respectively. The training time is proportional to the numbers of points and frames in the scan data. For instance, the point cloud sequence used for reconstruction in Figure 7 consists of 39k$\sim$56k points per frame with 200 frames. In that case, it takes about 20 and 30 minutes to train MLPs $f_{d}$ and $f_{l}$, respectively.

We evaluate the results of our method qualitatively and quantitatively on single-view and full-body point cloud sequences. We capture RGB-D sequences using an Azure Kinect DK (Kin, 2022) with 1280p resolution for color images and $1024\times 1024$ for depth images. Additionally, we use RGB-D sequences that are provided in DSFN (Burov et al., 2021). RGB-D sequences used for experiments contain 200 to 500 frames. We also evaluate our method on full-body point clouds using synthetic datasets: CAFE (Ma et al., 2020), and Resynth (Ma et al., 2021b). We show the reconstruction result from a 2.5D depth point cloud in mint color and the result of a full-body 3D point cloud in olive color.

In the reconstruction step, for the example in Figure 7, it takes 3ms and 35ms per frame to evaluate the MLPs $f_{d}$ and $f_{l}$, respectively. For solving a sparse linear system to obtain the final detailed mesh $\mathcal{S}$ that minimizes Eq. (10), it takes 4 seconds for the matrix pre-factorization step and 130ms to obtain the solution for each frame when the mesh contains 113k vertices.

Our approach using Laplacian coordinates preserves local geometric details better than the approach using absolute coordinates of mesh vertices. To verify the claim, we conduct an experiment by changing the neural surface Laplacian function to estimate displacements instead of Laplacian coordinates. We think that this surface function mimics a pose-dependent displacement map. To optimize the surface displacement function, we use the same energy function as Eq. (9) with the change of $\boldsymbol{\delta}$ to the displacement between surface point $\overline{\mathbf{p}}$ and scan point $\mathbf{p}$. There is no regularization term in Eq. (9), and the maximal capability of the displacement function is exploited for encoding surface details. Nevertheless, the resulting displacement function cannot properly capture structure details, producing rather noisy surfaces (Figure 7 middle). In contrast, our results using Laplacian coordinates are capable of capturing structure details (Figure 7 right) from the input point clouds (Figure 7 left). In Figure 7, we include quantitative results (Chamfer distance $d_{CD}$ and normal consistancy $NC$) on our dataset. The normal consistency $NC$ is computed using the inner products of the ground truth normals at input points and the normals of the corresponding points on the reconstructed surface. Our result shows a slightly higher Chamfer distance error than the displacement function, but produces more visually pleasing results.

We conduct an experiment that learns the local deformation defined in Sec. 5.2 for a subdivided base mesh that has the same topology as our final mesh. The results (Figure 8 left) show that estimating absolute coordinates is sensitive to the regularization weight $\lambda_{r}$ defined in Eq. (5). With a large value of $\lambda_{r}$, shape details are smoothed out in the reconstructed model. With a small $\lambda_{r}$, too small details are reconstructed. Then, it is hard to find the best regularization weight to handle spatially varying structures and sizes of shape details in the input scan (Figure 8 top right). On the contrary, our complete pipeline that learns differential representation performs better without a regularization term (Figure 8 bottom right). Note that all meshes in Figure 8 has the same number of vertices.

We use selective weighting $\mu_{t,i}$ in Eq. (5) and Eq. (9) to obtain the best details from the input depth map sequence. This weighting is especially effective in the face region (Figure 9).

DSFN (Burov et al., 2021) utilizes an explicit mesh to reconstruct a controllable 3D clothed human model from a RGB-D sequence. It represents local details with an offset surface function (dense local deformation), but its spatial regularization smooths out geometric details. Since the source code of DSFN is not published, we compared our results with DSFN on the dataset provided by the authors.

Their real data inputs are captured using an Azure Kinect DK (Kin, 2022) with $1920\times 1080$ pix. for color images, and $640\times 576$ pix. for depth images. Figure 10 shows results on the real dataset provided by DSFN, and our approach produces more plausible details than DSFN. Note that, in this dataset, the captured human stands away from the camera, so input point cloud lacks high-frequency details.

For quantitative evaluation, DSFN uses synthetic $640\times 480$ pix. RGB-D sequences rendered from BUFF (Zhang et al., 2017), where the camera moves around a subject to cover the whole body. Figure 11 shows the comparison results, and our method preserves the details of the input better. Table 2 shows the measured IOU, chamfer distance ($d_{CD}$), normal consistency ($NC$) scores. All three scores of our approach rate better than DSFN.

Wang et al. (2021) propose MetaAvatar that represents shape as SDF defined on 3D space. The resulting shape tends to be smoothed due to the limited number of sampling points in the training time. MetaAvatar utilizes the given SMPL parameters for reconstruction, and we use them in the comparison. MetaAvatar evaluates the reconstruction quality using unseen poses with the interpolation capability aspect. Similarly, we sample every 4th frame in the CAPE dataset (Ma et al., 2020) and measure the reconstruction accuracy on every second frame excluding the training frames. The results are shown in Figure 12. In Table 3, we show quantitative results, and the scores of our approach rate better than MetaAvatar.

Ma et al. (2021b) utilize point cloud representation to deal with topological changes efficiently. We evaluate their method on CAPE dataset in the same configuration used for MetaAvatar in Table 3. In addition, we compare our results with PoP on Resynth dataset (Ma et al., 2021b), which has various cloth types with abundant details. However, Resynth dataset includes human models wearing skirts that cannot be properly handled by our framework (Figure 18), so we do not conduct quantitative evaluation using all subjects. Instead, we select five subjects not wearing skirts to compare our results with PoP quantitatively. We then split the training and test datasets in the same manner as on CAPE dataset.

The original implementation of PoP queries the feature tensor with a $256\times 256$ UV map, resulting in a point cloud with 50k points. Since they are insufficient for representing details on the Resynth dataset, we modified the code to adopt $512\times 512$ UV map, and obtain 191k points. Table 4 shows quantitative comparisons with the above two settings, and our reconstruction is comparable with the original PoP. Figure 13 presents qualitative results where our mesh is more plausible than the original PoP, and comparable to dense PoP.

In Figure 14, the mesh result obtained from a reconstructed point cloud of PoP contains a vertical line on the back because PoP uses UV coordinates for shape inference. In contrast, our approach can reconstruct clear wrinkles without such artifacts.

Our neural surface Laplacian function encodes detailed shape information as the Laplacian coordinates defined at query points in a common domain. As a result, we can easily transfer shape details to other models by evaluating the original surface Laplacian function on the target pose-dependent base mesh. Figure 15 shows an example.

Laplacian coordinates predicted by a neural surface Laplacian function in the Laplacian reconstruction step (Section 7) can be scaled to change the amount of reconstructed shape details. Multiplying a value greater than 1.0 to the predicted Laplacian coordinates performs detail sharpening, and the opposite performs detail smoothing. Figure 15 shows examples.

The pose parameters used for evaluating surface functions $f_{d}$ and $f_{l}$ can be arbitrary. In the case of reconstructing a scanned animation sequence, we would use the pose parameters $\boldsymbol{\theta}_{t}$ estimated from the input scans (Section 5.1). On the other hand, once the functions $f_{d}$ and $f_{l}$ have been optimized, any parameters $\boldsymbol{\theta}$ other than $\boldsymbol{\theta}_{t}$ can be used to produce unseen poses of the subject. In that case, the validity of shape details of the unseen poses is not guaranteed but our experiments generate reasonable results. In Figure 16, we optimized the functions $f_{d}$ and $f_{l}$ on BUFF and CAPE datasets and adopted synthetic motions from AIST++ (Li et al., 2021a). The results show natural variations of shape details depending on the motions.

In our LaplacianFusion framework, all reconstructed models have the same fixed topology resulting from the subdivision applied to the SMPL model. Then, we can build a common UV parametric domain for texture mapping of any reconstructed model. In Figure 17, we initially transfer a texture from a RenderPoeple model (ren, 2022) to the T-posed canonical neutral SMPL model $\mathcal{M}_{C}$ by using deep virtual markers (Kim et al., 2021). Then, the texture of $\mathcal{M}_{C}$ can be shared with different reconstructed models through the common UV parametric domain.