N-Cloth: Predicting 3D Cloth Deformation with Mesh-Based Networks

Physically-based algorithms use explicit Euler integration [Provot, 1995], implicit Euler integration [Baraff and Witkin, 1998], iteration optimization [Liu et al., 2013, 2017], or projective dynamics [Bouaziz et al., 2014] to calculate the cloth deformation under external/internal forces. Many techniques have been proposed for robust collision handling [Bridson et al., 2002; Brochu et al., 2012; Tang et al., 2014; Govindaraju et al., 2005]. Impulse-based methods and impact zones [Bridson et al., 2002; Harmon et al., 2008; Provot, 1997; Tang et al., 2018] are used for penetration handling. Recently many techniques have been proposed to accelerate these simulations using one or more GPUs [Tang et al., 2016; Li et al., 2020]. In practice, accurate cloth simulators can generate high–fidelity simulations, and we regard them as the ground truth for our learning approach.

Many data-driven approaches have been proposed for cloth deformation synthesis [Feng et al., 2010; Wang et al., 2010]. By combining high-quality wrinkles with a coarse cloth simulation [Zhang et al., 2021b; Chentanez et al., 2020], visually plausible results can be generated at interactive rates. These methods commonly need to simulate coarse deformed meshes. Furthermore, these methods need to precompute a large dataset, and their generalizability to arbitrary scenarios tends to be limited [de Aguiar et al., 2010; Kim et al., 2013; Zurdo et al., 2013].

Recently, learning-based algorithms have been proposed for predicting cloth deformation in 3D. Using the synthetic training data generated using physics-based simulators, learning-based approaches can predict cloth deformation at interactive rates on commodity GPUs. A large number of learning-based algorithms [Patel et al., 2020; Santesteban et al., 2019; Loper et al., 2015; Bertiche et al., 2020a, b; Wu et al., 2021; Santesteban et al., 2021; Wang et al., 2019; Corona et al., 2021] have been designed for specific or parametric obstacle models such as SMPL-based [Loper et al., 2015] or skeleton-based human body models. This makes it difficult to use these methods in environments with arbitrary rigid or deformable objects that can interact with the cloth.

Some learning-based algorithms are not limited to SMPL models. These approaches aim to process human bodies with skeleton. Holden et al. [Holden et al., 2019] obtain the vector of the vertex attributes of in the subspace through PCA and divide the deformation into linear and nonlinear to make assumptions and to predict the subsequent deformation. The GarNet network architecture proposed by Gundogdu et al. [Gundogdu et al., 2019, 2020] predicts the cloth deformation from the target posture with DQS (dual quaternion skinning) pre-processing  [Kavan et al., 2007] from the initial state and uses it as the final cloth deformation.  [Zhang et al., 2021a] learns to generate rendered characters and cloth on posed skeleton joints. All these methods focus on human bodies with skeleton and can not handle scenes without human skeletons. Our aim is to find an approach capable of processing many scenes with human meshes and other rigid body meshes.

Our goal is to predict the target deformed cloth mesh based on the target obstacle mesh and the initial cloth mesh. We assume that they are represented as 3D meshes. The initial cloth mesh provides a template for deformation. The target obstacle mesh is used to guide deformation. We do not make any assumptions about the initial topology of the cloth, though it remains fixed during the simulation or deformation. Thus, our network is an end-to-end method for predicting the cloth deformation. Formally, our approach can be described as:

where $M^{c}_{t}$ is the predicted deformed target cloth mesh and $M^{o}_{t}$ is the target obstacle mesh. $M^{c}_{i}$ is the initial, undeformed cloth mesh template which is undeformed and constant for one specific kind of cloth. During the prediction, the topologies of the cloth mesh and the obstacle mesh are invariable. $\mathcal{N}_{\theta}$ is the mesh-based network and $\theta$ represents the network parameters obtained by training.

We extend the classic encoder-decoder neural network architecture [Ranzato et al., 2007] to generate the 3D cloth deformation. An overview of our approach is shown in Fig. 2. The deformation in mesh space is nonlinear and too complicated to be modeled. The nonlinearity of the mesh deformation is transformed to linear fusion in latent space through a cloth encoder and an obstacle encoder. Thus, we obtain the latent representation of the target cloth mesh by linear fusion. The target cloth mesh is obtained by a cloth decoder. We will describe the network in more detail.

We use the encoder network to transform the input cloth mesh $M^{c}_{i}$ and the obstacle mesh $M^{o}_{t}$ from the 3D mesh space into a latent space. Our goal is to handle arbitrary triangle cloth meshes. A triangle mesh is similar to graph data. Therefore, the networks for processing graph data are applicable to resolving relevant mesh problems. Referring to [Gao and Ji, 2019], graph convolution is used in this paper to perform feature extraction on a mesh with abundant triangles. Both the cloth encoder and the obstacle encoder have similar network architectures.The first network block of our encoder network is shown in Fig. 3. Both the cloth encoder and the obstacle encoder have several network blocks similar to this initial block. Then we elaborate on the various parts of the first network block in the encoder. As shown in Fig. 3, we use the GCN layer [Kipf and Welling, 2016] to extract features from the geometry information (vertex coordinates) and topology information (edge connectivity) of the mesh. In this manner, the GCN layer maintains the topology of the mesh. The GCN layer can be formulated as in [Kipf and Welling, 2016]:

where $\tilde{A}^{(l)}=A^{(l)}+I$, $A^{(l)}$ is the adjacency matrix of layer $l$. $I$ is the identity matrix for adding self-loops. $X^{(l+1)}$ and $X^{(l)}$ are the feature matrices of the layer $l+1$ and $l$, respectively. $\tilde{D_{ii}}=\sum_{j}\tilde{A}_{ij}$, and $W^{(l)}$ is a trainable weight matrix for layer $l$.

Since the output of the GCN layer has the same topology as the input, this formulation can result in a large number of training parameters for high-resolution meshes and thereby exceed the GPU memory budget. To reduce the model parameters and complex non-linearity in 3D space, we use top-k-pooling [Gao and Ji, 2019] to perform data down-sampling by outputting meshes with reduced topology information, i.e., with fewer vertices and connectivity information among them.

Top-k-pooling will pick $k$ vertices from the original vertex set and discard other vertices to perform down-sampling. This process may result in multiple isolated point sets, which may not work well for subsequent GCN layers because these layers extract features in terms of information related to the of the vertex. Thus, we recalculate the connectivity of mesh vertices before using a top-k-pooling layer to improve triangle connectivity. Therefore, we calculate the square of the adjacency matrix $A$ as follows [Gao and Ji, 2019]:

where $A^{(l)}_{u}$ is the new adjacency matrix for next top-k-pooling computation. The new adjacency matrix $A^{(l)}_{u}$ corresponds to introducing more vertices around a given vertex and will reduce the isolated points after pooling.

Fig. 4 shows the connectivity between a single vertex and surrounding vertices in the mesh where the adjacency matrix is $A^{(l)}$ or $A_{u}^{(l)}$. The connectivity between vertices is strengthened with the adjacency matrix $A_{u}^{(l)}$.

The output of the cloth encoder and the obstacle encoder are $\mathbf{I}_{c}$ and $\mathbf{I}_{o}$, respectively. $\mathbf{I}_{c}$ and $\mathbf{I}_{o}$ are vectors in the latent space extracted from the cloth mesh and the obstacle mesh, respectively. $\mathbf{I}_{c}$ and $\mathbf{I}_{o}$ are expressed as follows:

where $\mathbf{E}_{c}$ and $\mathbf{E}_{o}$ represent the cloth encoder network and the obstacle encoder network, respectively. $m$ and $n$ are the dimensions of $\mathbf{I}_{c}$ and $\mathbf{I}_{o}$, respectively. $x_{1},x_{2},...,x_{m}$ and $w_{1},w_{2},...,w_{n}$ are the component of latent vectors $\mathbf{I}_{c}$ and $\mathbf{I}_{o}$, respectively.

We use the fusion network to generate $\mathbf{I}_{t}$, which corresponds to the vector of the target cloth mesh in the latent space from $\mathbf{I}_{c}$ and $\mathbf{I}_{o}$. Our formulation of the fusion network is inspired by prior work in image processing and 3D character control. Rocco et al. [Rocco et al., 2017] added a correlation layer to the network for geometric matching between 2D images, and Holden et al. [Holden et al., 2017] proposed a phase-functioned network the weights of which are updated by a phase cyclic function for 3D character control.

We perform the fusion by linearly weighting a set of linear fusion functions $\{f_{1},f_{2},f_{3},\cdots,f_{n}\}$, which all take $\mathbf{I}_{c}$ as an input. Here $\mathbf{I}_{o}$ is used as the linear weight. The linear fusion functions are defined as follows:

where $\{\mathbf{\alpha}_{i}^{1},\mathbf{\alpha}_{i}^{2},\cdots,\mathbf{\alpha}_{i}^{m}\}$ is a set of trainable coefficients and $i\in[1,n]$. The overall fusion process can be expressed by the following formula:

We use this formulation to obtain the vector $\mathbf{I}_{t}$ of the target cloth mesh in the latent space. Thus, we obtain a linear latent space where the deformation can be expressed as a linear fusion. In practice, we observe that our linear formulation can predict plausible results, and the errors are rather small (see Section 4).

In the mesh space, the influence of the obstacle mesh on the deformed cloth mesh may be complex and non-linear (e.g., due to collisions between the cloth and the obstacle). With our fusion network, we are able to model the non-linearity as weighted combinations of latent vectors and obtain the parameters of the combination function by training. In addition, the dimensions $m$ and $n$ of $\mathbf{I}_{c}$ and $\mathbf{I}_{o}$, respectively, also govern the accuracy of our predicted deformation. In our implementation, we set $m=96$ and $n=80$ for most benchmarks. We choose these dimensions by experiments and find that increasing them does not obviously improve the results.

We use the decoder network to generate a cloth mesh in the world space from $I_{t}$. The problem of recovering the cloth mesh from the latent space has been investigated by [Chen et al., 2020; Chentanez et al., 2020; Chen et al., 2021]. Although these methods use graph convolution networks, the resulting decoder networks use the same sampling information as the encoder networks. In addition, there is almost no deformation between the input mesh and the output mesh. However, this formulation does not work well when the target cloth mesh involves a large deformation relative to the initial mesh. In our formulation, we assume the cloth mesh maintain the same topology during deformation. As a result, we reduce the problem to only computing the geometric information of the deformed mesh (i.e., the vertex coordinates).

To compute the coordinates of the deformed vertices, we use a Multilayer Perceptron (MLP) network to decode the feature vector $\mathbf{I}_{t}$ of the target deformed cloth. In each layer of cloth decoder, we apply dropout regularization by randomly disabling 20% of the hidden neurons to avoid overfitting the training data. The output of our decoder network is a one-dimensional vector that will be reshaped to $(num,3)$ as vertex coordinates of the target cloth mesh, where $num$ is the number of vertices of the target cloth mesh. In addition, we add the output of each layer of the obstacle encoder to the input of the corresponding layer of the decoder through a linear layer connection. In this way, we increase the effects of obstacles on the cloth decoder and find improved results.

Loss function is a key component of our learning-based algorithm. We use different loss terms to achieve plausible results and overcome penetrations. We use the position information of the deformed cloth meshes as the ground truth and calculate the MSE loss between it and the network prediction output. The MSE loss on the positions can be expressed as:

where $x_{p}^{i}$ is the position of vertex $i$ on the predicted cloth mesh, $x_{g}^{i}$ is the position of vertex $i$ on the ground mesh, $N$ is the number of vertices, and $\left\|...\right\|_{2}$ is the $L^{2}$ distance.

In addition, we also use a new type of loss to remove penetrations between the generated cloth and the obstacle. Our goal is to generate non-colliding cloth meshes. The penetration loss between the cloth mesh and the obstacle mesh can be expressed as:

where $x_{p}^{i}$ is the position of vertex $i$ on the cloth mesh and $x_{{o}}^{i}$ is the nearest point to $x_{p}^{i}$ on the obstacle mesh. $\mathbf{n}_{{o}}^{i}$ is the normal vector of $x_{{o}}^{i}$ on the obstacle mesh. $d_{\epsilon}$ is the minimum distance of penetration. $N$ is the number of vertices of the cloth mesh. As shown in Fig. 12, the penetration loss can greatly overcome penetrations between a cloth mesh and a human body. We build the AABB tree of the obstacle to find the nearest point on the obstacle mesh. Fig. 13 shows the effectiveness of the penetration loss on obstacles with different numbers of triangles (0.36k, 0.64k, and 2.75k).

To prevent self-penetrations in the generated cloth mesh, we use the following loss function:

where $x_{p}^{j}$ is the nearest vertex to $x_{p}^{i}$ on the cloth mesh. $i\neq j$ and $i,j\in[1,N]$. We concatenate two pieces of cloth with opposite normal vectors of vertices to validate the self-penetration loss in Fig. 14. However, the experiments reveal that Eq. 9 is limited and cannot handle all self-penetrations.

The overall loss function used to predict the cloth deformation is:

where $\lambda$ and $\mu$ are blending coefficients. In practice, we use $\lambda=\mu=1$ and observe good results for all the benchmarks with these values. Since the predictions tend to be random at the beginning of the training, Eq. 8 and Eq. 9 may result in inaccurate predictions. We use Eq. 10 to train the network at the beginning and add Eq. 8 and Eq. 9 during the final training process. The number of parameters of our network and the computed gradients will hinder the scalability of training on large meshes. Therefore, Eq. 7 is computed on a GPU, while Eq. 8 and Eq. 9 are computed on a CPU.

With the trained network, we can obtain the predicted cloth mesh by inputting the initial cloth mesh and the target obstacle mesh. Since the initial cloth mesh representation has a fixed topology for a specific cloth, we only need to input the target obstacle mesh; our network is used to predict the deformed target cloth mesh.

We have implemented our algorithm on a standard PC (Ubuntu 18.04.4 LTS/Intel I7 CPU@4.2G Hz/8G RAM, NVIDIA GeForce RTX 3090 GPU). We perform both network training and cloth prediction on the same platform. Our implementation uses PyTorch 1.7.0 and Python 3.8.8 as the underlying development environment.

Our network can also predict different types of clothes. The child, Andy, wears different types of clothes, including a t-shirt, pants, a jacket and a dress. The jacket and dress are loose, and their deformation is different from the t-shirt and pants. These clothing simulations are also obtained by ArcSim [Narain et al., 2012, 2013; Pfaff et al., 2014], as above.

We evaluate the accuracy of our predicted meshes by measuring the mean error of each benchmark (as shown in Fig. 5) with the following equation:

where $M$ is the number of animation key frames. $N$ is the number of vertices in the 3D mesh, and $x_{P}^{i,j}$ and $x_{G}^{i,j}$ are the positions of vertex $i$ on frame $j$. The unit of our mean error is meters. The details for each scene are shown in Fig 5.

Moreover, the obstacle and cloth meshes in our scene have different vertices and topologies. Thus, networks used for different scenarios have different numbers of parameters. Therefore, we train an exclusive network for each scenario. The training time for each benchmark varies from $24$ hours to $7$ days.

To accelerate the convergence of the network, we perform data normalization. We normalize the input and output vertex positions to zero mean and unit variance for all frames. During training, we uniformly reduce the learning rate from $1e-3$ to $1e-5$. We use an Adam optimizer [Kingma and Ba, 2014] to train the parameters of the neural network.

In this section, we highlight the performance of our method on different benchmarks and compare the accuracy with physically-based simulation results. All predictions are performed on new test sets that are different from the training data.

Fig. 6 highlights our results on scenes with obstacles unseen during training corresponding to moving, rotating, or scaling rigid bodies. Our network results in favorable generalization to obstacles with unseen locations, rotations, and scales. Our approach makes no assumption about the topology of the obstacles or the cloth. We also compare the accuracy with ArcSim (an accurate physics-based simulator) and observe a high level of similarity between our predicted 3D mesh and the ground truth mesh. The mean deviation error between the vertices is less than $5$mm in our benchmarks. If there are multiple disjoint obstacles, we combine these obstacle meshes into a single mesh and generate the cloth predictions, as shown in Fig. 7. Fig. 8 highlights our results on different human body models. We use the same cloth mesh corresponding to a t-shirt on different human models. In Fig. 8, all the human bodies are represented with triangle meshes. All predictions are on an unseen test set. For example, the predictions of Andy and Qman are on the new action sequences downloaded from mixamo website. The results of SMPL are on the test data split from TailorNet. For all these benchmarks, our predicted results are visually close to the ground truth. Fig. 9 highlights the predictions of our method on unseen body shapes. We train a single network for bodies with different shapes since these meshes have the same topologies. Figs. 10 and 11 highlight our results on cloths of different types and resolutions. For all these benchmarks, our algorithm can generate plausible results that match the ground truth meshes. All predictions from test data are totally different from the training samples. Fig. 12 highlights the benefits of our penetration handling approach based on a loss function. By adding the penetration term into the loss function, our algorithm tends to alleviate the penetrations and self-collision artifacts. Although no penetration is unavailable in the predictions on the test set, it can reduce the degree of penetration and the subsequent processing work. Fig. 13 and Fig. 14 highlight the benefits of Eq. 8 (with different obstacle discretizations) and Eq. 9, where (self-)penetrations are effectively alleviated by these loss functions.

To sum up, our network can not only handle SMPL and non-SMPL human bodies, but also rigid obstacles. Our network can also process various types of clothes without providing predefined skin models for those clothes. Compared with the previous method, our network can handle more scenarios and has more applications. The predictions also show that our network can satisfactorily generalize to new, unseen data. Even when trained to predict a static deformed cloth mesh, our network generates a series of deformed cloth with fine temporal coherence on an obstacle sequence (shown in the video).

In Table 1, we list the characteristics of different learning-based methods. We highlight the capabilities of different methods in terms of the kind of obstacles they can handle (e.g., SMPL models only or rigid objects). Compared to prior methods, our approach makes no assumptions about the type or topology of the cloth or the obstacles in the scene. Most previous methods [Patel et al., 2020; Bertiche et al., 2020a; Santesteban et al., 2019; Bertiche et al., 2020b] are based on the SMPL model, which limits results to the SMPL human model. Other methods [Gundogdu et al., 2019; Guan et al., 2012; Wang et al., 2019] are limited to human models represented using joints. Although they can handle the non-SMPL human body, they are unable to process other obstacles such as a bunny.  [Holden et al., 2019] is a complimentary method that uses PCA and subspace-only physics simulation. However, it recurrently inputs the previous prediction and accumulates errors. This makes the predicted cloth mesh appear flat with fewer wrinkles. Our mesh-based method overcomes these limitations and can handle multiple, disjoint obstacles. Our predictions have no accumulation errors and can retain fine details like wrinkles and folds.

We perform a detailed comparison of our method with TailorNet [Patel et al., 2020], as the code and dataset are easily available. In Fig. 15, we use the same dataset as TailorNet for network training. We compare the accuracy of the predicted cloth meshes generated by our method and TailorNet. We compute the difference maps for each mesh by comparing the results with the ground truth mesh. We observe that our method predicts similar output meshes with richer details. In addition, our method produces fewer vertex errors compared to the ground truth than TailorNet. In benchmarks with many or detailed wrinkles, we observe that our network generates better results than TailorNet, as shown in Fig. 16. For example, our prediction of the cloth mesh has more wrinkles in the belly and shoulder areas, while TailorNet’s prediction is flatter.

We use the following error metrics for quantitative comparison between our mesh-based network and TailorNet:

where $x_{P}^{i}$ is the position of vertex $i$ of the predicted mesh $P$. $x_{G}^{i}$ is the position of its corresponding vertex on the ground truth mesh $G$. $y_{P}^{i}$ and $y_{G}^{i}$ are the normal vector at $x_{P}^{i}$ and $x_{G}^{i}$, respectively. $N$ is the number of vertices of the cloth mesh. $\Delta$ is the Laplace operator.

Figure 17 shows the error curve of our method and the predictions of Tailornet with ground truth on the test frames. The error is calculated as described above. From the curve, the trend of the errors of our network prediction is lower than TailorNet [Patel et al., 2020]. We also calculate the error mean and variance for all test frames, as shown in Table 2. The statistical value of the prediction error of our method is significantly lower than that of TailorNet.

We implement each layer of the cloth encoder and obstacle encoder with MLP as the baseline. The decoder in our network uses MLP, so we keep it invariable. Figure 19 shows the GPU memory usage when our network and baseline predict a single mesh. Our network occupies less GPU memory when making predictions. In experiments, it is revealed that more GPU memory is required for baseline training, which makes it impossible to train meshes with higher resolutions. However, our method is able to train cloth meshes with more than 100k triangles.

We have compared the running time of our method with a GPU-based physics-based simulator called I-Cloth [Tang et al., 2018], as shown in Figure 20. Compared to I-Cloth, our network achieves an order of magnitude performance improvement. Furthermore, the running time of our method does not change much with a higher resolution mesh. The overall accuracy and visual fidelity of the cloth mesh generated by our method are similar to those of I-Cloth.

We observe that our approach can obtain an interactive frame rate (about $30-45$ fps on an NVIDIA GeForce RTX 3090 GPU). Compared with TailorNet, our method has no obvious advantage in running time. This is because the input of the SMPL model is a small number of parameters, while our method inputs a mesh with all the vertex and edge information. The human mesh provided by TailorNet has 55k triangles, which will increase the calculation time of our network. In practice, we concentrate more on the deformation of the cloth mesh. Therefore, we can simplify the human body mesh, which will accelerate our network.

We implement a series of ablation experiments to verify the effectiveness of our network architecture. We remove the connections of each layer of the obstacle encoder in the decoder as a variant of our network. Furthermore, we discard the fusion network and use simple concatenation in the latent space as another variant. Figure 18 shows the predictions of our network and its two variants. Our network architecture plays an indispensable role in the convergence of results.