DiffCloud: Real-to-Sim from Point Clouds with
Differentiable Simulation and Rendering of Deformable Objects

Many approaches in deformable object manipulation are limited to specific scenarios due to the complexity of real hardware setups [7, 8, 9, 10, 11, 12]. Furthermore, there is a lack of easily tunable yet realistic simulators that support deformables and could aid experimentation with novel tasks. Hence, automated ways to find simulation parameters are needed to make the behavior of simulated deformables resemble that of their real-world counterpart. In robotics, this real-to-sim problem has been extensively studied for rigid objects [13, 14, 15, 16, 17, 18]. However, these methods assume access to a low-dimensional state, such as object poses. Recent surveys have reviewed learning-based approaches for manipulation and perception of deformables [19, 20, 21], including methods for tracking and registration. However, such methods either employ markers [22], assume a known and simplified deformation model [23], or show applicability only to specific cases, e.g. a rope lying flat on a surface [24]. As of now, there is no generic tracking or registration approach for deformables that is robust in a wide range of scenarios.

One approach towards real-to-sim is to learn an inverse model that maps from a sequence of sensor observations to simulation parameters [25]. In robotics, several earlier works explored using inverse models for highly deformable objects. For example, in [26] the authors propose a piecewise linear elastic material model for cloth, then they fit the material model to real cloths by applying controlled forces and measuring the deformation response. This results in a paired dataset of cloth types and estimated material parameters, but limits generalization to unseen cloths without additional physical measurements. In [27], the authors propose using the dataset of [26] as a source of supervision for training a network to regress material parameters from RGB videos of cloth. This supervised training approach relies heavily on domain randomization to achieve generalization to cloths with unseen colors or patterns. Additionally, the proposed regression framework discretizes the space of outputs to deal with the high dimensional model of cloth deformation, limiting the range of cloth types that can ultimately be accurately modelled. In this work, we use a sequence of depth images (converted to point clouds) as input to our models. Using point clouds instead or RGB images allows us to circumvent the need for extensive domain randomization and for contrasting textures (for foreground/background segmentation). We adopt established methods for processing point clouds, such as PointNet/PointNet++ [28, 5], MeteorNet [6] and train them to regress to continuous simulation parameters.

Differentiable rendering of images allows using 2D observations to inform 3D understanding, by propagating gradients from images to illumination models, 3D objects in the scene, and camera pose. Various works have explored making the rasterization process (coloring image pixels based on the assignment of triangular mesh faces) fully differentiable, including DIB-R [29] and SoftRAS [30]. Other works explore making the processes of ray tracing [31] and volumetric rendering [32, 33] differentiable. Toolkits, such as Kaolin [34] and PyTorch3D [35] provide batched implementations of differentiable rasterization, allowing a significant speedup of these computationally expensive processes. In this work, we aim to do correspondence-free alignment of simulated object states, such as meshes, with real visual observations. While images are one such observation space to bridge the gap, achieving close alignment between images that are differentiably rendered from simulated meshes and real images still remains an open challenge due to the range of colors, textures, and fine details present in real images. Thus, we use point clouds to represent objects and build on PyTorch3D [35], which offers a way to differentiably sample a point cloud from a given mesh such that gradients from losses in the point clouds propagate to the mesh vertex gradients. Furthermore, PyTorch3D library exposes efficient data structures and CUDA-enabled batched operations for 3D meshes and point clouds, allowing seamless integration into simulator backends that support PyTorch.

Differentiable simulation is a promising paradigm that allows to adapt simulators by propagating gradients of the simulator outputs w.r.t. the lower-level physical simulation parameters. Recent differentiable simulators with support for deformables include [36, 37, 38, 39, 40, 41, 42, 43, 44, 45]. Most of these only support a limited set of interactions and types of deformation. For example, out of the above, only [36] (a differentiable version of ARCSim [46]) supports arbitrary meshes and modeling interactions of rigid and deformable objects. From the simulators listed above, only gradSim [37] provides a combination of differentiable RGB rendering and simulation. This approach is conceptually related to our work, but is still substantially different, since it does not handle point clouds, does not support interactions between rigid and deformable objects, and presents only sim-to-sim results with simple simulated images and plain textures. The differentiable simulation framework proposed in [47] shows results on inferring deformation properties of a real object from point clouds. We do not build on this approach directly, because it has several key limitations. First, there are limitations in the underlying simulator: the model is aimed for objects with low-to-medium levels of deformability, such as tight plush toys and pillows. In contrast, we are interested in highly deformable objects, such as cloth, garments, and stretchable bands. Second, there are limitations in the loss formulation: the point cloud loss proposed in [47] relies on a simple diffusion of the observed deformation into a deformation field. This is not suitable for the case of highly deformable objects that rapidly and drastically change shape under gravity and upon interacting with other objects. Finally, this prior work is geared towards analyzing cases with passive dynamics of a single object falling and colliding with the table or bending under gravity; the simulator also lacks support for modeling frictional contacts. We are instead interested in actuating a robot to perform manipulation with highly deformable objects that can also interact in non-trivial ways with rigid objects in the scene. We found that [36] is the only available simulator that readily supports such functionality, hence we incorporate it into our approach.

One candidate for a loss function on point clouds is the Chamfer distance:

$\displaystyle\begin{split}d_{\text{Chamf}}(\mathcal{P}^{a},\mathcal{P}^{b})\!=&\tfrac{1}{|\mathcal{P}^{a}|}\!\sum_{\boldsymbol{x}^{a}\in\mathcal{P}^{a}}\!\!\min_{\boldsymbol{x}^{b}\in\mathcal{P}^{b}}||\boldsymbol{x}^{a}-\boldsymbol{x}^{b}||_{2}^{2}\\ &+\!\tfrac{1}{|\mathcal{P}^{b}|}\!\sum_{\boldsymbol{x}^{b}\in\mathcal{P}^{b}}\!\!\min_{\boldsymbol{x}^{a}\in\mathcal{P}^{a}}||\boldsymbol{x}^{b}-\boldsymbol{x}^{a}||_{2}^{2}.\end{split}$

(1)

Here, $\mathcal{P}^{a},\mathcal{P}^{b}$ denote two point clouds; $\boldsymbol{x}^{a},\boldsymbol{x}^{b}$ are 3D points in $\mathcal{P}^{a}$ and $\mathcal{P}^{b}$ respectively. This distance metric is frequently used to compare the alignment between two point clouds that are either complete (sampled from meshes) or are generated by perception with the same camera perspective and noise properties. In our case of using low-cost depth sensors, the challenge of constructing the loss function on the real and simulated point clouds is that we need to avoid paying attention to the noise artifacts in the real point cloud. Statistical outlier filtering and post-processing techniques can alleviate noise, but some amount is likely to remain. For example, see the red patches highlighted in the real point cloud in Figure 1. Our insight is that a loss that disregards noise artifacts can be obtained by using a unidirectional Chamfer distance. This yields a loss that relieves the pressure for the simulated point clouds to match the noisy parts of the real point clouds:

$\displaystyle\vspace{-5px}\begin{split}d_{\text{Chamf}}^{\text{sim}\rightarrow\text{real}}(\mathcal{P}^{\text{sim}},\mathcal{P}^{\text{real}})\!=\!\!\sum_{\boldsymbol{x}^{\text{sim}}\in\mathcal{P}^{\text{sim}}}\!\!\min_{\boldsymbol{x}^{\text{real}}\in\mathcal{P}^{\text{real}}}||\boldsymbol{x}^{\text{sim}}-\boldsymbol{x}^{\text{real}}||_{2}^{2}.\end{split}$

(2)

While the naive computation of the Chamfer distance can be expensive, PyTorch3D provides an efficient GPU-based implementation (see Section 3.1 in [35]). With that, we can quickly propagate gradients from the point clouds through the mesh representation to optimize the low-level physical parameters of the simulator. We found that including point clouds from one or two depth cameras is sufficient to construct a partially occluded point cloud that is still informative enough for the overall optimization to be successful.

We build upon the differentiable simulator DiffSim [36], which supports mesh-based simulation and contact-handling of rigid objects and thin-shell deformables (e.g. cloth). In DiffSim, the simulation state is represented by generalized coordinates $\mathbf{q}=[\mathbf{q_{1}}^{T},\mathbf{q_{2}}^{T},\ldots,\mathbf{q_{n}}^{T}]^{T}$ of all objects in the simulation with corresponding velocities $\mathbf{\dot{q}}=[\mathbf{\dot{q}_{1}}^{T},\mathbf{\dot{q}_{2}}^{T},\ldots,\mathbf{\dot{q}_{n}}^{T}]^{T}$. The generalized coordinates of a rigid body have $\mathbf{q_{i}}\in\mathbb{R}^{6}$ for rigid objects, denoting position and orientation, and $\mathbf{q_{i}}\in\mathbb{R}^{3}$ for deformable nodes with 3 DoF for position alone. Here, $n$ is the cumulative total of the number of deformable nodes and rigid bodies in the scene. DiffSim uses the implicit Euler method to compute $\mathbf{q,\dot{q}}$ at each time step and performs collision resolution in localized impact zones. A given mesh has a body frame with the origin set to its center of mass at the start of simulation. A mesh vertex $p$ has coordinate $\mathbf{p_{0}}=(p_{x},p_{y},p_{z})^{T}$ in the body frame and world coordinate $\mathbf{p}=\mathbf{f}(\mathbf{q})=[\mathbf{r}]\mathbf{p_{0}}+\mathbf{t}$, where $\mathbf{r}=(\phi,\theta,\psi)^{T}$ and $\mathbf{t}=(t_{x},t_{y},t_{z})$ is the 6-DoF pose of the mesh. Propagating gradients from vertex $\mathbf{p}$ to the generalized coordinates $\mathbf{q}$ involves computing the Jacobian $\nabla\mathbf{f}$ and obtaining the partial derivatives $\partial{\mathbf{f(q)}}/\partial{\mathbf{q}}$. In this way, DiffSim is able to solve sim-to-sim optimizations by comparing current mesh states to target mesh states and propagating gradients from observed positional differences.

Since ground truth mesh states are not readily available for real deformables, we use point clouds as an observation space. To allow losses in the point cloud space to be propagated to simulated cloth vertices, we implement a differentiable point cloud sampling step. A triangular mesh face can be represented by its enclosing vertices $\mathbf{p_{1}},\mathbf{p_{2}},\mathbf{p_{3}}$. In barycentric coordinates, a random point on the surface of the face can be generated by sampling 3 random numbers $(u,v,w)$ such that $u+v+w\!\leq\!1$ [35]. Hence, we can obtain a random point $\boldsymbol{x}$ inside the triangle as:

$$\boldsymbol{x}=u\mathbf{p_{1}}+v\mathbf{p_{2}}+w\mathbf{p_{3}}.$$

(3)

To generate an $N$-point, uniform-density point cloud from a mesh, we first sample $N$ triangular faces $\{(\mathbf{p_{i,1},p_{i,2},p_{i,3}})\}_{i=1,\ldots,N}$, weighted proportionally to the area of each face. This step ensures that the resulting point cloud is evenly distributed over the surface of the mesh, instead of being disproportionately dense in regions where the mesh has closely packed faces. For the $i^{\text{th}}$ sampled face $(\mathbf{p_{i,1},p_{i,2},p_{i,3}})$, we generate random coefficients $(u_{i},v_{i},w_{i})$ and use Equation 3 to compute a point $\boldsymbol{x}_{i}$ that lies on the face. Concatenating the points obtained by applying this procedure to the $N$ sampled faces and coefficients yields the point cloud $\mathcal{P}\!=\!\{\boldsymbol{x}_{i}\}_{i=1,\ldots,N}$. We connect the PyTorch3D [35] implementation of this sampling procedure to the output of a differentiable simulator. With that, gradients from loss functions operating on $\{\boldsymbol{x}_{i}\}_{i=1,\ldots,N}$ can be propagated to mesh vertices $(\mathbf{p_{i,1},p_{i,2},p_{i,3}})$ via chain rule, observing that:

$$\partial{\boldsymbol{x}_{i}}/\partial{\mathbf{p_{i,1}}}=u_{i}\hskip 8.5359pt\partial{\boldsymbol{x}_{i}}/\partial{\mathbf{p_{i,2}}}=v_{i}\hskip 8.5359pt\partial{\boldsymbol{x}_{i}}/\partial{\mathbf{p_{i,3}}}=w_{i}$$

(4)

We consider scenarios where a robot executes a trajectory to manipulate a deformable object, which potentially also interacts with other objects in the scene. For each scenario, we get a real point cloud sequence of length $T$ and a resolution of $N$ points per frame: $\big{\{}\mathcal{P}^{\mathrm{real}}_{t}\!=\!\{\boldsymbol{x}_{i}^{\text{real}}\}_{i=1,\ldots,N}\big{\}}_{t=1}^{T}$. We instantiate each real scenario in DiffSim using representative meshes to model the deformable and rigid objects in the scene initially. Next, we begin optimizing the material properties of the simulated cloth such that the discovered parameters best explain the observed target point cloud data. Each simulated scenario involves manipulating a simulated cloth with the same trajectory as in real. Even with identical trajectory execution, we expect a discrepancy between the real and simulated point cloud sequences due to mismatch in the motion of the deformables, whose physical properties we aim to optimize. For each iteration of optimization, we run the simulation for the number of steps in the scenario horizon using the current set of estimated parameters. At each timestep $t$, we sample the simulated object surfaces in the scene according to Equation 3 to obtain a point cloud. The concatenated point clouds across frames give the simulated point cloud sequence: $\big{\{}\mathcal{P}^{\mathrm{sim}}_{t}=\{\boldsymbol{x}_{i}^{\text{sim}}\}_{i=1,\ldots,N}\big{\}}_{t=1}^{T}$. In practice, we compute the unidirectional Chamfer distance (Equation 2) on $(\mathcal{P}^{\mathrm{sim}}_{t},\mathcal{P}^{\mathrm{real}}_{t})$ for the corresponding point clouds observed at time $t$. For most experiments, we found that using only the last frame works well, i.e. $t=T$, as in [37, 36]. Other frames can be used, and their selection could be considered as a hyperparameter. This loss is propagated across all frames of the simulation and used to update the underlying parameters of the deformable object. This procedure is repeated for a fixed number of iterations or until the Chamfer distance falls below a threshold.

As experimental baselines, we use methods that view simulators as black-box, i.e. not treating observations as end-to-end differentiable w.r.t the parameters. These inverse models are implemented as regression networks that take point cloud sequences as inputs and predict $k$ simulation parameters. They are trained on simulated point cloud sequences generated by simulations with various simulation parameters.

• •

  MeteorNet: An architecture for learning representations of 3D point cloud sequences from [6], which we modify to predict $k$ simulation parameters. Specifically, we use MeteorNet-cls (Appendix D.2 in [6]).

• •

  PointNet++: A regressor similar to the above, but using the multi-scale group architecture from [5] (Appendix B.1) to extract features from a single point cloud; uses three set abstraction layers followed by three fully connected layers with output sizes $(512,256,k)$.

• •

  MLP: A regressor with a fully connected network that also operates on a single frame; uses five 2D convolutional layers with output sizes $(64,64,64,128,1024)$, a symmetric max pooling layer, two fully connected layers with output sizes $(512,256)$, a dropout layer, and a final fully connected layer with $k$ outputs.

Real point cloud sequences suffer from partial observability, self-occlusion, and noise. To minimize the gap between simulation and reality, we aim to generate a training dataset for each of the regressor methods that is as realistic as possible. As training data for the regressors (MeteorNet, PointNet++, MLP), we initialize $N$ simulations with uniformly sampled parameters and record the resulting point cloud sequences and ground truth parameters. This yields a dataset $\mathcal{D}=\big{\{}\{\boldsymbol{x}_{t}\}_{t=1}^{T},\boldsymbol{w}\!=\![w_{\mathrm{stiff}},w_{\mathrm{mass}}]\big{\}}_{i=1}^{N}$. For each simulation run, we generate a point cloud sequence from mesh states according to Equation 3, but bias the sampling of points to be on a subset instead of all faces of the deformable to mimic a partial point cloud with occlusion. Then, we apply random Gaussian noise to this point cloud to mimic the effect of real sensor noise. We generate a dataset of 1500 training and 375 test point cloud sequences with 3,500 points per frame across methods. For comparison, MeteorNet was trained on a dataset of 576 examples and the PointNet++ [5] literature uses thousands of examples.

We train the MeteorNet regressor to learn a mapping $f:\{\boldsymbol{x}\}_{i=1}^{T}\rightarrow\boldsymbol{w}$, which maps an input point cloud sequence $\{\boldsymbol{x}_{t}\}_{t=1}^{T}$ to simulator parameters $\boldsymbol{w}$ that would generate the observed behavior. PointNet++ and MLP methods learn a function $g:\boldsymbol{x}_{t}\rightarrow\boldsymbol{w}$ mapping only a single frame $t$ (selected from a sequence) to the material parameters. In practice, we choose the frame $t$ to be the same frame for which we compute the unidirectional Chamfer distance in DiffCloud optimization. We train each network using $L_{1}$ loss between the predicted and ground truth parameters, using the Adam optimizer with a learning rate of 0.001 for 100 epochs. We run dataset generation on an Intel i5-9400F CPU. We use an NVIDIA GeForce GTX 1070 GPU for training and optimization.

DiffCloud is our proposed approach described in Section III. For all simulations we use DiffSim [36] – a differentiable version of the garment simulator ARCSim [46]. As in [27], we initialize the deformable, in this case cloth, to a basis material and learn two scalar multipliers $(w_{\mathrm{stiff}},w_{\mathrm{mass}})$ for the stiffness and mass tensors. These multipliers represent the simulation parameters we aim to learn. We initialize them to the midpoint of each parameter range. We empirically determine this range as $[0.1,10]$ simulation units for all parameters, so that forward simulation is numerically stable. We want to encourage small changes in the learnable parameters to make a visually compelling difference in deformation. Thus, instead of directly optimizing $(w_{\mathrm{stiff}},w_{\mathrm{mass}})$, we optimize intermediate variables $(s_{\mathrm{stiff}},s_{\mathrm{mass}})$, where $(w_{\mathrm{stiff}},w_{\mathrm{mass}})=\sigma(s_{\mathrm{stiff}},s_{\mathrm{mass}})\times(10-0.1)+0.1$, and $\sigma(x)=\frac{1}{1+e^{-x}}$ is the standard sigmoid function. Intuitively, this maps $(s_{\mathrm{stiff}},s_{\mathrm{mass}})$ first to the range $[0,1]$ via the sigmoid function, and then interpolates this value within the parameter range of $[0.1,10]$. We choose the final parameters to be those that incurred the lowest loss during optimization.

Alignment: For quantitative evaluation we use unidirectional Chamfer distance (Equation 2) that characterizes how well the motions of the simulated and real point clouds align. For each of the baselines, we first infer the predicted parameters using the target point cloud trajectory as input. These parameters serve as input to the simulation engine. We then run the simulator using the inferred parameters and generate point clouds from the simulated meshes. We compute the Chamfer distance between the point clouds generated by the baselines and the real point cloud, then compare this against the loss from running DiffCloud on the real point cloud.

Efficiency: Aside from alignment, we also evaluate the computational resources of all methods. Across experiments, we report (1) the time it takes to run DiffCloud optimization on trajectories and (2) the combined dataset generation, training, and inference times for the supervised baselines.

We aim to infer cloth stiffness and mass from the lift and fold scenarios discussed in Section IV-D.