Extending DeepSDF for automatic 3D shape retrieval and similarity transform estimation.

Aligning 3D shapes is a well studied problem with varying solutions depending on the representations of the 3D shape. For example, when the geometry of a 3D object is represented using pointcloud, solutions for estimating rigid-body and orientation-preserving similarity transformations have been proposed [17, 18, 3, 49, 47].

Alternatively, and more pertinent to our work, 3D shapes can be aligned using signed distance functions (SDFs). [42] provided a good overview of some of these methods. For example, [40, 7] presented techniques to estimate rigid-body transformations between two shapes represented using SDFs. [27, 10] further tackled the problem of estimating rigid-body transformation and scale using SDFs, the same type of transformation studied in this paper. Their algorithms made use of geometric moments and the Fourier transform. In comparison, we develop a gradient-descent based algorithm to find a solution to an optimization formulation. Similarly, [28, 19] also made use of gradient-descent algorithms on optimization problems to estimate similarity transformations between SDFs. Yet, none of the works above explicitly allow for solving shape retrieval and transformation estimation in one optimization problem. Rather, they only deal with transformation estimation given the shape model.

There exists a body of work that explicitly deals with shape retrieval. Classical shape retrieval algorithms such as [38, 16, 44] usually follow a two-step approach: The first step involves shape recognition from a database by making use of shape descriptors such as [45, 35, 2]. The second step involves a correspondence search using 3D keypoints and descriptors followed by registration. It has been argued in [39] that dense alignment via SDFs can produce much better results since it does not make use of a correspondence search which may introduce errors. Our work favors this approach. We will compare our superior results against classical 3D feature based retrieval techniques.

More recent work has shifted from using hand-crafted descriptors to data driven ones such as [50, 12]. Alternative data driven approaches [31] make use of deep neural networks (DNNs) to replace the recognition module entirely. However, the sequential approach of recognition and then registration does not allow for corrections of errors in the registration phase caused by a mismatch during the recognition stage. To address this problem, [5] proposed an end-to-end shape retrieval and alignment module. In comparison, although our work also jointly optimizes over the space of shapes and transforms, the formulation is not a feed-forward neural network but an optimization problem that makes use of a neural network model. The main benefit of this is that the optimization process can be viewed as introducing feedback that allows iterative minimization of errors from predictions made by the network. A feed-forward module can only make a prediction once per input and has no mechanism to minimize errors after training. More importantly, the space of shapes over which we optimize allows us to retrieve objects that were not seen during training.

Assume that we are given a set of shapes $\mathbb{S}=\{S_{1},S_{2},S_{3},\ldots,S_{n}\}$ belonging to the same class, all in the canonical scale and pose (e.g., a set of chairs all normalized to lie in the unit sphere with forward directions aligned with the $+y$ axis). Further assume the shapes are represented as meshes although our approach also works for other representations as long as we can extract a signed distance function from the representation.

Then, for a new shape $S$ belonging to the same class but not necessarily in $\mathbb{S}$ or in the same canonical scale and pose, we would like to firstly build a dataset $\mathbb{D}$ of shapes using $\mathbb{S}$, and secondly find an item in $\mathbb{D}$ and a corresponding similarity transformation that best approximates $S$.

Note that we do not enforce $\mathbb{D}$ = $\mathbb{S}$, but only require that $\mathbb{D}$ is constructed using only $\mathbb{S}$. Typical shape retrieval problems tend to set $\mathbb{D}$ = $\mathbb{S}$. However, as we will show, this may restrict the richness of $\mathbb{D}$.

Our approach assumes that all given shapes have an associated SDF. This can be easily obtained, for example, from a triangle mesh representation of the shape.¹¹1Please refer to the supplementary for more details on SDF extraction and sampling.

As shown in Fig. 1, the input to our algorithm is a query shape $S$ and a set of canonical shapes $\mathbb{S}$. Its output are the optimal shape representation in canonical pose and scale from $\mathbb{D}$ as a mesh, the optimal similarity transformation, and a compact representation of the query shape for data compression.

A 3D similarity transformation $g\in Sim(3)$ is a tuple $(s,R,t)$, with $s\in\mathbb{R}^{+}$, $R\in SO(3)$, $t\in\mathbb{R}^{3}$, and $R$ can be written as a $3\times 3$ real matrix. Then, $g$ acting on a point $x\in\mathbb{R}^{3}$ is given by

Let $S$ be a solid 3D shape with a closed surface such that we have a well-defined notion of inside and outside. In addition, let $\partial\Omega_{S}^{\phantom{-}}$ represent the set of points on the surface of $S$, $\Omega_{S}^{\phantom{-}}$ represent the set of points on the surface and interior of $S$, and $\Omega_{S}^{c}$ the complement of $\Omega_{S}^{\phantom{-}}$. Then an SDF is an implicit representation of the shape defined as: $\Phi(x,\Omega_{S}^{\phantom{-}})=\begin{cases}-\underset{y\in\partial\Omega_{S}^{\phantom{-}}}{\text{ min }}||x-y||_{2}\hskip 5.0pt\text{, if }x\in\Omega_{S}^{\phantom{-}};\\ \underset{y\in\partial\Omega_{S}^{\phantom{-}}}{\text{ min }}||x-y||_{2}\hskip 5.0pt\text{, if }x\in\Omega_{S}^{c}.\end{cases}$

Intuitively, for a fixed shape $S$, its SDF at a point $x$ tells us the distance from $x$ to the surface of $S$, with the sign determined by whether $x$ lies in the interior of $S$.

Let $S,S^{{}^{\prime}}$ be two shapes related by a similarity transformation $g$ as defined in (1), i.e., $\partial\Omega_{S^{{}^{\prime}}}=\{g(x)|x\in\partial\Omega_{S}^{\phantom{-}}\}$. Under this condition, it is well known [28] that

In other words, SDFs are invariant to rotations and translations, but vary proportionally with isotropic scaling. The constant of proportionality is the scaling factor $s$.

Our proposed approach first learns a dataset $\mathbb{D}$ from the input set of shapes $\mathbb{S}$ and then formulates and solves an optimization problem to jointly estimate the transformation and shape parameters. Inspired by [29], we also learn a generative model and latent space of shapes. The benefit to this is that the latent space is more expressive than the input set of shapes $\mathbb{S}$. Specifically, the latent space has been shown in [29] to extrapolate the shapes in $\mathbb{S}$ and produce new shapes of the same class but not contained in $\mathbb{S}$. The generative model also allows us to formulate searching over the latent space as a tractable optimization problem. In addition, since the dimension of the latent space is relatively small and a similarity transformation involves only a few parameters, we can simultaneously obtain a compact representation of the query shape.

We have implemented JSRTE algorithm using PyTorch [30] and report some implementation details here for reproducibility. Specifically, the matrix exponential $\mbox{exp}(\hat{\omega}\theta)$ is expressed by Rodrigues’ formula [34, 26]: $\exp(\hat{\omega}\theta)=I+\hat{\omega}sin(\theta)+\hat{\omega}^{2}(1-cos(\theta)).$ Adam[23] as implemented in [30] is adopted to solve the optimization problem (6). To ensure that the search is in the feasible set of the problem, $s\in[\underaccent{\bar}{s},\bar{s}]$ can be restricted as $\underaccent{\bar}{s}=0.01,\bar{s}=10$. $\hat{f}$ is learned using open source code provided by [29, 13], with latent vectors in $\mathbb{R}^{256}$. For all objects, we have trained on subsets of the corresponding Shapenet [8] class. To encourage convergence of the network, [29] has used a penalty term on the norm of $z$ weighed by $10^{-4}$. We also include this term in solving (6). We generally make use of the same SDF extraction and sampling approach used in [29] (see supplementary for detail).

Gradient-descent based methods are susceptible to converging to locally optimal solutions. So it is important to find good initialization points for the variables. For this experiment, we want to observe the behavior of our algorithm when initialized within reasonable bounds of the groundtruth.

We sample objects from the chair, table, sofa, and bed categories from ShapeNet [8]. First, 30 objects from each category are sampled. Then for each object a random scale and pose are assigned by sampling the groundtruth parameters, $s\sim\mbox{Uniform}([\frac{1}{2},2))$ , $\psi,\rho,\theta,\sim\mbox{Uniform}([-\pi,\pi))$, and $||t||_{2}\sim\mbox{Uniform}([0,4))$. The direction of $t$ is chosen uniformly from the surface of a unit sphere. For each shape and transform pair, we solve the shape retrieval problem using our algorithm 50 times, with each trial using a random initialization. We carry out two sub-experiments to highlight scenarios where our algorithm is applicable. In the first scenario, we assume that we know the axis of rotation (and consequently $\psi,\rho$), but all other parameters are unknown. This is a reasonable assumption since many objects lie upright on flat surfaces such as floors or tables. One may use the normal to the surface as the axis of rotation. In the second scenario, we relax this assumption and introduce some uncertainty in the axis of rotation. The scenarios are used to randomly sample the initialization for our algorithm.

Concretely, we refer to the initialization parameters as $s^{0}=s(1+\Delta s),\psi^{0}=\psi+\Delta\psi,\rho^{0}=\rho+\Delta\rho,\theta^{0}=\theta+\Delta\theta,t^{0}=t+\Delta t$. The $\Delta$ variables correspond to the difference between the ground-truth and initialization parameters. We then sample the $\Delta$ variables uniformly from ranges determined by each scenario. Table 1 below summarizes these ranges for the different scenarios.

In all scenarios, We choose the direction of $\Delta t$ uniformly from the surface of a unit sphere. $z^{0}\sim\mbox{Normal}(\mathbf{0},\sigma\mathbf{I}),\sigma=0.01$. We use the corresponding ShapeNet [8] model class as input shape set $\mathbb{S}$.

To quantitatively measure the performance of our algorithm, we use the F-score between the output shape (transformed by our predicted transform parameters) and the test shape as recommended in [43]. The F-score measures the similarity between 3D surfaces as the harmonic mean between precision and recall. A point on the predicted surface is considered a true positive if it is within a threshold $\epsilon$ of the groundtruth surface. In this work, we have set this threshold as $5\%$ of the scale of the groundtruth object. We refer to this score as F@$5\%$. The F-score ranges from $0$ to $1$, with higher numbers indicating more closely matched shapes.

For each shape and transform pair, we categorize the F-score into bins and count the number of initializations out of 50 that fall into each bin. Then for each bin, we report aggregate statistics using box plots in Fig. 2. The position of each box-plot on the x-axis corresponds to the rightmost edge of its bin. The leftmost edge of its bin is the previous point on the x-axis.

Fig. 2 corresponds to scenarios with a known and unknown axis of rotation respectively. In both scenarios, we observe that our algorithm performs well with most F@$5\%$ scores greater than 0.8. We also observe a slight degradation in performance in the scenario with an unknown axis of rotation, as is to be expected due to more uncertainty. Intuitively, the results imply that with a known axis of rotation, if we know the amount of rotation up to $\pm 20^{\circ}$, the translation up to $\pm 15$ cm and the scale up to $30\%$ of the true scale, we can expect our algorithm to perform very well. We also perform more experiments to test the our algorithm in other parameter ranges and report those as well as qualitative results in the supplementary.

In the next set of experiments, we evaluate our algorithm on real data. In this paper, we demonstrate the results on the chair subset of the Redwood dataset [9]. The Redwood dataset contains scans (represented as triangle meshes) of real objects in varying scales and poses. We have manually segmented the dataset so that we remove other objects in the background, leaving behind only the floor and the main object in each scene. We assume that all objects in the scene lie on the floor and are upright. This allows us to obtain an estimate for $\omega$ as the normal to the floor plane. In this way, we collect 89 shapes and samples of their SDFs. However, we use 31 randomly selected shapes from these 89 for validating the baselines described below, leaving only 58 for testing. We discarded all other scenes that could not be segmented properly. For each shape, we initialize $s$ using the radius of its bounding sphere and $t$ using the center of the bounding sphere. We estimate $\omega$ using the normal vector to the floor plane and $z$ is initialized by drawing from $\mbox{Normal}(\mathbf{0},\sigma\mathbf{I}),\sigma=0.01$. The only parameter left is $\theta$. For $\theta$ we grid up the space in increments of $30^{\circ}$ and run our algorithm for each value of $\theta^{0}\in[0,\frac{\pi}{6},\frac{\pi}{12},\ldots 2\pi)$. We then select the best result from each of these initialization of $\theta$ using the F@$5\%$ score.

We compare our work to three different approaches for shape retrieval and transform estimation. In these approaches, we perform recognition to find the closest shape to the query shape in the set of input shapes and then perform registration to estimate a similarity transform.²²2 We are aware of other state-of-the-art SDF based methods for jointly estimating shape and similarity transformation [5]. However, we could not find a publicly available implementation to use for testing.

The first approach uses Harris3D keypoints and the SHOT[37] descriptor to perform recognition and provide correspondences for registration. Registration is then carried out using [47]. Specifically, for recognition, we describe each shape in the dataset using the descriptor and keypoint detecting method stated above. During test time, we compare the keypoints and descriptors of the query shape to those in the dataset. This method is referred to as Harris3D+SHOT.

The second approach replaces the keypoint detector and descriptor with the shape descriptor developed in [2]. For registration, points are sampled from the mesh and then applied to an Iterative Closest Point (ICP)[6] variant that makes use of [47] for transform estimation. Correspondences are assigned by solving an optimal assignment problem. We find that this significantly improves the result. This method is referred to as OURCVFH+ICP.

The last approach uses PointNet[33] as described in [31] for recognition and the ICP method described above for registration. This method is referred to as PointNet+ICP.

We report the average F@$5\%$ score of these methods in Table 2, which shows our algorithm significantly outperforms the three alternative approaches. We also provide qualitative results of randomly sampled test cases in Fig. 3, which shows that the shapes predicted by our method are qualitatively better than the alternative methods, with much less misalignment. Alternative methods using a two-step pipeline of recognition and registration can propagate errors in the recognition stage that cannot be corrected by just a similarity transformation in the registration stage. In addition, our method uses a richer search space for shapes by using the latent space as its dataset, rather than just the set of input shapes provided. Moreover, finding good correspondences can be difficult for other two-step approaches as clean CAD models and noisy real scans have differing local surface properties. Finally, the need to sample the surface to allow tractable registration may introduce errors in the sense that true corresponding pairs may not belong to the sampled sets. Our method on the other hand does not make explicit use of correspondences.

Finally, we conduct an experiment to explore the viability of our approach as a form of 3D data compression. Given the neural network parameters of $\hat{f}$, we can choose to store only the latent vector representing a shape and the associated similarity transform. With access to $\hat{f}$ and the transform parameters, one can then decode the latent vector and convert it back to a mesh model. In our experiment, the network size is approximately 7MB. This is a constant cost shared across many uses and is consequently amortized.

For this experiment, we save each mesh in our Redwood test set under two mesh simplification schemes, mesh simplification using Quadric Error Metrics[14, 51] and Vertex Clustering [51]. We have varied the parameters of each method. For mesh simplification using Quadric Error Metrics, we set the target number of faces to be reduced by a factor of 100 and 1000 from the number of faces in the original mesh. We refer to these experiments as QE-100, QE-1000 respectively. For Vertex Clustering, we vary the size of the grid cells used for clustering to be a factor of 0.1 and 0.2 of the radius of the bounding sphere for the shape. We refer to these as VC-0.1, VC-0.2 respectively. We save all meshes and compute the average size of the files as well as the mean F-score (compared to the original mesh) to give an indication of the representation power at each simplification parameter. We compare these scores against our mean F-score and the cost of saving the latent vector and transformation parameters as a $4\times 4$ matrix and report the results in Table 3. The result shows that our method provides an impressive tradeoff between reconstruction quality and storage size.

We perform additional experiments making use of synthetic data to observe the behavior of our algorithm under varying initialization conditions. Following the same procedure described in Section 4.1 of the main paper, we conduct two additional experiments that vary the initialization parameters as follows:

In the first scenario in Table 4, we assume a known axis a rotation, but increase the range of values that the initialization parameters can take on compared to those presented in the main paper. In the second scenario, we relax the assumption of a known axis of rotation and introduce some uncertainty. We retain the same increase in the ranges of the other parameters. The purpose of this experiment is to see how the performance degrades when we introduce more uncertainty.

As shown in Fig. 4, we observe that both scenarios perform worse than those presented in the main paper. We also observe that for most of the experiments the median F-score is still greater than 0.8. The difference here is that we observe significantly more F-scores in the $[0.5,0.7]$ range.

Figures LABEL:fig:supp_redwood_1 - LABEL:fig:supp_redwood_6 provide qualitative results from all the experiments carried out on the Redwood dataset [9] detailed in Section 4.2 of the main paper. We have displayed comparisons to the benchmark making use of PointNet [33] and a variant of ICP [6] described in the main paper. We report a similar level of performance as that presented in the main paper. Our algorithm consistently provides qualitatively better results with fewer shape mismatch and misalignments. The figures also indicate that our algorithm sometimes struggles with fine details and star shaped legs. This is partly due to poor convergence of the SDF model during training and also because in reality the star-shaped legs are free to rotate separately from the main body of the chair. These additional degrees of freedom are not well captured by our assumptions.

For experiments using synthetic data (Section 4.1 of the main paper), we follow the same sampling procedure described in [29]. For experiments using the Redwood dataset [9] (Section 4.2 of the main paper), we first preprocess the data by manually segmenting the floor and main object in each scene. We then make use of the plane fitting algorithm provided by [51] to segment the main object and the floor. We also obtain the axis of rotation in this fashion as the normal to the floor plane. Furthermore, we make use of the hidden point removal algorithm described in [22] and implemented in [51] to remove artifacts. Other artifacts and noisy points are also removed using the statistical outlier removal algorithm provided by [51].

Finally, for each point in the preprocessed mesh, we sample new points at a distance of $\pm 0.01$m in the direction of the normal to the surface at each point. We estimate the SDF at these new points as $\pm 0.01$ respectively. To obtain SDF samples representing the freespace, we take each point on the mesh and randomly and uniformly sample a new point between $0.07$m and $0.20$m away from the point on the mesh, in the direction of the outward pointing normal. We compute the SDF at these new points by finding the approximate distance to the closest point on the surface of the mesh. The sign is then determined by the whether each new point is in the direction of the outward pointing normal at its nearest neighbor on the mesh. We use a k-d tree and the set of vertices on the densely sampled meshes to approximate the nearest neighbors. We obtain 25000 freespace SDF samples in this manner.

The SDF samples used during optimization are composed of the freespace SDF as well as the SDF sampled at distances $\pm 0.01$m from points on the mesh.

To solve the proposed optimization problem, we employ Adam [23] as implemented in [30]. We follow a stochastic gradient-descent approach and make use of 8000 SDF samples at each iteration. The learning rate is set to $0.05$ for all parameters except for those assumed to be known. The learning rate is decreased by a factor of $5$ every 400 iterations. We solve the optimization problem for 800 iterations.

We report some implementation details for the benchmarks used for experiments on the Redwood dataset [9] presented in the main paper.