Surface Reconstruction from Point Clouds via Grid-based Intersection Prediction

In this section, we will initially present the complete pipeline of our method. Subsequently, we will delve into the prediction of intersection points and the pair-based sampling technique. Following that, we will discuss the two distinct components: predicting the relative sign and the intersection point individually. Lastly, we will elucidate the training loss.

The total pipeline is shown in Fig. 4. The network architecture is depicted in a straightforward manner, as illustrated in Fig. 5. Initially, the input point cloud with a total of $N$ points is inputted into the encoder to extract features with a dimension of $N*256$. The encoder comprises 4 transformer layers(Zhao et al., 2021, 2020), outputting dimensions of $64$, $256$, $256$, and $256$ respectively. Following a similar approach to GIFS (Ye et al., 2022), we adopt a paired query point sampling method, generating a total of $M$ pairs. Subsequently, an interpolation layer is employed to derive the features of the $2M$ query points. The network then diverges into two distinct paths. In the upper path, the focus is on predicting the relative sign of the two points. The feature of the point pair is fed into the Relative Sign Module to capture the relationship between the points. A multi-layer perceptron (MLP) is utilized to predict whether the two points exhibit the same sign, with a binary cross-entropy loss function applied. In the bottom path, the aim is to predict the position of the intersection point. The feature of the point pair is inputted into the Intersection Module, followed by the use of a MLP to predict the relative distance from the starting point to the intersection point. A simple regression loss is employed for this prediction. However, in cases where no intersection occurs between the cube edge and the surface, this term is disregarded in the loss calculation.

To sample point pairs for training and testing, we start by considering a point cloud $\mathcal{P}=\{p_{1},p_{2},\cdots,p_{N}\}$. During training, we uniformly sample points around $\mathcal{P}$ at varying scales and ratios. For each point $p_{i}$, a query point $q=p_{i}+U(-1,1)\times s$ is randomly sampled, where $s$ takes values of 0.005, 0.01, and 0.02, corresponding to sample ratios of 0.6, 0.3, and 0.1 respectively. Subsequently, a cube with an edge length of 1/256 and $q$ as the upper-left-front corner is generated, allowing us to sample 12 point pairs corresponding to the 12 edges of the cube. Following this, we determine the intersection point of each edge with the surface and assess whether the point pairs lie on the same side of the surface.

During testing, to reconstruct the entire surface within the given space, we begin by identifying the bounding box of the shape and dividing it into cube meshes with an edge length of 1/256. For each cube within the bounding box, we predict the relative sign of the cube’s corners and the intersection positions, where the reference point is the vertex at the upper-left-front corner of the cube. Utilizing the template matching method in Marching Cubes, we can then generate the triangle mesh. After sampling, we will elaborate on the specific predictions made by our network. Illustrated in Fig. 5, the lower pathway involves selecting a point pair AB, with point $A$ designated as the starting point and $B$ as the end point. The intersection point is denoted as $P$, and our aim is to forecast the proportionate distance from $A$ to $P$, expressed as $\frac{|AP|}{|AB|}$. Furthermore, as depicted in Fig. 5, within the upper pathway, we predict whether the point pairs lie on the same or different sides of the surface. The Relative Sign Module’s name suggests that we solely predict the relationship between the corners of the cube and the upper-left-front corner within the cube. It is not the sign in the global view but a relative sign to the upper-left-front corner.

In the Relative Sign Module, we formalize the problem as follows: given a start point $A$ and an end point $B$ with features $f_{A}$ and $f_{B}$ respectively, we denote the Relative Sign Module as $G(f_{A},f_{B})$. The Relative Sign Module $G(f_{A},f_{B})$ is required to exhibit the following property: when the positions of $A$ and $B$ are exchanged, $G(f_{A},f_{B})$ should vanish, i.e., $G(f_{A},f_{B})=G(f_{B},f_{A})$. This is because the relationship between $A$ and $B$ remains unchanged when their positions are swapped. To maintain this property and enhance the expressive capacity of $G(\cdot,\cdot)$, we design the Relative Sign Module as follows:

where $pos$ is the positional encoding(Tancik et al., 2020) of $A-B$. Here, in order to keep the property $G(f_{A},f_{B})=G(f_{B},f_{A})$, we abandon the traditional sin-cos positional encoding and adopt only cosine positional encoding. Specifically, $pos(A,B)=cos(\frac{A-B}{2^{i}})$, $i\in\{0,2,\cdots,19\}$.

In the Intersection Module, we aim to predict the position of the intersection point $P$ between point pair $A$ and $B$, each associated with features $f_{A}$ and $f_{B}$ respectively. The position of $P$ is represented by $\alpha\in[0,1]$, where $\alpha=\frac{|PA|}{|AB|}$. To ensure the continuity of the surface, we define the Intersection Module as $H(f_{A},f_{B})$, which should satisfy the property $H(f_{A},f_{B})=1-H(f_{B},f_{A})$. This condition is crucial for maintaining surface continuity. The relationship can be intuitively understood by observing that $\frac{|PA|}{|AB|}=1-\frac{|PB|}{|AB|}$. Based on this, we formulate the design of the Intersection Module as follows,

where $pos_{2}$ is also the positional encoding of $A-B$. However, in this part, in order to keep the property $H(f_{A},f_{B})=1-H(f_{B},f_{A})$, we adopt only sine positional encoding. Specifically, $pos_{2}(A,B)=sin(\frac{A-B}{2^{i}})$, $i\in\{0,2,\cdots,19\}$.

Next, we can delve into the explanation for why a surface may not exhibit continuity if $H(f_{A},f_{B})\neq 1-H(f_{B},f_{A})$. Referring to Fig. 6, the two cubes represent adjacent cubes in the Marching Cube algorithm. It is evident that $A$ is equivalent to $C$ and $B$ is equivalent to $D$. Pints $P$ and $Q$ must coincide; otherwise, the mesh will lack continuity. Therefore, the relationship $\frac{|AP|}{|AB|}=1-\frac{|DQ|}{|CD|}$ must be upheld. This condition serves as both necessary and sufficient for ensuring $H(f_{A},f_{B})=1-H(f_{B},f_{A})$ and $G(f_{A},f_{B})=G(f_{B},f_{A})$.

In this approach, we utilize two types of loss functions for training. For sign prediction, we employ a straightforward binary cross-entropy loss. For intersection prediction, we apply a regression loss to minimize the disparity between the predicted $\alpha_{j}$ and the ground-truth $\alpha_{j_{GT}}$, as illustrated below:

where $\alpha_{j}$ means the predicted intersection position of $j-th$ point pair, $\alpha_{j_{GT}}$ means the ground-truth intersection position of $j-th$ point pair,

Compared to SDF, UDF does not require predicting signs, making it seem simpler than SDF. However, this is not the case. We designed a small experiment to demonstrate the difficulties of learning UDF, using a simple neural network to fit the circle function on a two-dimensional plane. Our goal is to learn the UDF and SDF of a circle on the plane, using a three-layer MLP (Multi-Layer Perception) to represent this sphere, with optimization done through gradient descent. We provide two three-layer MLP functions $\mathcal{F}_{\theta_{1}}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{+}$ to represent the UDF field on the plane, and $\mathcal{F}_{\theta_{2}}:\mathbb{R}^{3}\rightarrow\mathbb{R}$ to represent the SDF field on the plane. The parameters of the experiment are as follows: the output dimensions of the three-layer MLP are $32,64,1$ respectively, the number of sampled points is $20000$, the initial learning rate is $0.001$, and the optimizer used is Adam. The visualization of the learning results is shown in Fig. 7.

From the results of SDF and UDF in the first row, it can be seen that the neural network indeed learns the approximate distance field. When the distance field changes along the x-axis at the position of $y=0$ in the second row, we focus on the error between the GT distance field and the learned distance field to the circle as x changes. It can be observed that when learning SDF, the error between the two is almost zero, while when learning UDF, the error is significant. This is because near the surface of the curve, the gradient of the SDF remains unchanged, resulting in a continuous change in the distance field, while the gradient of the UDF rotates by $180^{\circ}$, making the change in the distance field discontinuous. Specifically, when x approaches $-1$ from the left side, the gradient of the SDF is

When x approaches -1 from the right, the gradient of the SDF is

They are same with each other in SDF. when x approaches $-1$ from the left side, the gradient of the UDF is

When x approaches -1 from the right, the gradient of the UDF is

The two are exactly opposite. Although the Signed Distance Function (SDF) also has some errors at $x=0$ due to the collision of gradient directions, it is insignificant because it occurs at the center of the sphere, which is not important for surface reconstruction. Therefore, the error here can be ignored. However, the Unsigned Distance Function (UDF) has a discontinuity in the vicinity of the surface, and the distance field near the surface is particularly important for surface reconstruction. For regression methods, fitting a continuously changing function is relatively easy, while fitting a function with gradient discontinuities is very difficult. This is the difficulty of reconstruction based on UDF. The difficulty of reconstruction based on SDF is easier to understand, mainly in complex topological surfaces where it is difficult for neural networks to determine the sign information at a specific point.

Our methodology is assessed on three prominent open-source datasets. Initially, we randomly select 1300 shapes from 13 distinct categories within the ShapeNet dataset (Chang et al., 2015), with each category comprising 100 shapes. The point number of each input point cloud is 3000. Subsequently, we employ the MGN dataset, encompassing both pants and shirts. The point number of each input point cloud is 3000. Finally, we include 100 scenes from the real-world ScanNet dataset (Dai et al., 2017), obtained through scanning technology, resulting in 10,000 input point clouds. To facilitate a comprehensive comparison with existing techniques, we utilize standard evaluation metrics commonly employed in surface reconstruction. One such metric is Chamfer-$L_{1}$ ($CD_{1}$), which measures the distance between two sets of points. Additionally, to demonstrate enhancements in artifact elimination, we utilize normal consistency ($NC$) as a metric to evaluate the smoothness of the reconstructed surface.

When calculating $CD_{1}$, we follow O-Net(Mescheder et al., 2019) and randomly sample 100000 points on the reconstructed mesh and 100000 points on ground-truth mesh. $CD_{1}$ equation is shown as the Eq.10

$\{\mathbf{x}_{i},i=0\cdots N_{x}\}$ is the points set sampled from ground-truth mesh. $\{\mathbf{y}_{i},i=0\cdots N_{y}\}$ is the point set sampled from reconstructed mesh. $\mathcal{S}_{y}(\mathbf{x}_{i})$ means the nearest point to $\mathbf{x}_{i}$ in reconstructed point set. $\mathcal{S}_{x}(\mathbf{y}_{i})$ means the nearest point to $\mathbf{y}_{i}$ in ground-truth point set. $\|\cdot\|_{1}$ means $L1-distance$.

Normal consistency is closely related to $CD_{1}$, shown in Eq.11

$\mathbf{n}(x)$ means the normal of point $x$. $<,>$ means inner-product. Here, we calculate the absolute value of inner product, because our mesh is not orientated.

Several previous studies (Park et al., 2019; Peng et al., 2020; Mescheder et al., 2019; Chibane et al., 2020b) have commonly utilized the watertight surface reconstruction approach for implicit surface reconstruction, a key focus of this point cloud reconstruction research. In order to evaluate the effectiveness of our proposed method, we conducted a comprehensive analysis of watertight surface reconstruction using the ShapeNet dataset (Chang et al., 2015). Specifically, we uniformly sampled 3000 points from the watertight mesh to serve as input data. We compared our method against four well-known techniques, excluding PSR (Kazhdan et al., 2013), all of which are UDF-based methods. The methods included GIFS (Ye et al., 2022), CAP (Zhou et al., 2022), and SuperUDF (Tian et al., 2023). For the PSR (Kazhdan et al., 2013) method, we incorporated ground-truth normals as input data. The visualization results are presented in Fig. 8 and Fig.  9. Our method not only preserves more details but also produces smoother results. One of the reasons for the unsatisfactory performance of CAP (Zhou et al., 2022) is the insufficient number of input points in the point cloud. With only 3000 points available for training, CAP typically requires over 20000 points in the point cloud to achieve optimal results. Typically, methods such as ConvOccNet(Peng et al., 2020) and POCO(Boulch and Marlet, 2022) initially learn features from sparse point clouds and then decode the distance field of query points. Conversely, approaches like Neural Pull(Ma et al., 2020) and CAP(Zhou et al., 2022) train neural networks to approximate the distance field, often necessitating dense point clouds. Our method falls within the former category.

The quantitative evaluation results are summarized in Table 1. These results demonstrate that our method achieves excellent performance across all 13 classes with a significant margin, showcasing its state-of-the-art capabilities. Particularly noteworthy is the performance on the $NC$, indicating that our reconstructed surfaces are not only faithful to the ground truth but also exhibit smoothness.

By leveraging a pair-point structure akin to GIFS for representing surfaces within our framework, we are able to proficiently reconstruct open surfaces. We achieve this by uniformly sampling 3000 points on each MGN mesh to reconstruct the implicit surface, a task that poses challenges for SDF-based methods given the characteristics of open surfaces. The comparative analysis presented in Table 2 illustrates the superior performance of our approach in comparison to various methods including PSR (Kazhdan et al., 2013), CAP (Zhou et al., 2022), GIFS (Ye et al., 2022), and SuperUDF (Tian et al., 2023). Moreover, the visual results depicted in Fig.  10 and Fig.  11. exhibit the faithfulness of our reconstructed mesh, capturing intricate details and closely mirroring the ground-truth.

Real scene reconstruction is characterized by a heightened level of complexity, primarily due to the inherent openness and incompleteness of the input point cloud data. In this study, we meticulously selected 100 scenes from ScanNet and illustrated the visualization results in Fig. 13 and Fig. 12, along with the quantitative results in Table 3, in comparison with other methods.

Upon examining the visualization results, our reconstructed surface exhibits a notably smoother appearance compared to other methods. Furthermore, the quantitative analysis reveals that our approach outperforms other methods in terms of both $CD_{1}$ and $NC$, underscoring the efficacy of our design in accurately predicting the intersection points between the surface and cube edges.