Multiview Textured Mesh Recovery by Differentiable Rendering

Multi-View Stereo methods try to estimate the depth map by matching feature points across different views. In general, most of them assume that the appearances of a surface point are consistent in all visible views [2, 3, 15, 16]. They evaluate the matching cost of image patches for all depth hypotheses in order to find the best one. Li et al. [17] propose a marker-less shape and motion capture approach for multi-view video sequences. 3D coordinates can be obtained by triangulating the correspondences through matching the image patches across all region of interests uniformly sampled or propagated from neighboring pixels and adjacent views [1, 18]. Depth fusion and implicit surface reconstruction [4] are required to extract the watertight mesh from point clouds, where the surface details may be smoothed due to depth fusion. Moreover, these meshes are usually texture-less. Although the color can be estimated from the input images using camera projection, the rendered images are not fidelity enough to be viewed from free perspective.

Recently, the learning-based MVS methods have received a bit attentions [19, 20, 21]. DeepMVS [19] reprojects images onto 3D plane-sweeping volumes and performs intra-volume aggregation, where the hand-crafted features are replaced by deep ones. MVSNet [20] warps deep features into the reference camera frustum to build a cost volume via differentiable homographies. R-MVSNet [21] replaces the 3D CNNs regularization module with 2D recurrent network. Despite these promising results, it is still difficult to build the correspondences in those texture-less pixels or non-Lambertian regions. The reliability of the estimated depth map is also important for practical applications. Su et al. [22] propose an uncertainty-guided network and an uncertainty-aware loss to perceive the uncertainty in an unsupervised manner.

In order to obtain the fine detailed results, a coarse-to-fine framework is often used. Zhang et al. [23] propose a coarse-to-fine framework to integrate pixel-wise occlusion information, which significantly improves the depth accuracy. Zhang et al. [24] and He et al. [25] incorporate the coarse-to-fine strategy to enable the network to estimate the high-resolution depth maps. Liu et al. [26] propose a coarse-to-fine differentiable sphere tracing to reconstruct 3D shapes from multi-view images. Liu et al. [27] propose a coarse-to-fine reconstruction framework for point cloud upsampling. These methods employ the coarse-to-fine strategy to reduce computation cost. Coarse module results are sent to fine one as input to get the detailed results. Differently from these coarse-to-fine methods, we suggest a coarse-to-fine strategy to greatly reduce optimization time. In our proposed approach, the coarse module results are upsampled and optimized in fine module.

By taking advantage of differentiable renderer [9, 11],  [8, 6, 7] try to estimate the object’s intrinsic color and geometry by inverse rendering, where the gradients can be backpropagated to geometry. Therefore, the objects’ geometry and appearance can be recovered by minimizing the differences between the synthesized photo and input image.

Rasterization is the key operation in rendering pipeline. Liu et al. [8] propose a differentiable renderer for image-based shape fitting. The rendering is treated as a differentiable aggregating process that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Wang et al. [28] present a differentiable renderer for point cloud that is considered as a disk. Moreover, the discontinuities are approximated by a linear function. Zhang et al. [29] propose a progressive point cloud upsampling framework supervised by the point-based differentiable rendering. Jiang et al. [10] propose a differentiable renderer for shape optimization using signed distance functions (SDFs), where the gradients can be backpropagated to the whole SDF grid by regularization.

As the soft rasterization operation is an approximate solution, which does not provide the gradient with respect to variables other than pixel coordinates. This may lead to the inferior reconstruction results. Li et al. [30] present a Monte Carlo differentiable renderer that produces the unbiased gradients through edge sampling. Similarly, Luan et al. [7] propose an analysis-by-synthesis pipeline for high-quality geometry reconstruction and spatially varying reflectance. As is well-known, recovering the geometry and appearance from 2D images is a highly under-constrained problem, in which the pixel intensity is affected by lots of factors like lighting, materials, occlusions, and etc. Most of existing inverse rendering methods assume that there is a single light source or the directions of all the light sources are known. This may rarely happen in the real-world applications. In addition, it is difficult to deal with the complex topology changes using the inverse rendering optimization, where the gap between the initial geometry and target cannot be very large. To avoid this issue, MVS is usually employed to obtain a good initial shape.

Implicit neural representation directly estimates the objects’ geometry from the input image through minimizing the photometric loss, which has achieved the encouraging results in 3D reconstructions and view synthesis. Sitzmann et al. [31] employ LSTM to represent the scene by simulating the ray marching process. Mildenhall et al. [32] represent the scene in terms of volume density and view-dependent radiance, where the color of each pixel is obtained by accumulating the grid having the ray passed through. Due to the resolution limitations in volume rendering, the geometry extracted by the marching cube algorithm [5] is usually rough. Instead of using volumetric density, Yariv et al. [12] employ neural networks to represent scenes through SDF and light field implicitly, where color is only calculated at the intersection. Moreover, the two implicit networks are trained by the loss of color and silhouettes. Zhang et al. [33] use mixtures of spherical Gaussians to represent the bidirectional reflectance distribution function (BRDF) and environmental illumination for physically-based rendering. Zhang et al. [13] leverage the feature consistency in stereo matching to fit the implicit surface. Oechsle et al. [34] make use of both surface rendering and volume representation, which obtain the accurate reconstruction results without masks. Although the implicit representation has the merits of watertight and topology-agnostic, it tends to produce the smoothed surface. Due to the implicit representation using neural networks, it takes long time to converge. Since the color value of every pixel needs to be calculated by a neural network, it is quite computationally expensive to render an image.

Recently, Peng et al. [14] propose a differentiable Poisson Solver to represent mesh with the oriented point clouds. The differentiable Poisson Solver bridges the explicit 3D point representation with 3D mesh via the implicit indicator field. In contrast to neural implicit representation of MLP, the oriented point cloud is more interpretable and lightweight, which accelerates inference time by an order of magnitude. Comparing to those explicit representations like patches or meshes, the oriented point cloud can produce topology-agnostic and watertight manifold surfaces through differentiable Poisson Surface Reconstruction. Instead of using the Chamfer distance to estimate the oriented point cloud, we optimize the oriented point cloud with depth loss and photometric loss in a weakly supervised manner. The geometry and materials can be optimized from multi-view images efficiently.

To facilitate the effective textured mesh recovery, we firstly obtain the coarse mesh from multi-view silhouettes. Instead of deforming a sphere like IDR [12], we directly triangulate the visual hull [35] through the marching cube algorithm. Visual hull is the maximal object that gives the same silhouette from any possible viewpoint. Each silhouette forms a cone in its corresponding camera view, and the convex hull of real object’s shape is the intersection of all these visual cones. Obviously, the reconstruction results can be improved with the increasing number of views. Practically, we can get roughly correct result for convex objects. When the object is concave, the results of visual hull are far from the realistic. To tackle this critical problem, we try to recover the accurate textured mesh by taking advantage of multi-view constraints with a differentiable renderer, which is able to deform the coarse mesh to the target geometry.

Specifically, we sample the learnable oriented point clouds from the coarse initial mesh as the shape representation, and a differentiable Poisson Surface Reconstruction [14] is employed to reconstruct the mesh. Then, texture of the optimized mesh is sampled from a learnable texture grid through trilinear interpolation. Thirdly, we rasterize the predicted texture and depth map of the reconstructed mesh via differentiable rendering. Physically-based rendering is performed, and the learnable environment map is used to as the scene lighting. Finally, the textured mesh and environment map are supervised by the original images and MVS depth outputs. Fig. 2 shows the overview of our proposed coarse-to-fine framework.

The overall optimization is supervised by the following loss function,

where $\mathcal{L}_{c}$ is the photometric loss to minimize the difference between the inverse rendered photo and input image. $\mathcal{L}_{s}$ is the silhouette loss for the mask, and $\mathcal{L}_{d}$ is the depth loss. $\lambda_{c}$, $\lambda_{s}$, and $\lambda_{d}$ are the weighting coefficients to balance the different terms.

Instead of using the explicit mesh with vertices and faces or implicit surface representation, we take advantage of a differentiable Poisson Solver (SAP) [14] to represent the shape, which is able to produce topology-agnostic and watertight surfaces. Comparing to MLP representation, it is interpretable, lightweight and fast. The key step in Poisson Surface Reconstruction [4] involves solving the Poisson equation, where an oriented points set can be viewed as samples of the gradient of the underlying implicit indicator function capturing the geometry. SAP solves the Poisson equation using the spectral methods [36]. As the Fast Fourier Transform (FFT) operation is well supported on GPUs, SAP can be implemented very efficiently. The mesh is generated by differentiable marching cube so that the gradients can be effectively approximated by the inverse surface normal [37]. Since all the computations are differentiable, the gradients can be backpropagated to points and normals directly.

To this end, we initialize the oriented point clouds by uniformly sampling the points and normals $P=\{\boldsymbol{x}\in\mathbb{R}^{3},\boldsymbol{n}\in\mathbb{R}^{3}\}$ from the coarse mesh produced by visual hull, which is further used to build the signed distance field (SDF) through Differentiable Poisson Surface Reconstruction ($\mathcal{R}$)

$\Phi$ is an $r\times r\times r$ grid, which represents the SDF obtained by solving the Poisson equation from the oriented point clouds $P$. $r$ is the grid resolution. We first solve for the unnormalized SDF grid $\Phi^{\prime}$ without considering the boundary conditions

where $IFFT$ represents the inverse fast Fourier transform. $\widetilde{g}_{\sigma,r}(\mathbf{u})$ is a Guassian kernel of bandwidth $\sigma$ and grid resolution $r$. $\mathbf{u}:=(u,v,w)$ is the spectral frequencies corresponding to $(x,y,z)$ spatial dimensions respectively. $\odot$ represents element-wise product. The normalized SDF grid $\Phi$ is obtained by subtracting the mean of SDF value at the point clouds $P$ and scaling by SDF value

where $m=0.5$. $\mathbf{x}\in\mathbb{R}^{3}$ denotes a spatial coordinate. For more information, please refer to the original SAP paper [14]. Since the overall computation is fully differentiable, the gradient can be backpropagated to the oriented point clouds efficiently. Then, the topology-agnostic and watertight mesh can be obtained via differentiable marching cube $\mathcal{C}$

where $V\in\mathbb{R}^{n\times 3}$ and $F\in\mathbb{R}^{f\times 3}$ denote the vertices and faces of the mesh $\mathcal{M}$ triangulated from SDF grid, respectively. Since $\mathcal{M}$ is generated by marching cube, the number of vertices $n$ and the number of faces $f$ are indeterminate. The forward inference of $\mathcal{C}$ is the normal Marching cube algorithm. The backward of $\mathcal{C}$ can be decomposed by the chain rule

where $\mathcal{L}$ is the objective function. $\frac{\partial\mathcal{L}}{\partial V}$ can be computed by differentiable rendering, and $\frac{\partial V}{\partial\Phi}$ can be approximated by the inverse surface normal [37]

Once the mesh is reconstructed, the texture corresponding to each vertex in mesh is obtained from a learnable texture grid through trilinear interpolation $\mathcal{F}$

where $V_{tex}\in\mathbb{R}^{n\times 7}$ is the sampled texture. $T\in r_{tex}\times r_{tex}\times r_{tex}\times 7$ is the learnable texture grid, and $r_{t}ex$ represents the resolution of texture grid. Then, the textured mesh can be reconstructed under the supervision of the original multi-view images and MVS outputs via differentiable rendering.

To optimize the coarse mesh obtained by visual hull, we propose a differentiable rendering-based optimization framework. Given the input mesh $\mathcal{M}(V,F)$, a differentiable renderer [11] interpolates the attributes on vertices to pixels regarding to the camera parameter $\pi$. The gradients with respect to vertex positions related to occlusion, visibility, and coverage can be computed by the differentiable renderer $\zeta$. The rendered silhouette $\hat{\boldsymbol{S}}$ can be obtained by interpolating the constant value of one

As the differentiable renderer is able to backpropagate the gradient on the pixel back to the position of vertices, we impose the silhouette loss to limit the boundary of the generated mesh within the mask annotations,

where $||\cdot||^{2}_{2}$ represents $L_{2}$ norm. $N$ is the number of views.

Similarly, we can render the depth map $\hat{\boldsymbol{D}}$ by a differentiable renderer $\zeta$ using the camera projection matrix and current mesh prediction, which interpolates $z$ coordinate of each vertex

In this work, we make use of the off-the-shelf MVS method [23] to estimate the depth map from the input images. Therefore, we can improve the mesh geometry by minimizing the difference between the rendered depth map and predictions from images as follows

where $|\cdot|$ denotes $L_{1}$ norm. $D$ is estimated by  [23]. $P^{valid}$ is obtained from the MVS method [23], which represents the valid indices in the estimated depth map.

Instead of using MLP to represent the mesh texture like the conventional approaches [12, 13], we propose a physically based inverse rendering approach to jointly estimate the lighting and reflectance of the objects.

The Fresnel function $F$ is used to simulate the way light that interacts with a surface at different angles

where $\theta$ is the angle between the viewing direction and the half vector. $F_{0}$ is the material response at normal incidence, which is set to $0.04$ empirically.

The distribution function $D$ is used to describe the statistical orientation of micro facets at the given point

where $\alpha$ is the roughness of surface. $h$ is the half vector.

The geometry function $G$ is used to describe the attenuation of the light due to the microfacets shadowing each other

As for the light source of the scene, we adopt an HDR light probe image [41] in the latitude-longitude format, which is simple and direct. Specifically, we use a $4\times 8$ resolution for our lighting environments. In our implementation, we create an environment map for each image.

Let $V_{tex}$ represent the texture parameter for each vertex, which is sampled from the learnable texture grid by trilinear interpolation. Therefore, the interpolated texture map $\hat{T}$ can be obtained by

Let $(u,v)=\pi(p)$ denote the 2D projection of 3D intersection point $p$. The pixel color of the rendered image $\hat{I}$ is computed by the rendering equation

From the above all, the inverse rendering optimization can be directly self-supervised by the input image through minimizing the photometric loss $\mathcal{L}_{c}$ as below

During the joint optimization, the probe pixels in the lighting environments are updated by the photometric loss.

In this paper, we firstly compute the visual hull at the resolution of $128^{3}$. Secondly, we uniformly sample 10,000 oriented points from the initial mesh triangulated from the visual hull using marching cube, which are used to minimize the depth and silhouette differences through the differentiable rendering-based optimization. It is performed at a resolution of $128^{3}$ for 150 epochs. To obtain the fine details, we increase the resolution to $256^{3}$ and the number of sampled oriented points to 60,000 for the extra 150 epochs. Moreover, we resample points and normals every 50 epochs in order to increase the robustness of the optimization process, and replace the original point cloud with the resampled ones. Thirdly, the texture of the generated mesh is interpolated from a learnable texture grid with an initial resolution of $128^{3}$, which is interpolated into $256^{3}$ after 150 epochs. Finally, the texture grid and environment map are supervised by the photometric loss. In our implementation, we use Adam optimizer in minimization. The learning rates for the oriented point clouds and texture grid are $5e^{-4}$ and $1e^{-4}$, respectively. The weights for the loss term are $\lambda_{c}=5$, $\lambda_{s}=10$, and $\lambda_{d}=30$. The optimization and inverse rendering are implemented by PyTorch. We use the differentiable renderer nvdiffrast [11] to obtain the silhouette and texture map. Then, we perform Cook-Torrance BRDF rendering to get the rendered image.

We firstly evaluate our proposed approach on the DTU MVS dataset [42], which contains 128 scans. For each scan, there are 49 calibrated cameras surrounding the captured object. We evaluate both the generated mesh and rendered image using the same set of scans as the conventional method [12, 13].

To facilitate fair comparison, Chamfer distance is employed as the evaluation metric to measure the accuracy of generated mesh, and PSNR is used to evaluate the rendered images. We compare our method against the recent state-of-the-art 3D reconstruction methods, including PhySG [33], Vis-MVSNet [23], IDR [12] and MVSDF [13]. Depth maps from Vis-MVSNet are fused into point clouds, which are further converted into meshes by the screened Poisson Surface Reconstruction (sPSR) [43] with trim parameter 5. The color is assigned from input images by back projection. The reconstruction results of Vis-MVSNet and MVSDF may include some extrapolated surfaces as they do not use the pre-defined mask. Therefore, a graph-cut based algorithm is performed to trim the extra faces for fair comparison.

Table I shows the quantitative results. It can be clearly observed that our proposed approach achieves the lowest mean Chamfer distance on DTU dataset. This indicates that we achieve the most accurate reconstruction results. Due to the lack of GPU RAM, we use an environment map of $4\times 8$, which cannot simulate the complex lighting with high fidelity. Our rendered results on the image quality are a bit inferior in some objects with high reflectance like scan 37 and 97. While Cook-Torrance BRDF model can simulate rough surfaces well, this leads to good PSNR results, such as scan 83 and 105. Moreover, our proposed method achieves the highest mean PSNR on DTU dataset.

In order to better measure the quality of synthesized images, we also employ SSIM and LPIPS as evaluation metrics to further evaluate the image synthesis results. We compare our proposed approach with IDR and MVSDF, as they perform well on PSNR. Table II shows the experimental results. It can be seen that our proposed method outperforms other methods under SSIM and LPIPS evaluation metrics. These results show that our proposed can generate the photo-realistic images. Fig. 3 gives the example reconstruction results. It can be seen that our method obtains more accurate meshes and synthesizes the photo-realistic images.

We further evaluate the computational cost on DTU dataset. All the experiments on computational time are conducted on the same machine with an Intel i9 CPU (32GB RAM) and a NVIDIA 2080Ti GPU (11GB RAM). Table III summarizes the computational cost for the compared methods. It can be seen that our proposed approach requires 30 minutes to obtain the 3D reconstruction results while the implicit representation based-methods need several hours. This is because the gradients of MLP decreases as the layer goes forward. The parameters of MLP only change a little at each iteration, which leads to longer training time. Due to the implicit shape representation of MLP, a sphere tracing algorithm is performed to find the intersection of rays and the optimized mesh at each iteration, which consumes a lot of time. We employ the oriented point clouds as the shape representation, where the gradient can be efficiently backpropagated to points and normals. At each iteration, we obtain the optimized mesh from marching cube while rasterization is completed by the differentiable renderer. This is much more efficient than sphere tracing algorithm.

In terms of rendering time, our approach is able to run at 25Hz by taking advantage of the physically-based rendering. Due to the short rendeirng time of our method, we render 1000 times and measure the average time. Moreover, we can export mesh and texture from the oriented point clouds and texture grid. In addition, we can export the reconstruction results into the existing rendering engines to enable real-time rendering. For the implicit representation-based methods, it takes dozens of seconds to render an image of resolution $1600\times 1200$, since a forward network inference is required for each valid pixel.

To compare the performance of the scene reconstruction, we evaluate our proposed method on EPFL dataset. We conduct experiments on Fountain-P11 and Herzjesu-P8, which have ground truth meshes. We compare our method against Vis-MVSNet, IDR and MVSDF. As EPFL dataset does not provide the mask, we generate it by projecting the ground truth mesh onto the image. The experimental results of Vis-MVSNet and IDR are from MVSDF [13].

Table IV gives the quantitative results. We use the same evaluation metrics as the DTU dataset. It can be seen that our approach performs better than the state-of-the-art methods on both synthesized images and reconstructed meshes. Fig. 4 shows the reconstruction results on the EPFL dataset. Obviously, our method both estimates high-quality mesh and renders the high quality images. Compared with other methods, our proposed approach recovers more details such like the trace of blocks, a nd the decoration on door. The reconstructed mesh of IDR has the inflated surfaces, since it is difficult to recover the accurate shape only from color constraints. By taking advantage of the extra supervision from multi-view stereo, both MVSDF and our presented method can recover the correct geometry. The experimental results on EPFL dataset show that our proposed method is not only effective for small indoor objects, but also achieves promising results for in-the-wild outdoor scenes.

We provide the qualitative results on BlendedMVS, Tanks and Temples dataset. The masks of the Horse and Family are generated by an off-the-shelf image segmentation framework [44], and the camera parameters are estimated by COLMAP [2]. Fig. 5 shows several examples. It demonstrates that our method is able to reconstruct the accurate geometry and texture from both synthesized images and in-the-wild images. The results in the second row of Fig. 5 show that our proposed method is able to extract the accurate mesh and textures from multi-view images without masks as well.

As our method explicitly extracts the triangulated mesh and texture, which can be combined and rendered with the arbitrary lighting. Instead of using two MLP to represent the implicit surface and radiance field, we employ the point-based mesh and physically-based rendering that decouples the texture and illumination. We can synthesize the images by combing the extracted meshes under various illuminations. Mesh composition and illumination changes are difficult for those MLP representation as the geometry and color information are encoded in MLP. While our proposed method can extract mesh and texture from the oriented point clouds and texture grid efficiently. We arbitrarily combine meshes and change the environment map, where the hidden surface removal can be done efficiently by the renderer. Fig. 6 illustrates the qualitative results. We present the textured meshes and rendered images in different lighting. It shows that we can synthesize the images with different meshes and various lighting. Moreover, the synthesized images are with high quality.

We treat every pixel of the environment map as a light source, which greatly increases the GPU memory consumption. As the resolution of environment map is quite small, the relighting results are not very obvious. Our reconstruction outputs are fully compatible with the existing rendering engines, such as Unreal Engine [45], Blender [46] and so on. Although PhysG [33] can render realistic images with different lighting, Sphere Gaussians are used to represent texture and lighting. It is not compatible with current rendering engines. We can export the reconstruction results into these engines to generate the photo-realistic images with complex lighting. Fig. 7 shows the results rendered by Blender. It can be seen that we can render photo-realistic images by our reconstructed meshes under arbitrary complex lighting.

In this section, we firstly discuss the effect of depth loss on the surface recovery. Previous methods [12, 33] mainly try to minimize the rendering loss, where the shape is just a by-product. Although the geometry can be estimated accurately in some cases, it is usually difficult to recover the accurate shape due to the ambiguity between geometry and appearance. We replace the depth loss with the rendering loss used in previous work to learn the shape. Specifically, we employ an MLP to estimate the color of each pixel. The gradient is backpropagated to the shape through the proposed intersection points representation described in [12]. The SDF value is obtained from the SDF grid by trilinear interpolation, which is optimized by the rendering loss. The qualitative results are shown in Fig. 8. We can get roughly correct results with rendering loss. When the topology or texture of the optimized mesh is complex, it is hard for the rendering loss to handle the ambiguity of shape and appearance. The depth maps predicted by multi-view stereo provide the correct geometry information, which result in more accurate results and enable the fast convergence. Table V shows the quantitative results. It can be seen that we cannot get the accurate shape using the rendering loss only.

We investigate the effectiveness of silhouette loss that helps to recover only the parts of interest. Without silhouette constraints, we may reconstruct the whole scene. As depicted in Table V, we will get higher mean Chamfer distance without mask supervision. This is because the Chamfer distance evaluation will be affected by the surroundings like floor. Although the Chamfer distance above a certain threshold will not be counted in evaluation metrics, there are still some reconstructed parts with the Chamfer distance less than the threshold. As the recovered mesh is extracted from an SDF grid at the resolution of $256^{3}$, the reconstruction results are affected by the size of scene. Fig. 9 shows the example of silhouette loss on the reconstruction results. It can be seen that our proposed method also can reconstruct correct results without mask supervision. The meshes obtained without mask supervision are rougher, since the reconstruction range is larger while the resolution of SDF grid does not change.

We also perform ablation study on the different initialization schemes. We compare the results of sphere and visual hull initialization. Results are shown in Table V. It indicates that visual hull initialization barely affects the reconstruction results. Moreover, we still can get the accurate meshes from the sphere initialization. Fig. 10 gives the example results. It is difficult to dig a hole from sphere initialization. This is because the gradients estimated by differentiable renderer is incorrect in this case. Since the view of DTU dataset does not fully cover the object, there is still some extra surfaces when deforming from the sphere.

Moreover, we conduct ablation study on different losses. We keep the same weight of the loss, and only change the type of depth loss and silhouette loss. Table VI shows the quantitative results. It can be seen that the type of depth loss has great influence on the results while the type of silhouette loss has little affect on the results. This may be due to the small value of $L_{2}$ norm depth loss, resulting in insufficient supervision of the reconstruction results. We use the visual hull initialization so that the silhouette loss is small at the beginning. It has barely effects on the results. The qualitative results are shown in Fig. 11. It can be seen that we can get smooth and accurate results with $L_{1}$ norm depth loss while the results of depth loss become rough and incorrect using $L_{2}$ norm.