Volume Feature Rendering for Fast Neural Radiance Field Reconstruction

Standard volume rendering integrates color along a cast ray according to density [12]. A high density of a point on the ray indicates a high probability of hitting the surface. The NeRF and its various variants adopt this standard volume rendering technique, where the color and density of a point are typically predicted by a neural network. For a cast ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$, where $\mathbf{o}$ is the camera origin and $\mathbf{d}$ is the view direction, the rendered color $C(\mathbf{r})$ using standard volume rendering is:

$$C(\mathbf{r})=\int T(t)\sigma(\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),\mathbf{d})\,dt,\>\text{where}\>T(t)=\exp\left(-\int^{t}_{0}\sigma(\mathbf{r}(s))\,ds\right)$$

(1)

where $\sigma(\mathbf{x})$ and $\mathbf{c}(\mathbf{x},\mathbf{d})$ are the density and color at position $\mathbf{x}$, repsectively. In practice, the above integral is solved by sampling and integration along the cast ray. After transforming the densities to weights by $w(t)=T(t)\sigma(\mathbf{r}(t))$, the color will be rendered as follows:

$$C(\mathbf{r})=\sum_{i=1}^{N}w_{i}\text{NeuralNet}\left(F(\mathbf{x}_{i}),\mathbf{d}\right)$$

(2)

where $\mathbf{x}$ is the position of a sample point on the ray and $F$ is a function to query the corresponding feature vector of $\mathbf{x}$. Pure MLP-based methods including the NeRF [15], Mip-NeRF [2], Ref-NeRF [24] and Mip-NeRF 360 [3] represent $F$ by use of MLPs. However, this representation imposes a computational burden as $N$ times MLP evaluations are required to render a ray. Alternatively, recent research shows that modeling $F$ by extra learnable features is significantly faster than pure-MLP based representations, because linear interpolation of learnable features is much more efficient than MLP evaluations. These learnable features are typically organized in the forms of the 3D grids, hash tables, and decomposed tensors. With the learnable features, a small NN can achieve the state-of-the-art rendering quality but accelerate both the training and rendering time significantly.

Furthermore, hierarchical importance sampling guided by coarse geometries is employed to greatly reduce the number of samples in (2). This importance sampling can be achieved by modeling the density fields independently [21, 5, 9], occupancy grid [16], and proposal networks [3, 4]. As shown in Fig. 1, importance sampling makes the model focus on valuable samples on the surface. For instance, by using two levels of importance sampling, the Zip-NeRF reduces the final number of valuable sampling to 32. However, the NN still needs to run 32 times to deliver the best rendering quality. Although integrating colors predicted by the NN is in line with the concept in classical volume rendering, the many times NN running is the main obstacle for fast training and rendering. On the other hand, the importance of a large NN in realistic rendering especially in modeling the view-dependent effect is highlighted in recent work [3, 9]. Considering the fact that valuable samples on the surface are close in terms of spatial position, the queried feature vectors are also very similar as they are interpolated according to their position. As such, evaluating the NN with similar input feature vectors is not necessary. In the next subsection, we will introduce the proposed volume feature rendering method that integrates these feature vectors to enable a single evaluation of the NN.

As shown in Fig. 1, the proposed volume feature rendering method integrates queried feature vectors instead of predicted colors to form a rendered feature vector. A subsequent NN is then applied to transform the feature vector to the final rendered color. By doing this, the NN only needs to run once. Specifically, we take the NN out of the integral as follows:

$$C(\mathbf{r})=\text{NeuralNet}\left(\left(\sum_{i=1}^{N}w_{i}F(\mathbf{x}_{i})\right),\mathbf{d}\right).$$

(3)

This fundamental change to the standard volume rendering framework relieves the rendering framework from the high computational cost caused by many NN evaluations. Consequently, we can use a larger network to achieve a better rendering quality while maintaining a similar training time.

It is reasonable to integrate feature vectors of non-empty samples on the surface in the proposed VFR. However, at the beginning of training, the model does not have a coarse geometry to focus on samples on the surface. The VFR will integrate feature vectors of all samples during the early training stage. We found that this feature integration works well for most scenes but fails to converge for some scenes. Such failure is more likely to occur when employing a comparably large NN. The VFR normally converges using the NN with two hidden layers each of 256 neurons, but it does not converge when using 4 hidden layers. Fig. 2 illustrates a failure case on the scene of ship from the NeRF synthetic dataset [15].

As increasing the size of the NN only has a slight implication to reconstruction and rendering speeds but can lead to a better rendering quality thanks to our proposed VFR framework, we are motivated to use a large NN. We hypothesize that the reason for the aforementioned convergence failure is that the model cannot find the correct geometry due to the feature integration of all samples in the early training stage. Motivated by this insight, we design a pilot network that functions as standard volume rendering to aid the model in finding a coarse geometry in the early training stage. A small pilot network and a few training steps are sufficient to achieve this goal. In this work, we use the pilot network with two hidden layers, each with 64 neurons. The pilot network is only used in the first 300 training steps. After this training, the pilot network will be discarded, and we can use the proposed VFR to train the model normally. The number of pilot training steps is small compared with the total training steps, e.g., 20K steps. Thus, it has a negligible effect on the overall training time. As shown in Fig. 2, the model converges normally with the aid of the pilot network.

We use the spherical harmonics (SH) feature encoding method to encode view direction for modeling view-dependent effect. This approach can be seen as a variant of the rendering equation encoding in [9]. Different from the vanilla SH encoding [16] employed in the literature, we multiply each encoded SH coefficient by a feature vector as shown in Fig. 3. Mathematically, an SH feature encoding $\mathbf{e}^{m}_{l}$ can be expressed as:

$$\mathbf{e}^{m}_{l}=\mathbf{f}^{m}_{l}Y^{m}_{l}(\mathbf{d})$$

(4)

where $\mathbf{f}^{m}_{l}$ is an SH feature vector and $Y^{m}_{l}$ is the SH basis function with degree $l$ and $m=\{-l,-(l-1),...,0,...,l-1,l\}$. $\mathbf{f}^{m}_{l}$ is a feature vector predicted by a spatial MLP from the rendered feature vector. All resultant SH encodings form a comprehensive encoding vector, which will be concatenated with a view-independent bottleneck feature vector. The concatenated one will feed into a directional MLP to predict the final rendered color. The NN in (2) thus includes the spatial and directional MLPs.

The proposed VFR achieves the best rendering quality but uses the minimum training time compared with state-of-the-art fast methods on the NeRF synthetic dataset. Since the original Instant-NGP [16] uses the alpha channel in the training images to perform background augmentation, we report the results of the Instant-NGP from NerfAcc [13], where such augmentation is not employed for fair comparison. As shown in Table 2, our VFR outperforms the Instant-NGP by 2.27 dB and NerfAcc by 1.51 dB, but only uses 3.2 minutes for training. Compared with TensoRF [5], our method only uses 33% of its training time while surpassing TensoRF by 1.48 dB in PSNR.

Specifically, the proposed VFR significantly outperforms the compared methods on all scenes. On most scenes, over 1 dB PSNR improvements are achieved by our proposed method compared with the Instant-NPG and NerfAcc. Noticeably, over 2 dB quality improvements on the scenes of ficus and mic are observed in Table 2. In addition, our VFR only needs 6K training steps to reach a similar rendering quality compared with the existing algorithms. Such 6K training steps can be completed within 1 minute on one RTX 3090 GPU, which is less than the 25% of training times of the Instant-NGP and NerfAcc, and only 10% of the training time in the TensoRF.

This significant improvement in rendering quality stems from the increased size of the NN. The MLP in the Instant-NGP and NerfAcc has 4 layers and 64 neurons in each layer. By comparison, we use a 6-layer MLP with 256 neurons. The computational cost of our MLP is roughly 24 times higher than MLPs in the Instant-NGP and NerfAcc. However, thanks to the proposed VFR framework, our MLP only needs to run one time instead of many times as in the standard volume rendering framework employed in the compared methods. As can be observed from Table 2, the single NN evaluation property in our VFR leads to a minimum training time even when we use a much larger NN.

Fig. 4 provides a visual comparison of the rendered novel views by the NerfAcc (based on the Instant-NGP) and our VFR. On the compared scenes, both NerfAcc and the proposed VFR can recover good geometries. However, for scenes drums and ficus, obvious visual artefacts appear in the rendered views from NerfAcc while our results do not have such visual artefacts. Besides, our VFR produces more realistic visual effects such as shadow and reflection, as observed on the scenes of hotdog and mic. The presented error maps and objective PSNRs in Fig. 4 also demonstrate the advantage of our VFR in terms of rendering quality. While NerfAcc and our method use the same underlying feature representation, i.e., hash grids, we believe the quality advantage of our method stems from the large NN enabled by the proposed VFR framework.

We also experiment on 7 real-world unbounded scenes from the 360 dataset [3]. The scenes of flowers and treehill in this dataset are not included as they are not publicly available. We use the unbounded ray parameterization method proposed in the Mip-NeRF 360 [3], where points are mapped to a sphere if their radius to the scene center is larger than one. The latest Zip-NeRF [4] reports on the rendering qualities and training times of state-of-the-art algorithms on this dataset. So we use their results of NeRF [15], Mip-NeRF [2], NeRF++ [30], Mip-NeRF 360 [3], and Instant-NGP [16] for comparison. As the training times of these methods in [4] are measured on 8 V100 GPUs, we follow the Zip-NeRF that rescales the training time by multiplying the number of used GPUs. We acknowledge that this approach may not be rigorous, but the rescaled times provide a rough comparison in terms of training efficiency.

As shown in Table 3, our proposed VFR is able to achieve state-of-the-art rendering quality but using the minimum training time compared with the existing methods. Although the Mip-NeRF 360 is able to synthesize novel high-fidelity views, it requires days to train on the high-end GPUs because it is based on a pure MLP representation. Using learning features organized in hash grids, the Instant-NGP achieves a comparable rendering quality but reduces the training time from days to hours. The most recent Zip-NeRF uses multisampling for high-quality anti-aliasing view rendering at the expense of training time relative to the Instant-NGP. Our method achieves a slightly better rendering quality in terms of the mean PSNR. However, we achieve this state-of-the-art rendering quality with a significantly less training time. As a comparison, our model only uses 11 minutes to train on one RTX 3090 GPU, while Zip-NeRF’s training time is approximately 7.7 hours on one V100 GPU.

It should be noted that we achieve this high-quality view rendering using an entirely different approach than the Zip-NeRF. As the Zip-NeRF focuses on anti-aliasing, multisampling is a reasonable solution but leads to a higher computational cost. Besides, the authors of the Zip-NeRF found a larger NN helps in improving the rendering quality. Since the Zip-NeRF still employs the standard volume rendering method, a larger NN will undoubtedly increase the training time. Our proposed VFR, however, relieves the rendering framework from the high computational cost caused by many network evaluations. As only one network evaluation is required in our VFR, NN evaluation is no longer the computing bottleneck in the rendering framework. As a result, the VFR enables one to achieve a similar rendering quality as the Zip-NeRF but with a significantly less training time.

The visual comparison in Fig. 5 shows that our VFR is able to render more realistic views compared with NerfAcc (based on Instant-NGP). Our model provides a more accurate geometry reconstruction on the scene of bicycle. As can be observed from Fig. 5, the rendering results of NerfAcc contain obvious visual artefacts on the scenes of bonsai and kitchen, while our results are more realistic and closer to the ground truth. The reflective effect in the highlighted patches on the scene of kitchen is also well modeled by our model thanks to our novel SH feature encoding method.

Table 4 shows the ablation study of the proposed methods. Compared with the commonly used ReLU, we found that GELU activation improves the rendering quality for both the standard volume rendering (A, B) and our VFR (D, E). Larger MLP (B, C) does increase the quality using the standard volume rendering but at the expense of more training time. For our VFR, when increasing the size of the NN by using more hidden neurons (F:64 neurons, G:128, I:256), the rendering quality constantly improves but the training time only slightly increases. Replacing the SH encoding (H) with the SH feature encoding (I) also leads to a better quality with negligible extra training time.