SpNeRF: Memory Efficient Sparse Volumetric Neural Rendering Accelerator for Edge Devices
^††thanks: * Corresponding author. ^††thanks:

Vanilla NeRF [1] has revolutionized novel view synthesis by integrating computer graphics techniques with deep learning. As a pioneering work, NeRF introduced the use of deep neural networks to represent and render 3D scenes from a sparse set of input views. However, the original NeRF implementation relies on computationally expensive multi-layer perceptrons (MLPs) with a large number of parameters. Consequently, the rendering process can be time-consuming, often requiring hours or even days to generate a single novel view. Therefore, the volumetric method is proposed to boost training and rendering efficiency [11]. It estimated the color features and density of sampling points by interpolating the data stored in the voxel grid.

VQRF [10], as shown in Fig. 1, identifies the redundancy in voxel grid data and proposes voxel pruning and vector quantization to minimize the memory size. Also, VQRF has a relatively smaller MLP (only 3 layers with channel sizes of 128, 128, 3). However, it cannot fully take advantage of the sparsity brought by the voxel pruning since it requires restoring the full voxel grid using the pruned voxel grid points. Therefore, it leads to low efficiency when rendering on edge computing platforms, such as XNX and ONX.

Various encoding methods have been proposed for SpMM to enhance memory and computation efficiency [12]. However, current approaches have limitations. The COO format requires storing all coordinates, which brings an extra 630 KB of memory usage for each scene on average in our experiments. CSR and CSC formats necessitate row-wise or column-wise data storage, respectively. While COO offers simple implementation but high memory overhead, CSR provides efficient row-wise access at the cost of poor column-wise performance, and CSC excels in column-wise operations but struggles with row-wise access. These encoding methods often result in excessive memory consumption and numerous lookups during irregular data access. Consequently, these factors lead to frequent off-chip memory accesses and the large on-chip memory demand, both of which significantly impact performance. A detailed analysis of these performance implications will be presented in Section II-C.

To identify VQRF bottlenecks, we profile the neural rendering process across various computing platforms and characterize sparse data in different datasets. We utilize one high-end computing platform, NVIDIA A100 (A100), and two edge computing platforms, ONX, and XNX, whose specifications are summarized in Table I. Evaluating VQRF on Synthetic-NeRF [1] datasets for each GPU, we present the runtime breakdown in Fig. 2(2). Profiling results reveal that edge computing platforms spend the most time accessing memory, while the A100 allocates minimal time to this task. We also examine voxel grid data redundancy by separating non-zero and zero components and calculate sparsity, as illustrated in Fig. 2(2).

Our profiling results yield two key observations: First, the rendering process on edge devices is predominantly memory bandwidth-bound. The proportion of time spent on memory access in edge computing platforms is $4.79\times\sim 5.14\times$ higher than in high-end computing platforms, due to the smaller L2 cache size and relatively low DRAM bandwidth of edge devices. Second, there is substantial redundancy in voxel grid data. As illustrated in Fig. 2(2), non-zero points occupy only $2.01\%\sim 6.48\%$ of total voxel grid data, indicating significant potential for leveraging sparsity to enhance efficiency. These findings underscore the need for an on-chip processing flow to mitigate expensive off-chip memory access and alleviate the memory bandwidth bottleneck.

The preprocessing step encompasses three stages, as shown in the red line in Fig. 3. Initially, we identify the non-zero points in the sparse voxel grid and extract their position coordinates $(x,y,z)$ into a vector $\boldsymbol{p}=[x,y,z]^{T}$. We then aggregate all these position vectors into a set $P_{nz}=\{\boldsymbol{p}_{i}|i=1,2,...,N\}$, where $N$ is the total number of non-zero points. This step preserves the spatial information and saves it for further voxel grid partition and hash mapping.

Subsequently, we partition the identified non-zero points into $K$ subgrids based on their $x$ coordinate values. This partitioning is defined as $S_{k}=\{\boldsymbol{p}_{i}|\lfloor x_{i}/w\rfloor=k,\boldsymbol{p}_{i}\in P_{nz}\}$, where $k\in{0,1,...,K-1}$, $w$ is the width of each subgrid, and $x_{i}$ is the $x$-coordinate of $\boldsymbol{p}_{i}$.

Consequently, to efficiently support the irregular data access patterns inherent in neural rendering, we map each subgrid $S_{k}$ into a separate hash table $H_{k}$ using the following function from [2]:

where $T$ is the number of entries per hash table level, $\pi_{1}=1$, $\pi_{2}=2654435761$, and $\pi_{3}=805459861$. Note that each hash table entry stores the index for non-zero value lookup in online decoding. This final mapping process enables rapid lookup during the rendering process and eliminates the need for storing coordinates, thus solving the memory-bound problem and optimizing overall performance.

We incorporate an online decoding step between ray sampling and interpolation to support efficient data retrieval, as illustrated by the blue line in Fig. 3. This online decoding process unfolds as follows: Initially, for each sample point, we retrieve its position and calculate the corresponding hash index using (1). Subsequently, we utilize this hash index to fetch the lookup index for non-zero values from the hash table. Following this, we employ the retrieved 18-bit index to locate non-zero values in both the codebook and true voxel grid. To streamline this process, we implement a unified 18-bit addressing scheme for both the codebook and true voxel grid. Finally, we implement a bitmap masking technique to mitigate errors primarily caused by hash collisions. This approach utilizes a bitmap that stores a single bit for each voxel grid point, indicating whether the point is zero (0) or non-zero (1). By representing all voxel grid points’ status in just 1 bit each, the bitmap provides a memory-efficient method to track non-zero values. During the decoding process, we consult this bitmap to effectively mask all erroneous values resulting from hash collisions, setting them to zero. Our observations indicate that hash collisions are the dominant source of errors in this process. Therefore, the bitmap masking step is crucial for maintaining accuracy in our decoding procedure. The detailed result for bitmap masking will be analyzed in Section V-B.

In this section, we present the SpNeRF architecture as illustrated in Fig. 4, designed to support our algorithm. The SpNeRF accelerator comprises two primary modules: the Sparse Grid Processing Unit (SGPU) and the MLP Unit. The SGPU is specifically tailored to execute the series of lookup operations required during online sparse voxel grid decoding. Complementing the SGPU, the MLP Unit is implemented as an output-stationary systolic array, a design commonly employed in conventional DNN accelerators. Our design method aligns with [6], extending a standard DNN accelerator with a specialized unit (in our case, the SGPU) to efficiently support neural rendering tasks. The overall dataflow of our system is initiated at the position buffer. Upon retrieval of a position, it is forwarded to the Hash Mapping Unit, which performs a lookup operation to locate the corresponding voxel grid data. The Trilinear Interpolation Unit subsequently processes this voxel grid data to compute the interpolation result. The resulting output is then concatenated with the view direction vector and stored in the input buffer of the MLP unit. The on-chip computing is in FP16 while the true voxel grid data is saved in INT8 format on off-chip memory to save the memory usage and reduce the communication overhead. To ensure high throughput, the entire design is fully pipelined. Furthermore, all buffers in the system are double-buffered, enabling simultaneous data fetching and processing.

The Sparse Grid Processing Unit (SGPU) is designed to support our proposed online sparse voxel grid decoding flow. It comprises four key components:

Grid ID Unit (GID). The GID computes ceiling and round results for each point position to locate the corresponding voxel grid vertex. It also calculates the weight by FP16 multipliers and subtractors for trilinear interpolation using the following equation introduced by [6]:

where $(x_{p},y_{p},z_{p})$ represents the sample point position and $(x_{g},y_{g},z_{g})$ denotes the voxel grid vertex position.

Bitmap Lookup Unit (BLU). The BLU stores the bit mask for the current subgrid. All bit masks are stored sequentially in a contiguous memory space, enabling efficient location of bits using voxel grid vertex positions as memory addresses and selection signals. The BLU lookup results are used to mask incorrect non-zero values caused by hash collisions in the Hash Mapping Unit.

Hash Mapping Unit (HMU). The HMU is the core module of the SGPU, facilitating the hash table lookup crucial for our online sparse voxel grid decoding. It processes the voxel grid vertex results from the GID and computes the hash index using Equation (1). Once the hash index is determined, the color feature index and density are fetched from the hash table stored in the Index and Density Buffer. Our unified 18-bit addressing scheme differentiates color codebook and true voxel grid buffer lookups by comparing the index value. For a color codebook size of $4096\times 12$, lookup requests with color feature index values below 4096 are served by the color codebook, while others are directed to the true voxel grid buffer. The final output is filtered by the bit mask from BLU.

Trilinear Interpolation Unit (TIU). The TIU plays a crucial role in processing and interpolating color features. Initially, it converts the original color features from the true voxel grid buffer, stored in INT8 format, to FP16 format by multiplying the lookup results with the scale factor. Meanwhile, the density data and the color feature vector from the codebook will be directly sent to later computations. Following this conversion, the TIU multiplies each transformed color feature with its corresponding voxel grid vertex weight, as computed by the voxel grid ID Unit. Finally, it accumulates the weighted color features from all eight surrounding voxel grid vertices to produce the final interpolation result. This process can be represented by the equation $C_{interp}=\sum_{i=1}^{8}w_{i}\cdot(s\cdot C_{i})$, where $C_{interp}$ is the interpolation result, $w_{i}$ are the weights for each voxel grid vertex, $s$ is the scale factor for de-quantization, and $C_{i}$ color feature vector in corresponding voxel grid vertex.

The MLP Unit consists of an output-stationary systolic array, activation unit, controller unit, and data buffers. It computes a 3-layer MLP with channel sizes of 128, 128, and 3, respectively. To enhance efficiency, we implement batch processing with a batch size of 64. Addressing the challenge of inconsistent vector sizes between the previous unit’s output and the systolic array’s input dimension, we propose a block-circulant storage format. This optimized memory pattern reduces both memory overhead and read time.

As illustrated in Fig. 5, the block-circulant storage format interleaves the input $39\times 1$ vector across banks 0 to 9. Each input vector is segmented into 10 blocks, with each block containing four consecutive elements. Elements within the same block are stored successively, while adjacent blocks are stored in neighboring banks with a block offset of 4. The write sequence depicted in Fig. 5 demonstrates the process of writing the first vector into the input buffer. To ensure divisibility by 4, we pad the last element with 0. The reading sequence follows the vector element number in ascending order, with each read result undergoing a block shift to maintain the correct sequence of the first vector. As shown in Fig. 5, the current reading vector passes through shift logic, which repositions the first block from bank 0 to ensure that elements of the first vector are correctly aligned with the first row of the systolic array.

Implementation. For Algorithm Evaluation, we implement our proposed SpNeRF algorithm based on VQRF [10] in PyTorch [14]. For Hardware Evaluation, we implement our accelerator in Verilog and synthesize our design using Synopsys Design Compiler based on TSMC 28nm CMOS Technology to obtain the power and area metrics. The operating clock frequency for our design is 1 GHz. On-chip SRAMs are generated by the provided memory compiler with the same technology. To evaluate the overall performance of our SpNeRF architecture, we develop a cycle-level simulator verified against our RTL design. The DRAM timing and power characteristics are obtained using Ramulator [15] with the configuration of LPDDR4-3200, which provides a bandwidth of 59.7 GB/s.

Datasets & Baseline. To evaluate the performance of the proposed SpNeRF, we conduct experiments on the Synthetic-NeRF [1] dataset. For the hardware evaluation baseline, we select the edge computing platforms and edge accelerators as our baselines. For edge computing platforms, we compare our design with Jetson Xavier NX 16 GB and Jetson Orin NX 16 GB (commonly used edge GPUs). For edge accelerators, we compare our design with RT-NeRF.edge and NeuRex.edge (both are dedicated ASIC accelerators for neural rendering).

Subgrid Number & Hash table size. Fig. 7 illustrates the relationship among PSNR, subgrid number, and hash table size. The results reveal that PSNR increases rapidly initially but slow down beyond certain values for both subgrid number and hash table size. Based on this analysis, our design adopts a subgrid number of 64 and a hash table size of 32 k, as larger values yield only marginal improvements in PSNR.

Memory Size Reduction. Fig. 6(6) compares the voxel grid data size of our proposed SpNeRF with the original VQRF. SpNeRF achieves an average memory size reduction of $21.07\times$ compared to VQRF. This significant reduction in memory footprint is primarily attributed to the online sparse voxel grid decoding technique, which eliminates the necessity of restoring the full voxel grid data.

PSNR Performance. Fig. 6(6) illustrates the PSNR results for VQRF, SpNeRF before bitmap masking, and SpNeRF after bitmap masking. Higher PSNR values indicate better image quality. The results demonstrate that our SpNeRF, with bitmap masking applied, maintains PSNR levels comparable to VQRF while simultaneously achieving substantial memory reduction. This indicates that the proposed method effectively preserves image quality despite the significant decrease in memory usage.

Performance. Fig. 8 shows the normalized speedup and energy efficiency compared with edge devices and Table II gives the comparison between our design and previous works. The results indicate that SpNeRF achieves great advances in both speedup and energy efficiency. In comparison, SpNeRF achieves $52.4\times\sim 157.1\times$ speedup compared with XNX and $34.9\times\sim 112.2\times$ speedup compared with ONX. This improvement is attributed to hash mapping adopted in preprocessing and online decoding, eliminating the need for frequent off-chip memory access.

Area. Fig. 9(9) illustrates the area breakdown of our SpNeRF accelerator, while Table II presents the total area and area efficiency of our design. In the area breakdown, the MLP buffer accounts for 58 KB SRAM (comprising input buffer, output buffer, and weight buffer), and the SGPU contains 571 KB SRAM. Notably, in our design, on-chip SRAM occupies only a small fraction of the overall area. This contrasts with other designs, where on-chip SRAM typically dominates the chip area. The results demonstrate that the memory reduction achieved through online sparse voxel grid decoding effectively minimizes on-chip SRAM size, leading to superior area efficiency of $2.67\times$ to $3.04\times$ compared to previous works.

Power. Fig. 9(9) depicts the power breakdown of our SpNeRF accelerator and Table II provides the total power and power efficiency of our design. The power breakdown reveals that the systolic array accounts for the dominant portion of overall power consumption in our design, contrasting with previous studies where on-chip SRAM was the primary power consumer. Our approach achieves a $4\times$ to $4.37\times$ improvement over previous works. When comparing with XNX and ONX, as shown in Fig.8(8), SpNeRF achieves $346.4\times\sim 1030.9\times$ and $288.7\times\sim 937.2\times$ energy efficiency improvement, respectively. This advantage comes from two key factors: First, our design minimizes on-chip SRAM size, which has been observed to be a dominant contributor to power consumption in previous works. Second, our design fully leverages voxel grid sparsity, significantly reducing off-chip memory access, another major source of power consumption.