INV: Towards Streaming Incremental Neural Videos

Layered Representations. Existing methods differ in how they represent 3D scenes. Multiplane Images (MPI) [41, 54, 39, 44, 4, 6] are variable-viewpoint images that represent a scene as a set of fronto-parallel images, layered-meshes, or multilayer-spheres. However, these methods usually exhibit stack-of-card artifacts when viewed at a close distance or from steep angles.

Neural Radiance Fields. NeRF [26] introduced a simple but powerful representation based on Multi-Layer Perceptrons (MLPs) and differentiable volumetric rendering. It achieved unprecedented photorealism while allowing free viewpoints rendering. The high-level idea is to use a neural network to model the radiance of 3D points in the target scene. Notably, positional encodings[42, 38] play a vital role in improving the quality of the reconstructed radiance field. Many works (e.g. [52, 5, 45, 8]) have been inspired by NeRF. However, static NeRF faced two main challenges:

(1) Slow training and rendering. Multiple approaches (e.g. [18, 27, 49, 1, 13, 12, 10, 7]) have been proposed to improve training and rendering speed, but long training time is still the main bottleneck of NeRF-based models.

(2) Generalization to new scenes. Many works like [50, 8, 45, 9] have tried to achieve generalization by adding ConvNets or Transformers to NeRF, but it remains a challenge to learn generalizable neural radiance fields.

Explicit 3D Representations. One of the fundamental difficulties in view synthesis is to model the scene in 3D based on 2D images. To alleviate the burden from view synthesis algorithms, some works [34, 35, 14, 36, 2] rely on preprocessed raw 3D geometry (e.g. point clouds and meshes), obtained from multi-view stereo software such as COLMAP[37]. Such approaches show fewer fog-like artifacts which are common in NeRF-based methods. Methods like [34, 35] also demonstrate good generalization to new scenes. Point cloud-based neural rendering approaches [14, 36, 46] show impressive sharpness and details in large scenes with thin structures. However, the aforementioned approaches rely on good input 3D models to render high-quality images.

Human-Specific Approaches. Some recent works [33, 21, 40, 32, 15] focus on animating clothed humans only. They often use colored videos as inputs and leverage human body templates (e.g. SMPL[23] and STAR[29]) and deep textures. On the other hand, LookinGood[25] uses multiple RGBD cameras to reconstruct human mesh in real-time. They then use a neural network to render colored mesh into high-quality novel views. HVSNet [28] uses monocular RGBD videos and renders from feature point clouds. Our work focuses on general scenes with complex human-object interactions. Therefore, approaches limited to human subjects would not apply to our target scenario.

Dynamic Novel View Synthesis of General Scenes Yoon et al.[48] utilizes multi-view stereo and monocular depth estimators to estimate spatially and temporally coherent depth maps and motion fields. Their approach generates 3D videos without per-video optimization or prior knowledge of the scene. Many NeRF-based approaches[47, 17, 11, 43] encode dynamic scenes as spatio-temporal radiance fields. Many learn a radiance field and a motion field for each frame. The motion field is then used to establish 3D correspondences between frames, thus allowing for temporal consistency regularizations. Optical flow and monocular depth estimators are often used to guide the optimization to a good local optimum. TöRF[3] uses Time-of-Flight sensors to achieve better modeling of both static and dynamic scenes. Approaches like Nerfies[30], HyperNeRF[31], and Neural 3D Videos (N3V)[16] use latent codes to help model dynamic contents. N3V also shows that smart sampling strategies can accelerate training and improve rendering qualities. However, all of these approaches require hours of training per frame, leading to weeks of training for a short video lasting several seconds. Our work shows how to significantly accelerate training and reduce storage via a concise incremental representation that naturally extends to streaming applications.

Prior works [16, 17, 47, 11, 43] can render impressively realistic 3D videos after hours of training per frame. The cause of such high training costs, aside from the well-explored sampling bottleneck, can also be attributed to the redundancy in training. Many works map 4D points $[x,y,z,t]$ (or equivalent) to $RGB\sigma$ values. This input formulation essentially views 3D points from different frames as different 4D points. Therefore, even if the scene is static, the same 3D points would need to be re-learned when the time variable changes. While loss terms enforce consistencies between mappings for different frames, duplicated mappings still consume model capacity and training resources. A similar argument also applies to dynamic points.

We notice that, when the frame rate is infinitely high, adjacent frames should be almost identical. Therefore, given the MLP representation $\Phi_{t}$ for a frame at time $t$, it should take almost no effort to learn $\Phi_{t^{\prime}}$ for the next frame at $t^{\prime}$. Thus, the difference $\Delta\Phi=\Phi_{t^{\prime}}-\Phi_{t}$ should decrease as time difference $\Delta t=t^{\prime}-t$ decreases.

Therefore, a model should reuse most of the previous information given high frame rates. Following this intuition, we propose Incremental Transfer (I.T.) to explicitly reuse information from frame to frame. We train one model per frame, with the earlier frame as the initialization for the later frame. As shown in Fig.3 and 5, I.T. leads to significantly less training by directly improving upon previous frames. The model thus gradually converges to a good quality similar to NeRF with only a fraction of the training. However, such a naive training scheme leads to significant temporal artifacts in rendered 3D videos, like flickering. In later sections, we show how INV reduces such artifacts.

Moreover, in Table. 1, we show that incrementally trained InstantNGP (”Ours NGP”) acheives good results with merely 6 sec/frame of training.

Another benefit of Incremental Transfer is that we can now analyze the effect of different $\Delta\Phi$’s. This leads to our core discovery: When mapping from image/volume coordinates to colors (and/or densities), an MLP naturally partitions itself into Structure Layers and Color Layers. Structure Layers are early layers of the model that store structural information. Color Layers are later layers that store color/style information (i.e. object color, shadows, etc.). Moreover, the mapping spaces learned by each layer are similar across nearby frames. We perform several experiments to show this interesting behavior.

In Sec. 3.1, we showed that Incremental Transfer (I.T.) significantly reduces the training cost for the original NeRF while achieving good per-frame image quality. However, I.T. has two main disadvantages: (1) the storage cost is extremely high. A 5-minute 30FPS 3D video would require 40.1GB of storage. (2) There is significant flickering despite good per-frame image quality. To address these two issues, we propose the Incremental Neural Videos (INV) Representation that drastically reduces storage cost while providing high-quality visual results with notably improved temporal stability over I.T.

The Plenoptic Video Datasets (PVD) [16] is released by Neural 3D Videos (DyNeRF) as their benchmark dataset. It is captured with 21 GoPro cameras at $2028\times 2704$ and 30 FPS. It contains challenging volumetric effects (e.g. flames), view-dependent effects (e.g. specularity from silverware, transparency from bottles), complex actions (e.g. cooking), changing topologies (e.g. cutting beef, pouring liquid), etc. Same as DyNeRF, INV is evaluated at $1352\times 1014$ on the same and only sequence that DyNeRF reported quantitative results on, i.e. the first 10 seconds of the ”flame salmon 1” sequence. We report quantitative results on other sequences in the supplementary.

Immersive Telepresence Dataset. As mentioned in Sec. 3.2.3, we also perform analysis of Structure and Color Layers on our custom dataset. The dataset is captured with 3 camera bars each containing 5 compact cameras, and we follow PVD [16] to use 14 views for training and 1 middle view for evaluation. Different from PVD [16], our dataset is captured closer to the subject and thus much more suitable for analyzing and visualizing Structure Swap. However, since compact cameras have lower image qualities, this dataset is only suitable for visualization and analysis but quantitative evaluation. Please refer to the supplementary for more details.

We follow DyNeRF to use these metrics:(1) Peak Signal-To-Noise Ratio (PSNR), (2) Perceptual Score (LPIPS[53]), and (3) Just-Objectionable-Difference(JOD)[24]. We omit FLIP and DSSIM since we deem the three widely used metrics enough for reliable measurement. The following methods are evaluated:

NeRF [26]: Our baseline NeRF is trained from scratch and stored for each frame separately. This baseline helps to quantify the training acceleration and storage savings provided by INV (which uses the same NeRF internally).

Neural 3D Videos (DyNeRF) [16]: DyNeRF proposed to use learnable latent codes as inputs for each frame (in addition to positional encoding) to improve the rendering quality. DyNeRF also accelerates training via hierarchical training and ray importance sampling. Longer videos are divided into chunks of 10-second clips for better results. The authors also released the Plenoptic Video Datasets [16]. They show showed superior performance to their baselines. Since the DyNeRF code is not available and only results on the ”flame salmon” sequence are reported, we cannot provide side-by-side visual comparisons and can only compare with them on that sequence.

InstantNGP [27]: We train InstantNGP on the scene with 800 iterations (6̃sec) from scratch for each frame.

INV: INV uses the original NeRF with the same settings as our NeRF baseline, but we disable skip connections. This is because they activate more Structure Layers, leading to higher storage costs while providing insignificant improvements. In most experiments, we train 3 Structure Layers. Although the frozen and shared layers also include some Structure Layers, we count them as part of the Shared Color Layers for simplicity. ”INV+TWC” uses TWC to batches and compress all frames after Warm-Up. ”SplitINV” uses a separate frozen NeRF to model the backgrounds, and INV is used to capture the foreground. All metrics are measured on all 300 frames, including the Warm-Up frames which show worse results.

SplitINV: We use a separate NeRF to model the static background, while the foreground is encoded by INV. This allows INV to focus computation on the foreground and also helps stabilize the background. Please refer to the supplementary for more details on training.

Ours NGP: We train InstantNGP on the scene with 800 iterations/frame with the previous frame as the initialization for the MLP and hash grid. Importantly, the hash grid of NGP can be seen as the output of Structure Layers stored in 3D, and the MLP can be seen as the Color layers. In this way, NGP skips the Structure Layers and directly accesses the Color layers. Therefore, after 30 frames of warmup, we fix the MLP and train the hash grid only on the foreground pixels (as estimated by an optical flow estimator[51]). This allows NGP to focus computation on the foreground and also helps stabilize the background.

Multi View Stereo (MVS), Local Light Field Fusion (LLFF), NeuralVolumes (NV), and NeRF-T: We adopt the numbers as reported by DyNeRF[16]. Please refer to [16] for more details.

As shown in Fig. 3 and 5, INV’s quality quickly and continually improves during the Warm-Up Stage. During the Structure Transfer Stage, INV performance stabilizes at a good quality. Moreover, INV can model challenging visual effects (e.g. flames, reflections, etc.) even though the Color Layers are frozen and shared between frames.

As shown in Table. 1, INV achieves state-of-the-art results on per-frame quality metrics (PSNR & LPIPS) with 50 minutes/frame less ($-19\%$) training than DyNeRF. Moreover, with merely 8min/frame of training, we achieve an average PSNR of 28.77db (”INV+TWC”). We also notice that more trainable Structure Layers result in better per-frame image metrics, but it almost completely removes the stabilization effect of freezing later layers as seen in the table and supplementary videos. This is because more structural contents are allowed to change with no guarantee that the change is consistent over time.

In ”INV$\star$”, we show that well-trained Shared Color Layers can notably improve quality. During Warm-Up, the models are trained for 210min/frame. During Structural Transfer, the Structure Layers are only trained for 15min/frame. However, compared to the basic INV trained for 15min/frame throughout the video, PSNR increases by 0.42db (from 28.95db to 29.38db). This quality is similar to the basic INV trained for 90min/frame. Pretraining INV could potentially be useful for many live-streaming settings (e.g. game streaming, lectures, and remote meetings). Since many live-streaming sessions capture the same background and people, it could be helpful to maintain or continually improve a long-term memory via a color/style space that is shared across sessions. Moreover, ”SplitINV” increases the stability via its frozen background NeRF.

Due to its incremental nature, INV can synthesize 3D videos on-the-fly and does not have to wait for video chunks before processing. This property makes it naturally suitable for interactive 3D streaming. Additionally, INV doesn’t make strong assumptions about the modeling backbone, thus it can likely be enhanced by more sophisticated techniques.