IL-NeRF: Incremental Learning for Neural Radiance Fields with Camera Pose Alignment

Neural Radiance Fields (NeRF) [26] has recently shown great promise in producing photorealistic images from sparse image sets by encoding a 3D scene with a neural network that maps the location of 3D points to color and volume density. However, to achieve this level of performance, NeRF typically requires access to all training data at once in order to estimate the camera pose based on the entire dataset [7, 26]. In practical applications such as automotive and remote sensing, data is acquired sequentially, necessitating an immediately available updated 3D scene representation. Moreover, scenarios arise where a user acquires scans of a scene to train NeRF, only to find that the training yielded less effective results than expected. Consequently, the user rescans the scene with new data to enhance the fidelity of the images rendered by NeRF. In these instances, the scene representation must be trained in an incremental training environment, where the model can only access a limited number of views at each training stage, while still undertaking the task of reconstructing the entire scene.

In the context of incremental NeRF training, existing works [12, 7, 34, 5] generally operate under the assumption that data is segmented into multiple sequential chunks, with access limited to the current chunk while previously processed data is discarded. This assumption to incremental learning presents a notable challenge for NeRF, which requires updating its knowledge without erasing prior learned information, a phenomenon known as catastrophic forgetting [53]. To address this challenge, prior works [12, 7, 34, 5] have investigated the implementation of incremental learning strategies for NeRF training, incorporating a knowledge distillation technique [53] to mitigate catastrophic forgetting. Specifically, before training the NeRF model with the current data chunk, pseudo-RGB values for the scene are generated using the previously trained NeRF model. These RGB values are subsequently utilized to train the NeRF model with the current data chunk, and the process is iteratively repeated, with the prior NeRF model acting as the supervisory teacher for subsequent data chunks. This enables the NeRF model to learn from new data while retaining knowledge from previously discarded data.

Motivation. While existing works propose effective incremental learning methods through knowledge distillation, these approaches are founded on the assumption that continuous data chunks comprise not only 2D images but also corresponding camera pose parameters, pre-estimated from the complete dataset. This assumption poses a paradox as the required camera pose must be estimated from the entire dataset, yet the data is intended to arrive sequentially, relatively independently, with future data chunks remaining unknown and inaccessible while previous chunks are discarded. In contrast, as shown in Figure 1, our work addresses a more practical scenario where pre-estimated camera poses are unavailable for each training data chunk, necessitating the consideration of camera pose estimation and the alignment of its coordinate system.

Challenge. Since the previous training data have been discarded, the incoming training data cannot simply be used directly for camera pose estimation because the isolated estimated camera pose will not be in the same coordinate system as the previous camera pose, which will lead to NeRF training misalignment and failure to render the 3D scene. Therefore, accurately estimating the camera poses of the sequential coming data within the same coordinate system in incremental NeRF training becomes a crucial issue that needs to be addressed.

Contribution. To deal with the above challenge, we present a novel framework for incremental NeRF training, named IL-NeRF, which can incrementally estimate the incoming data’s camera poses and effectively tackle the issue of catastrophic forgetting. (1) We propose an incremental camera pose alignment module that selects a suitable set of camera poses from previous estimates. These chosen poses help render prior training images from NeRF and aid in the joint camera pose estimation process. They also act as a reference coordinate system, facilitating the alignment of newly arrived and previous camera poses. (2) To ensure the appropriate selection of camera poses from the previously estimated ones, we transform this selection task into a graph-based reward-collection optimization problem. We then introduce a practical greedy algorithm to effectively solve this optimization problem. (3) To align camera pose coordinates, we use selected camera poses as a reference to derive a transfer matrix, transforming the current camera poses into the previous coordinate system. (4) We utilize a joint optimization method for camera poses and replay-based NeRF distillation, mitigating catastrophic forgetting and refining the accuracy of the camera poses.

The experimental results on three diverse datasets show that our proposed framework can improve PSNR, SSIM, and LPIPS by up to $54.04\%$ compared to the original NeRF, significantly mitigating the negative impact of catastrophic forgetting in NeRF’s incremental learning process. Moreover, our framework can effectively estimate and align the camera pose parameters in a consistent coordinate system.

NeRF. NeRF [26] employs volume rendering to depict a continuous scene and achieve high-quality view synthesis. Several subsequent works have been introduced based on the success of NeRF, aiming to improve view synthesis efficiency and quality, including faster training and rendering [28, 6, 10], deformable or dynamic scene synthesis [32, 35, 36], editable view synthesis [30, 49], light changes [23, 27], surface enhancements [31, 45, 51], depth priors [8, 37, 48], etc. However, most of these methods presume access to all training data and require pre-estimated camera pose parameters from the entire dataset. In this study, we tackle a more practical scenario where NeRF learns the scene incrementally with a sequential data stream, without pre-estimated camera pose parameters.
Incremental NeRF Learning. Incremental learning, constrained by limited data availability during each training iteration, often causes catastrophic forgetting [9]. Methods to mitigate this issue include parameter isolation [13, 22, 21], replay [20, 38, 41], and regularization [1, 50, 2]. There is limited research on combining NeRF with incremental learning. Existing studies [12, 7, 34, 5] use knowledge distillation [53] to mitigate catastrophic forgetting by accessing past training data, specifically RGB values, from a pre-trained network. The retrieved data is merged with new incoming data to train NeRF. In [12], a regularization-based filter is used to select relevant information from randomly sampled camera views. In [7], a small network is trained to generate previously seen rays that are directed toward the scene. The authors in [34] employ the same replay-based method in [7] but they substitute NeRF with Instant-NGP [28] to expedite the training process. In [5], a prioritized replay buffer is introduced to keep the images and their camera poses with the lowest historical rendering qualities for continual learning. However, these methods require pre-estimated camera poses from the entire training data for each coming data chunk. In this work, we consider a more realistic scenario where pre-estimated camera poses are unavailable for each training data chunk, thus requiring incremental camera pose alignment.
NeRF With Pose Refinement. Pose refinement is widely used in NeRF training. iNeRF [52] refines camera poses for novel view images using a reconstructed NeRF model. NeRFmm [47] jointly optimizes both camera intrinsic and extrinsic parameters during NeRF training, and BARF [19] proposes a coarse-to-fine positional encoding strategy for camera poses and NeRF joint optimization. SC-NeRF [15] further considers camera distortion refinement and employs a geometric loss to regularize rays. In this paper, IL-NeRF uses the pose refinement in SC-NeRF [15]. Note that IL-NeRF is not limited to only the pose refinement in SC-NeRF [15]; any other well-designed pose refinement methods can also be transferred to IL-NeRF.

NeRF. NeRF aims to learn a 3D scene with a simple neural network, e.g., MLPs, that takes 3D location x and view direction $\textbf{r}_{d}$ as input and produces RGB color c and volume density $\mathbf{\sigma}$ as output. For each ray $\textbf{r}=(\textbf{r}_{o},\textbf{r}_{d})$ emitted from the camera origin $\textbf{r}_{o}$ in direction $\textbf{r}_{d}$, NeRF samples $M$ 3D points along the ray $\textbf{x}_{i}=\textbf{r}_{o}+z_{i}\textbf{r}_{d}$, where $z_{i}$ is the distance from a camera to the sample point $\textbf{x}_{i}$. Thus the pixel color $C$ can be integrated by the volumetric rendering as follows:

where $\delta_{i}=\exp(-(z_{i}-z_{i-1})\mathbf{\sigma}(\textbf{x}_{i})$ represents the transmittance of the ray segment between sample points $\textbf{x}_{i-1}$ and $\textbf{x}_{i}$, and $\alpha_{i}=\prod_{j=1}^{i-1}\delta_{j}$ is the ray attenuation from the origin $\textbf{r}_{o}$ to the sample point $\textbf{x}_{i}$. Since the whole pipeline is differentiable, NeRF can be trained by minimizing the photometric error between the rendering views and ground truth views.

where $R$ represents a group of rays from one or more camera views, which is obtained from the camera pose parameters $\mathcal{P}$. $C^{*}$ is the ground truth of pixel color. $\Theta$ denotes the parameters of the network. For more details of NeRF, we refer the readers to [26].

Incremental NeRF Training. To achieve impressive performance, NeRF assumes access to all training data that covers all views of a scene at once. However, the entire training data might not be available simultaneously in practical applications due to physical or hardware limitations, e.g., edge devices with a limited amount of memory and limited data storage. As a result, data needs to be processed sequentially. In other words, NeRF will incrementally learn the scene with new training data without revisiting the old ones. Concretely, we consider a time-slotted system wherein each time slot $t\in\{0,\cdots,T-1\}$, a chunk of image data $G^{t}$ consists of $N$ number of images $I^{t}$, i.e., $G^{t}=\{I_{n}^{t},n\in\{0,\cdots N-1\}\}$. Generally, only the latest image data chunk $G^{t}$ is available while previous image data chunks $G^{0:t-1}$ are inaccessible. The objective of incremental NeRF is to minimize reconstruction loss across all provided chunks of image data in $\{G_{0},\dots,G_{T-1}\}$. Note that unlike the existing works [7, 12, 34, 5], in our work, each incoming data contains only the images, without corresponding camera poses. Thus, in this paper, the main aim of incremental learning for NeRF is to enable the neural network to learn and adapt continually by ensuring that the camera poses from new image data are estimated and aligned in a consistent coordinate system while preventing catastrophic forgetting across all previously seen image data.

In this section, we introduce our proposed framework, IL-NeRF, which prevents the catastrophic forgetting problem by replay-based knowledge distillation retrieved from NeRF itself (Section 4.1) and utilizes the incremental camera pose alignment to estimate and align the camera poses of incoming training image data within the same coordinates system as the previous camera poses (Section 4.2). The overall pipeline is illustrated in Figure 2.

Dataset. We use three different datasets to evaluate different aspects of our model, namely Mip-NeRF360 [3], LLFF [25], and NeRF-real360 [26]. To simulate the incremental scenarios, we reorganize the camera order so that it moves sequentially and we select a portion of the dataset where the previous images are not revisited. All the training images of each scene are divided into four training chunks denoted as $\mathcal{G}=\{G^{0},G^{1},G^{2},G^{3}\}$. We acquire the camera pose parameters using COLMAP [40].

Baseline and Metrics. IL-NeRF is compared with the following baselines: NeRF: Original NeRF is incrementally trained with only current image data chunk with ground truth camera poses but without NeRF distillation, making it susceptible to catastrophic forgetting. EWC [16]: Similar to the NeRF, EWC incrementally trains the model with only current image data chunk with ground truth camera poses however it utilizes a widely-used regularization-based method, which penalizes the changes in parameters that are important for past training sets. NeRF^∗: NeRF is incrementally trained with ground truth camera poses under replay-based NeRF distillation. Note that the ground truth camera poses are estimated from all the training images. NeRF^∗ can be treated as the representation of the existing works [7, 12, 34, 5], which require the ground truth camera poses in each incoming image data chunk. NeRF-SLAM: we follow the general implementation of NeRF-SLAM [39], which use the SLAM to align camera poses of the coming image data. We replace Droid-SLAM [42] in the NeRF-SLAM to ORB-SLAM2 [29] because Droid-SLAM utilizes a complex deep learning model for camera pose estimation, which needs training on image data before NeRF training.

We evaluate IL-NeRF and baselines in three aspects, including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [46] and Learned Perceptual Image Patch Similarity (LPIPS) [54]. We use AlexNet [17] as the backbone of LPIPS. It should be noted that as joint optimization of poses and NeRF is performed for all camera poses, including the previous and current poses, at each time step, the camera poses are continuously changing. This means it is not possible to fix the camera poses for test data, and all evaluation metrics compare the rendered images from the training camera poses with the ground truth.

In this study, we introduce an incremental learning algorithm called IL-NeRF that tackles the problems of catastrophic forgetting and coordinate shifting in NeRF training in incremental learning settings. The IL-NeRF algorithm employs a replay-based NeRF distillation pipeline to retain past information and learn from new data independently. Furthermore, a method for aligning camera pose coordinates is introduced to identify camera poses associated with incoming tasks during the NeRF model training. Experimental results reveal that the IL-NeRF algorithm outperforms the original NeRF model in sequential data settings.

To find previous $D$ optimal camera poses, we formulate a reward-collection optimization problem on a graph. In this graph, the nodes represent camera positions (i.e., the translations in the camera poses) with each node assigned a reward corresponding to the negative value of the preceding training loss. The edges represent Euclidean distances between each camera pair’s positions. The goal is to find a path that collects as much reward as possible, subject to constraints on the total number of visited nodes and camera view coverage. Concretely, the objective optimization problem can be formulated as

where $x_{k}$ is the binary decision variable: $x_{k}=1$ if node $k$ is visited otherwise $x_{k}=0$. $S(K)$ is the shortest path that connects all the selected nodes. $E(x_{k})$ is the number of incoming edge of each selected node. The first constraint (16) makes sure that only $D$ previous camera poses are selected. The second constraint (17) means that the view coverage of the selected cameras is larger than a threshold. This is because a large field of view coverage of the selected cameras improves the accuracy of the camera pose estimation. The third constraint (18) guarantees that every node only has one incoming edge. In other words, every node is visited at most once. Consequently, this reward-collection optimization problem can be viewed as a hybrid of the Knapsack Problem and the Travelling Salesperson Problem, which is an NP-hard problem.

Related Work for Reward-Collection Optimization Problem. The reward collection problem, also named orienteering problem, is an optimization issue that aims to determine the most efficient route for visiting multiple locations while maximizing the value or score of each place seen, all within a specified time frame and beginning and ending at a particular point [11]. This problem is widely utilized in the tourism sector [44], robot routing [33], food delivery [43] and transportation control [24]. As the orienteering problem belongs to the NP-hard class of problems, no algorithm can solve it optimally within a reasonable amount of time [14]. Different from the traditional orienteering problem, the optimization problem in this paper is more complex. Firstly, we do not limit the start and end points, and at the same time, we have a limitation on the number of accessible points, which makes it impossible to apply the existing proposed approaches to our method.

Brute-Force Method. The straightforward approach to address this problem is the Brute-Force method, demonstrated in Algorithm 2. This method involves: (1) Determining the shortest path for visiting all nodes for each $D$ camera combination. (2) Selecting the camera combination with the highest total rewards, while ensuring compliance with all constraints. However, the time complexity of this approach is $O((2^{D}\times D)\times\binom{N}{D})$, where $\binom{N}{D}$ represents the number of $D$-combinations derived from a given set of $N$ previous camera poses. $2^{D}\times D$ is the time complexity of finding the shortest path for each D camera combination. While this method guarantees an optimal solution, it becomes impractical for large numbers of nodes due to its time complexity.

Proposed Greedy Algorithm. Let $\mathbb{G}(V,E)$ be the graph of the previous camera poses, where $V$ denotes the set of cameras (nodes) and $E$ denotes the set of edges connecting each pair of two cameras. Let $e_{i,j}=e_{j,i},e_{i,j}\in E,e_{j,i}\in E$ denote the edge between camera $i$ and $j$. Note that $\mathbb{G}(V,E)$ is an undirected weighted complete graph. The core concept of our greedy algorithm is to traverse all unvisited nodes starting from a specific node. During this process, the algorithm calculates the approximate edges between each pair of the current node and its connected unvisited node, subsequently selecting the node with the maximum approximation edge as the next starting node. This process is repeated until a total of $D$ nodes have been selected. The complete description of the greedy algorithm is outlined in Algorithm 3.

Step 1 (line 1 to line 2): Introducing an auxiliary starting node $V_{0}$ into the graph, which establishes connections to all nodes with an edge length of 0. We define a set $\mathcal{B}$ to keep track of the visited nodes during traversal and initialize it as $\mathcal{B}={V_{0}}$. Additionally, the current node index is initialized as $k=0$.

Step 2 (line 4 to line 10): Identifying all the connected nodes of the current node $V_{k}$ that have not been visited yet (i.e., $V_{i}\notin\mathcal{B}$). Based on the reward $R_{i}$ and the edge length $e_{k,i}$ for each unvisited node, we compute the approximation edge length $\hat{e}_{k,i}$ and insert it into a temporary set $\hat{E}$. Specifically, $\hat{e}_{k,i}=R_{i}+\lambda(\frac{S_{th}}{D}-{e}_{k,i})$, where $\lambda$ is a parameter for adjusting the units of $R_{i}$ and $\frac{S{th}}{D}-{e}_{k,i}$. This approximation edge is similar to the Lagrange multiplier [4] for handling the constraint (17).

Step 3 (line 11 to line 12): Selecting the node with the maximum $\hat{e}_{k,i}$ as the next visited node, updating the current index as $k=\arg\max\hat{E}$, and inserting the next visited node into the set of visited nodes $\mathcal{B}$.

Step 4: Repeating Step 2 and Step 3 until the greedy algorithm has visited a total of $D$ nodes.

The time complexity of our greedy algorithm is $O(D\times N\log N)$, which reduces computation time by several orders of magnitude.

Comparison of Brute-Force Method with Ours. Here, we show the performance comparison of Brute-Force Method and our greedy algorithm in terms of PSNR and computation time. As the results in Table 2, 3 and 4, L-NeRF can achieve comparable PSNR with Brute-Force method, however, the runtime of the Brute-Force method is several orders of magnitude larger than that of our proposed greedy algorithm, which makes the Brute-Force method impractical for incremental training scenarios. It should be noted that the runtime of the Brute-Force method increases exponentially with the size of the training data. For example, the ‘Bicycle’ scene in Mip-NeRF360 contains 194 training images and ‘Horns’ scene in LLFF contains 62 training images, but the Brute-Force method’s runtime for these two scenes are 5 days 6 hours and 1 hour 47 min, respectively.

In the main text, we only show PSNR, SSIM and LPIPS for some scenes of the three datasets, and here we give the full results. Table 5 shows the results obtained by IL-NeRF and baseline methods on the Mip-NeRF360 dataset with seven real-world indoor and outdoor scenes. Similarly, Table 6 presents the results obtained on the LLFF dataset with eight forward-facing scenes. Additionally, Table 7 also shows the results obtained on the NeRF-real360 dataset with two real-world object-orientation scenes. From the results, we can see that IL-NeRF outperforms the original NeRF and achieves comparable results with NeRF^∗.

Furthermore, we compare the original NeRF and IL-NeRF on two scenes, the ’Kitchen’ and ‘Counter’ scenes in the Mip-NeRF360 dataset. Here, we give more visualization results of original NeRF and IL-NeRF on all scenes of three datasets.

Figure 9 to Figure 16 provide additional insight by presenting a qualitative comparison of the performance of the original NeRF and IL-NeRF. Specifically, we demonstrate the rendering results on the first task after each incremental training. It is evident that the original NeRF suffers from the catastrophic forgetting problem, resulting in images with significant distortions such as noise and blur, whereas IL-NeRF generates highly realistic images with quality comparable to the ground truth. This observation indicates that IL-NeRF is highly effective in mitigating the forgetting problem and addressing the coordinate shifting issue.

Video Demo. To further show the performance of IL-NeRF, we post a video demonstration in the supplementary material, named ‘sm$\_$video.mp4’. In this video, we show rendered images from all baselines and IL-NeRF.

In the main paper, we analyze the effectiveness of the camera coordinate alignment and the pose refinement that has been added to IL-NeRF on the scene ’Garden’. Here, we give more numerical results for the ablation study.

Table 8 to Table 10 shows the performance comparison of the original NeRF, NeRF^∗, IL-NeRF, IL-NeRF w/o TM, and IL-NeRF w/o PR on all three datasets. As we can see, IL-NeRF outperforms the original NeRF and achieves comparable results with NeRF^∗. The results reveal a significant decline in performance on all test data without the transfer matrices (i.e., IL-NeRF w/o TM). This decline can be attributed to separate camera pose estimation for two tasks resulting in camera poses in two independent coordinate systems, which could mislead the model during training, leading to decreased performance. The results of IL-NeRF w/o PR, indicate that IL-NeRF with pose refinement outperforms IL-NeRF without it as the aligned camera poses by the transfer matrices may still contain noise and inaccuracies.

Figure 17 shows the camera pose trajectories of GT and IL-NeRF. We treat the COLMAP estimation from all training images as ground-truth (GT) camera poses. As we can see, IL-NeRF recovers accurate camera poses due to the help of incremental camera pose alignment.

For large-scale scenes with limited overlap between views in the training dataset, the performance of IL-NeRF may be suboptimal because the limited overlap between views can result in significant errors or even the inability to calculate the transfer matrices during the camera coordinate alignment.