ActiveRMAP: Radiance Field for Active Mapping And Planning

NeRF[37] was first proposed for solving novel view synthesis. The core of NeRF is the use of multi-layer perceptron networks (MLPs) as an implicit representation of the scene. The MLPs map a 3D position $x$ and a viewing direction $d$ to the corresponding density $\sigma$ and view-dependent color $c$:

where $\Theta_{pos}$ and $\Theta_{rgb}$ are the learnable MLP parameters, and e is an intermediate embedding to help the shallower $\text{MLP}^{(rgb)}$ to learn the color representation. Given a set of posed images, the training of MLPs relies on the photometric loss between the observed pixel colors $C(r)$ in posed images and the rendered pixel colors $\hat{C}(r)$ in the images viewed from the camera poses , $\mathcal{L}_{photo}=\frac{1}{|R|}\sum_{r\in R}||C(r)-\hat{C}(r)||$, where $R$ denotes a set of rays in a sampled mini-batch. Volume rendering [23] is employed to render the pixel colors by casting a ray $r$ from the camera center through the pixel. Volume rendering approximates light radiance by integrating the radiance along the ray. $K$ points are sampled along the ray between pre-defined near and far planes. After querying for their densities and colors ${(\sigma_{i},c_{i})}_{i=1}^{K}$ via MLPs, the results are accumulated into a single color with the use of quadrature approximation [36] as,

where $\alpha_{i}$ is the light termination probability at the point $i$; $T_{i}$ is the accumulated transmittance from the near plane to point $i$; $\delta_{i}$ is the distance between adjacent sampled points.

Training a NeRF with the method presented above can be time-consuming and usually takes hours for a simple scene, due to the querying cost of volume rendering on a significant number of points. More recently, some works [59, 50, 39, 9] have demonstrated super fast radiance fields with a hybrid representation, i.e. implicit for color inference and explicit for density inference. With the potential for real-time applications, we choose DVGO[59] among these works as the hybrid representation to use in this paper for the following reasons: (1) a grid-based explicit representation for the density field, which is efficient for computing volumetric information gain in the later stage; (2) the dual-stage design allows a fast convergence in the coarse stage and a high-quality reconstruction in the fine stage. This flexibility in the level of detail suits different levels of active vision task requirements; (3) competitive results and speed when compared to other hybrid representations. However, note that our proposed method can be adapted to the other NeRF methods and is not limited to DVGO only.

We introduce the core of DVGO here and refer readers to [59] for more details. DVGO speeds up NeRF by replacing the density MLP with voxel grids. In the coarse stage, two volumes store densities and colors explicitly, represented by $\textbf{V}^{(\sigma,c)}$ and $\textbf{V}^{(rgb,c)}$, respectively. A query of any 3D point $x$ for density $\sigma^{(c)}$ and color $c^{(c)}$ in the coarse stage is efficient with an interpolation:

where shift $b$ is a hyperparameter. The density volume stores a raw density $\ddot{\sigma}^{(c)}$, which is further activated via a softplus operator for the final density. Note that [59] initializes $\textbf{V}^{(\sigma,c)}$ with zero raw density to prevent suboptimal geometry.

In the fine stage, DVGO refines a higher-resolution density voxel grid $\textbf{V}^{(\sigma,f)}$ initialized with the optimized coarse geometry $\textbf{V}^{(\sigma,c)}$. The view-invariant $\textbf{V}^{(rgb,c)}$ is replaced with a feature voxel grid $\textbf{V}^{(feat,f)}$ and a shallow MLP for inferring view-dependent color emission. This hybrid implicit-explicit representation provides a good balance between speed and quality. Finally, queries of 3D points $x$ and viewing-direction $d$ at the fine stage are formulated as,

The major supervision for training DVGO models at the coarse and fine stages is the photometric loss. In the following context, we denote the DVGO models at the coarse and fine stage as $\textbf{M}^{(c)}$ and $\textbf{M}^{(f)}$ respectively.

Differing from most NeRF methods, which collect a set of images of the scene and thus optimize the radiance field offline, we create an online system that acquires and processes the observed images incrementally.

Coarse mapping: Given an observed image $I_{t}$ at time-t, we first store $I_{t}$ to a database $\{I_{i}\}$. Since the database size expands as the agent explores the environment, it is not practical to train the model with all the pixels at each iteration. Therefore, we random sample a set of pixels from $\{I_{i}\}_{i=0}^{t}$ for training $\textbf{M}^{(c)}_{t}$ at times-t. Note that rendering a pixel requires sampling points along the ray, which is time-consuming. Instead of re-sampling rays at each iteration, we only sample rays once at the beginning of the optimization until a new observation is acquired.

Fine mapping: Once the agent has completed exploring the environment, we can perform fine reconstruction if a high-quality reconstruction is required.

One of our major contributions is the proposed planning module, which considers collision avoidance, information gain, and path efficiency via a multi-objective optimization formulation. Given the current agent state (pose) $\textbf{s}_{t}$, a goal state $\textbf{s}_{g}$, and the DVGO coarse model $\textbf{M}^{(c)}_{t}$ at times-t, we aim to generate a trajectory that yields various task-dependent objectives. The trajectory is defined as a sequence of agent states $\textbf{S}=(\textbf{s}_{t},\textbf{s}_{t+1}...,\textbf{s}_{t+n},\textbf{s}_{g})$, where $\textbf{s}_{i}$ can be lie-algebra or spherical coordinates representation depending on the desirable representation in the task. This problem of finding the optimal trajectory $S^{*}$ can therefore be expressed as,

where $\mathcal{L}_{plan}$ is the overall planning objective, $\mathcal{L}_{cp}$ is the collision penalty objective, $\mathcal{L}_{ig}$ is the information gain objective, $\mathcal{L}_{pe}$ is the path efficiency objective, and $[\lambda_{cp},\lambda_{ig},\lambda_{pe}]$ are the weighting terms for each objective. Once the trajectory is optimized, we choose the furthest state $\textbf{s}^{*}_{i}$ that fulfills motion constraints as the next state. We consider a maximum translation (0.5m) and a maximum rotation angle (10) as our motion constraints.

Bézier Trajectory: Though we aim to find an optimal sequence of agent states, we do not simply make the agent states the optimizable parameters as this approach can cause infeasible agent motions such as teleporting between distanced viewpoints. To solve this issue, we use parametric curves for trajectory formation. Specifically, we adopt the Quadratic Bézier curve to create a set of discrete waypoints that define a smooth, continuous trajectory. For the Quadratic Bézier curve, the discrete points can be defined as $\textbf{s}_{i}=(1-r)[(1-r)\textbf{s}_{t}+t\textbf{s}_{c}]+r[(1-r)\textbf{s}_{c}+r\textbf{s}_{g}]$, where $r=(i-t)/(n+1)$, and $\textbf{s}_{c}$ is an optimizable Bézier control point, which is initialized as the mid-point of $\textbf{s}_{0}$ and $\textbf{s}_{g}$.

Collision Penalty: The density of the environment is represented as a continuous radiance field, which can be used as a means for computing collision penalty. Eq. 3 computes the light termination probability at a point $x$ based on the density at $x$. [1] assumes that a 3D point $x$ with high density (termination probability for light) is a strong proxy for the probability of terminating a mass particle. We can thus minimize the collision penalty by minimizing the density of the sampled waypoints. Thus, the collision penalty is formulated as

where $\sigma^{(c)}(x)$ queries the density at 3D point $x$.

Uncertainty-based Information Gain: Active mapping requires the evaluation of the quality of un-visited viewpoints – usually termed information gain. A challenge is to quantify the quality and define a meaningful criterion for evaluation, which has not been well studied with the implicit representation. We address this challenge by using termination uncertainty evaluated from each viewpoint. For each viewpoint $\textbf{s}_{i}$, we create a frustum originating from the camera center and sample points within the frustum $\textbf{F}_{i}$. Applying the same assumption of termination probability, we further compute the uncertainty of termination probability for the sampled points. Therefore, the entropy of a point $E_{x}(x)$ and a state $E_{s}(\textbf{s}_{i})$, and the information gain objective are represented by,

where $\bar{\alpha}(x)=1-\alpha(x)$. Intuitively, the viewpoint with a higher entropy means higher uncertainty about the termination probability of the associated 3D points. The entropy can be decreased once the model obtains more information about the 3D points. Thus it is more certain about the density (thus termination probability) at the sampled points.

Note that although we can directly compute $\alpha(x_{i})$ from the density volume, it comes with an initialization issue. DVGO initializes the density volume with zeros to avoid sub-optimal geometry, which means that the scene has nearly zero entropy at the beginning thus it cannot guide the agent to choose next-best view with the highest information gain. To overcome this issue, we introduce an additional entropy volume $\textbf{V}^{(E,c)}$ represented by $\alpha^{(E,c)}$ in DVGO coarse model $\textbf{M}^{(c)}$. We initialize $\alpha^{(E,c)}$ with 0.5, which represents the highest entropy. For the voxels of $\textbf{V}^{(\sigma,c)}$ that have been updated in the reconstruction stage (Sec. 4.3), we synchronize $\alpha^{(E,c)}$ with $\alpha^{(\sigma,c)}$.

Path Efficiency: For some active vision tasks, e.g. visual navigation, we want a trajectory not only to avoid potential obstacles but also with the shortest path length. To achieve path efficiency, we regularize the trajectory by minimizing the following objective, $\mathcal{L}_{pe}=\frac{1}{(n)}\sum_{i=t+1}^{t+n}||\textbf{s}_{i}-\hat{\textbf{s}}_{i}||$, where $\hat{\textbf{s}}_{i}$ is a reference state generated by linear interpolating $\textbf{s}_{0}$ and $\textbf{s}_{g}$.

In this section, we present an application of our proposed dual-stage active vision framework for the Active 3D Reconstruction task. For Active 3D Reconstruction, a robot is tasked with autonomously reconstructing a scene or object of interest. We want the agent to create a high-quality and complete 3D reconstruction autonomously without hitting obstacles in the scene while moving around. A summary of the algorithm is presented in Algorithm 1.

Initialization: We first initialize a 3D bounding box around the space to be reconstructed. Thus, a set of goal-state candidates are uniformly sampled from the view space. Note that the goal candidates are not necessarily located in empty spaces. We then initialize a DVGO coarse model and an agent state from the current state. The initial image observation is appended to the image database.

Coarse Stage: At each time step, path planning involves two steps. First, a global policy step selects the next-best-view $\textbf{s}_{g}^{*}$ from the goal candidates within the whole space, which provides a temporary destination $\textbf{s}_{g}$. To evaluate the goal candidates, we adopt the following criterion that considers density and information gain,

where $x_{\textbf{s}_{g}}$ is the 3D location of the state $\textbf{s}_{g}$ Next, we execute the path planning method introduced in Sec. 4.2 to obtain an optimized trajectory and the local next-best view $\textbf{s}^{*}$. Once the agent maneuvers to the new location and obtains a new observation, The image database is updated with the new observation. Samples are drawn from the image database for optimizing the DVGO model, including synchronizing the entropy volume $\textbf{V}^{(E,c)}$. At the end of the iteration, we check the early stop condition and stop the exploration process if the stop criteria are met. Here, we consider two stopping criteria: (1) the change in the entropy volume is below a threshold; and (2) the maximum information gain of the feasible goal candidates is below a threshold.

Fine Stage: Once the exploration is completed, we initialize the DVGO fine model with the trained coarse model. Stochastic training is performed by sampling pixels (rays) from the image database.

We evaluate our framework on synthetic and real-world datasets. Synthetic-NeRF [37] includes eight objects with realistic images. Following [37], we set the image resolution to $800\times 800$ pixels. The dataset has been divided into 100 training views and 200 novel views for testing. Tanks&Temples [25] is a real-world dataset captured by an inward-facing camera. We use a set of 5 scenes following [32]. The image resolution is $1920\times 1080$. $1/8$ of the images in each scene are used for testing.

Evaluation: There are two categories of evaluation metrics adopted in this work. For evaluating rendering performance, we adopt the metrics used in [37], including PSNR, SSIM, and LPIPS [70]. For reconstruction evaluation, we evaluate the reconstruction quality on a mesh surface where 10,000 points are sampled for evaluation. Four metrics are reported, including (1,2) Accuracy/Completeness measures the fraction of predicted/reference points that are closer to the reference/predicted points than a threshold distance, which is set to 1cm; (3) F1 score is the harmonic mean of accuracy and completeness, which quantifies the overall reconstruction quality; (4) Chamfer Distance measures the distance between nearest neighbor correspondences of two point clouds. Note that we use the DVGO models trained offline with full training set as the reference ground truth for evaluation.

View space: NeRFs [37, 59] are trained offline with a collection of images. However, as we are interested in active vision tasks, we provide images incrementally while training. We first divide our experiments into two sets, defined by the motion spaces, namely discrete space and continuous space. Discrete space: viewpoints are limited to the viewpoints in the training set. This is further divided into discrete (free-view) where the camera is allowed to teleport between any viewpoints and discrete (local) where the camera is only allowed to move to one of the three nearest viewpoints. Continuous space: we allow the camera to move freely within a predefined view space. To this end, we need to employ a simulator for rendering novel viewpoints. Since offline DVGO [59] has shown compelling rendering and reconstruction results, we employ it for a simulation purpose, which implies that DVGO is our model’s performance upper bound.

For Synthetic-NeRF, the view space is defined as a hemisphere, where the discrete experiments are on the surface of the hemisphere while the continuous experiments are within the space. For Tanks&Temples, the view space is within a 3D bounding box defined by the cameras in the training set.

Mapping optimization: In the coarse mapping stage, the model is provided with new observations incrementally in the coarse stage, and all the observations are provided in the fine stage. We sample 8,192 rays from the images to train the coarse models for 5,000 iterations and perform refinement for 20,000 iterations. New observations are inserted for every 50 iterations in the coarse stage. Adam optimizer [24] is used with an initial learning rate $10^{-}1$ for all experiments. Scene parameters, e.g. grid size, are dataset-dependent. We adopt the settings from [59].

Planning optimization: The path planning optimization is carried out with Adam optimizer [1], with a learning rate $10^{-}1$ and 100 training iterations for each planning. [$\lambda_{cp}$, $\lambda_{ig}$, $\lambda_{pe}$] are set to be [10, 1, 0.1]. For the early stop condition, we set the thresholds to be $5\times 10^{-5}$. Applying the parametric Bézier curve, we generate 100 samples along the curve for planning optimization.

We ran three times for all sets of the experiment and reported the mean result for each set. For each run, we start with a random initial location, which is shared across all sets of the experiment.

In Sec. 4.2, we propose the use of termination uncertainty (entropy) of 3D points observed by a viewpoint to determine the information gain of the viewpoint. To validate the effectiveness of the criterion, we compare the approach with some baselines and a prior work[42]. The comparison is presented in Tab. 1. Baseline (Random): a viewpoint is selected randomly as the next best view (NBV) from the viewpoint candidates. Baseline (FVS): furthest view sampling is applied to select the most distanced viewpoint compared with the current observation set as the NBV. ActiveNeRF selects NBV based on image rendering uncertainty. Few-shot active learning experiments are designed to compare the performance of the approaches. New observations are added incrementally while training, up to 10/20-views. We evaluate the learned representations based on rendering and 3D reconstruction quality. In the cases the camera is limited to local movements, our proposed entropy-based method consistently outperforms other baselines. Our method shows better performance in most of the metrics when compared to ActiveNeRF [42].

We evaluate our proposed active framework for 3D reconstruction based on rendering and reconstruction results. We compare our method against prior arts on the task of novel view synthesis. To our best knowledge, we are the first RGB-based work that trains radiance field from images incrementally acquired in a continuous space, without the need for depth data [48] or pre-training [27]. Though this online setting is more difficult than offline training, we still show competitive results when compared to the state-of-the-art methods.

We present a quantitative comparison of our method against baselines in Tab. 1(b) in Fig. 3. To compare against prior work, we compare against [22] and [27] qualitatively. The results of [22, 27] are obtained from [27]. However, as the F1 threshold is not provided in [27], we cannot compare the quantitative results fairly. From Fig. 3, we can see that our methods provide better details among the active reconstruction methods. More examples are shown in Fig. 4.

In the main paper Sec. 4.2., we have introduced a collision penalty for calculating the collision cost based on the continuous radiance field. We described the process as minimizing the density of the sampled waypoints. However, in practice, we add a safe zone. The safe zone is defined by a 3D bounding box centered at the waypoints to simulate the agent body and maintain a safe distance away from high-density regions. A set of 3D points $\textbf{B}_{i}$ centered at waypoint state $\textbf{s}_{i}$ are sampled from the 3D bounding box for computing the penalty. Thus, the full collision penalty is formulated as

where $\sigma^{(c)}(x)$ queries the density at 3D point $x$.

The safe zone setting should be agent-dependent as it should consider the agent’s size. As this work performs experiments in simulations, we do not have a specific robotic agent. Thus, we create a 3D bounding box with $1\times 1\times 1$ unit as the safe zone. 100 points are sampled uniformly from the bounding box for computing the collision penalty.

For Synthetic-NeRF, we define the view space as a hemisphere where the maximum radius is 4. We use the spherical coordinate system as the viewpoint representation. However, the spherical coordinate system only represents the 3D coordinates without rotation. Therefore, we constrain the viewpoint to be pointing towards the center of the object of interest. Sampling is performed when generating waypoints from the trajectory and generating 3D points of interest for information gain calculation. We sample 100 waypoints uniformly from the Bézier trajectory. For calculating information gain, we sampled 10 viewpoints from the waypoints to reduce the computation cost. Instead of sampling 3D points from all the rays (pixels) for information gain calculation, we only sample 3D points from $1/100$ of the pixels.

For Tanks & Temple, we define the view space as a 3D bounding box whose corners are defined from the raw training set. The viewpoint state is represented by 6DoF $se3$. The sampling process is as same as the one described above.

In Sec. 5.3, we present a study about using termination entropy as the criterion for calculating information gain. Here we present more details of the study with qualitative comparisons of individual scenes in the dataset, and quantitative comparisons of the scenes. The quantitative comparison is presented in Tab. 3 and Tab. 4 while the qualitative comparison are shown in Fig. 6 and Fig. 7.

For the quantitative result, termination entropy-based method performs consistently better than the baselines in all the scenes. The qualitative comparison shows that the entropy-based method allows the agent to explore more diverse viewpoints and builds better 3D models with limited number of viewpoints.

We present a qualitative comparison between our proposed method and DVGO [59] in Fig. 5. As same as most NeRF-based prior work [37, 20], DVGO optimizes the scene representation based on a set of collected images in an offline training manner. In contrast, we optimize the representation in an online fashion by adding new observations into the training set incrementally. The agent has to plan for the next best views and perceive new observations based on the current scene representation. Moreover, our method uses the images rendered by DVGO as the training images, which implies that DVGO serves as the “ground-truth” and the upper bound of our method.

We also include a video showing the active 3D reconstruction process.