Moving in a 360 World:
Synthesizing Panoramic Parallaxes from a Single Panorama

Generating novel panoramic views with only one omnidirectional image of the scene is a challenging task. Previous perspective-based methods show promising results on scene reconstruction and new viewpoints rendering by using multi-view data with known camera parameters. Our idea is to adapt a similar process to our scenario by simulating multi-view images from the single RGB-D panorama image. We first produce a set of 3D points from the given RGB-D panorama and then reproject these 3D points into multiple panoramas that correspond to different virtual camera locations. The generated omnidircetional images are likely to be imperfect as there might be gaps and cracks between pixels due to occlusion or limited resolution. OmniNeRF solves this problem by taking advantage of the pixel-based prediction property of its MLP model, which takes a single pixel rather than an entire image as the input. By considering valid pixels only, we are able to augment our training set given that we have a sparse point cloud of the scene derived from the auxiliary depth map of the input RGB-D panorama.

We project the current panorama into 3D coordinates by the following procedure. First, all pixels can be projected to a uniform sphere by their 2D coordinates. For a pixel $(x,y)$ on the panorama, its vertical and horizontal viewing angles can be defined by $\theta=\pi y/H$, $\phi=2\pi x/W$, where $H$ and $W$ are the height and width of the panorama. The coordinate center would be the current camera position, namely the ray origin. Likewise, a ray direction simply means a unit vector from the center to the sphere. A novel panoramic view can therefore be determined by moving the camera to a new position and examining what would be sampled on the new sphere by the emitted rays based on the above equations. Not all pixels are supposed to be visible from the new viewpoint. Moreover, the sparse source input might wrongly allow a ray to pass through some occluding object, which causes a visibility problem needed to be solved. (We will come back to address this issue in the next section.) Assuming that we have removed the false visible pixels, we then project the points from the new sphere back to the original pose to obtain the final training image. This transformation is crucial because the scene coordinates are defined by the original source image; the coordinate frame should be kept consistent across all camera poses. As a result, the key of our data augmentation mechanism is to verify which parts of the ground truth will be visible to a given ray origin.

To fully sample the scene, our strategy is to transform views along each axis. We uniformly sample the camera poses along x-axis and y-axis, within a range of ${[\mathrm{Coord}_{\mathrm{min}},\mathrm{Coord}_{\mathrm{max}}]}$ multiplied a scaling factor $\lambda\in[0,1]$. The range is divided into equal intervals to collect the training views. A smaller $\lambda$ means that samples would be closer to the center. A larger $\lambda$ guarantees better performance while moving farther away from the original center, but the number of applicable pixels would also decrease. We set $\lambda=0.6$ to balance between quality and displacement. All experiments are trained on $100$ incomplete panoramas derived from a single input panorama and its auxiliary depth map.

To produce self-simulated multi-viewpoint panoramic images, we transform an image from the original camera pose to any desired pose. One critical issue is the ambiguity of ray visibility from new viewpoints due to the sparsity of the projected 3D points from only a single image. More specifically, a ray from a translated camera view could “see through” the sparse points of an obstacle and reach a 3D point visible to the original view (see the left-most image in Fig. 3 for an illustration). To mask out the “see-through” rays, we first apply a median filter on the depth map of the translated view. Note that only valid pixels on the depth map should be considered. We then simply filter out pixels whose depth values are larger than the local median depth multiplied by a tolerance ratio (which is set to $1.3$ in this work). With this simple modification, we can remove most of the incorrect “see-through” samples. A visualization demonstrating the effectiveness of our filtering strategy is shown in Fig. 3.

Although our method only needs one input panorama, it can also combine multiple omnidirectional images once their relative camera positions are known. Since Matterport3D dataset [3] is the only dataset with multi-view panoramas, we could only apply this experiment on Matterport3D dataset.

Our initial attempt with the basic setting occasionally suffers from the artifacts of blurry edges, which might come from forcing the model to predict the uncertain regions of the scene. The given training data are not dense enough to cover all areas in the scene; therefore, to render images at new positions would force the model to predict some regions that the model has never seen before. We introduce an additional loss term to improve our model’s performance regarding color gradient prediction. The key of OmniNeRF is to predict unseen pixels between neighboring samples. The model should be able to learn to interpolate from one pixel to another according to ray origin and direction information. Inspired by recent depth estimation methods, e.g. [9], we include a gradient loss term to enforce the structure-preserving property for color prediction. We use a Laplacian filter to obtain the gradient of the ground truth. The gradient loss can help produce smoother color prediction as well as reduce artifacts. However, we are not able to directly compute the output gradient because the input pixels are shuffled (due to 3D reprojection) and thus the neighbor ordering is not maintained. Instead, we add one more head which is parallel to color output in the MLP model to predict gradient. Gradient contains information about neighboring pixels and thus can improve the quality of generated images near boundaries.

The purpose of an implicit neural representation model is to learn a mapping between 3D coordinates and RGB color space. At each discrete sample on the ray ${\mathbf{r}}(t)={\mathbf{o}}+t{\mathbf{d}}$, where ${\mathbf{o}}$ and ${\mathbf{d}}$ denote the ray origin and ray direction, the final RGB values $C({\mathbf{r}})$ are optimized from aggregation of color $c_{i}$ and opacity $\sigma_{i}$. A positional encoding technique [29] is applied to rays for capturing high frequency information. The function of color composition follows the rule in volume rendering [13]:

	$\displaystyle\hat{\mathcal{C}}({\mathbf{r}})=\sum_{i=1}^{N}T_{i}\left(1-\exp(-\sigma_{i}\delta_{i})\right)c_{i},$		(1)
	$\displaystyle\text{where}~{}T_{i}=\exp\left(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}\right),$

and $\delta_{i}=t_{i+1}-t_{i}$ is the interval between two adjacent samples. The overall volume sampling principles are done in a hierarchical way: a ‘coarse’ and a ‘refined’ stage. The coarse and refined networks are identical except the process of sampling pixels on a ray. At the coarse stage, $N_{c}$ intervals are uniformly sampled alone the ray, while at the refined stage, $N_{f}$ intervals are decided in accordance with densities from the coarse stage. These two predictions would be optimized by the ground-truth color respectively. The overall loss is the sum of two terms: the color loss and the gradient loss. The color loss is simply the $L_{2}$ loss between the observed color in Eq. (1) and the ground truth. Both the coarse and refined models participate in the optimization process for the composite color. Gradient is aggregated by the same principle as the color term; its loss is also the total squared error between the estimated results and the ground truth.

We test our method on both synthetic and real-world datasets. In this work, all the panorama images are under equirectangular projection at the resolution of $512\times 1024$.

For each dataset, we discard scenes that are not feasible for our application, i.e., more than $10\%$ of the pixels do not have depth values. Since our task is to generate novel panoramic views from a single panorama, consequently, the only supervision to the model comes from those pixels with depths, and it is not applicable to learn from a scene with too many missing depths. For Matterport3D, we exclude the polar regions (corresponding to ceiling and floor in most cases) as the RGB-D information is unavailable.

We compare the detailed settings of the objective function for the proposed OmniNeRF in Table 1. Our augmented method significantly improve the performance to a plausible level. Learning with additional gradient supervision further uplifts the resemblance between the generated image and the ground truth. Gradient information carries supervision for local structure-preserving, and it also generates more fine details in the scene. See Fig. 5 for the visualization.

Some panoramas in Matterport3D [3] have view overlap. The distance between cameras can be computed with the provided camera extrinsic. We select two nearest frame in the same scene to train on one panorama and test on the other. A qualitative comparison is shown in Fig. 7.

We encourage the reader to see the supplementary video demonstration for a fly-through experience provided by our method from just a single panorama.

To examine the limitation of the proposed OmniNeRF for scene coverage from only a single panorama, we render images at a broader range of camera viewpoints and check the quality of the outcomes. Specifically, we manually draw a path to render images at some sample positions in a scene. Some representative results on the three datasets with diverse scenes are visualized in Fig. 6. For irregular or complicated scenes, the single source image does not contain enough information to infer the occlusion and the opposite side of objects, and thus the results are degraded. Our model shows better results for simpler scenes with less occlusion, and the range it can render is therefore broader.