Deep Learning for Omnidirectional Vision: A Survey and New Perspectives

A normal camera has an FoV less than $180^{\circ}$ and thus captures view at most a hemisphere. However, an ideal $360^{\circ}$ camera can capture lights falling on the focal point from all directions, making the projection plane a whole spherical surface. In practice, most $360^{\circ}$ cameras can not achieve it, which excludes top and bottom regions due to dead angles¹¹1https://en.wikipedia.org/wiki/Omnidirectional_(360-degree)_camera. According to the number of lenses, $360^{\circ}$ cameras can be categorized into three types: (i) Cameras with one fisheye lens, which is impossible to cover the whole spherical surface. However, if the intrinsic and extrinsic parameters are known, an ODI can be achieved by projecting multiple images into a sphere and stitching them together; (ii) Cameras with dual fisheye lenses located at opposite positions, each of which covers over $180^{\circ}$ FoV, such as Insta360 ONE²²2https://www.insta360.com/product/insta360-one and LG 360 CAM³³3https://www.lg.com/sg/lg-friends/lg-360-CAM. This type of $360^{\circ}$ cameras have minimum demand for lenses, which are cheap and convenient, favoured by industries and customers. Images from the two cameras are then stitched together to obtain an omnidirectional image, but the stitching process might lead to edge blurring; (iii) Cameras with more than two lenses, such as Titan (eight lenses)⁴⁴4https://www.insta360.com/product/insta360-titan/. In addition, GoPro Omni⁵⁵5https://gopro.com/en/us/news/omni-is-here is the first camera rig to place six regular cameras onto six faces of a cube and its synthesized results have higher precision and less blur in edges. This type of $360^{\circ}$ cameras are professional-level.

We first define the spherical coordinate $(\theta,\phi,\rho)$, where $\theta\in(0,2\pi)$ ,$\phi\in(0,\pi)$, and $\rho$ represent the latitude, longitude, and radius of the sphere, respectively. We also define the Cartesian coordinate $(x,y,z)$. The transformation between spherical coordinate and Cartesian coordinate can be formulated as follows [18]:

where $\gamma=\sqrt{u_{t}^{2}+v_{t}^{2}}$ and $c=\tan^{-1}\gamma$. With Eqs. 2 and 3, we can build one-to-one forward and inverse mapping functions between the spherical coordinates and pixels on the tangent images [25].

Spherical stereo is about two viewpoints displaced with a known horizontal or vertical baseline [18]. Due to the spherical projection, spherical stereo is more irregular than stereo with traditional pinhole cameras. According to Eq. 1, we define the baseline as $\textbf{b}=(\delta x,\delta y,\delta z)$, and the derivative correspondence between the spherical coordinates and Cartesian coordinates can be formulated as follows:

In Eq. 4, $(\delta_{\theta},\delta_{\phi})$ represents the angular differences in the spherical coordinates $(\theta,\phi,\rho)$. According to Eq. 4, we can find that for the vertical baseline $\textbf{b}_{v}=(0,\delta y,0)$, there is no difference in $\theta$, which is simpler. However, for horizontal baseline $\textbf{b}_{h}=(\delta x,0,0)$, differences occur in both angles $\theta$ and $\phi$.

As the most common sphere-to-plane projection, ERP introduces severe distortions, especially at the poles. Considering it provides global information and takes less computation cost, Su et al. [20] proposed a representative method named Spherical Convolution, which leverages regular convolution filters with the adaptive kernel size according to the spherical coordinates. However, as shown in Fig. 4(a), the regular convolution weights are only shared along each row and can not be trained from scratch. Inspired by Spherical Convolution, SphereNet [21] proposes another typical method that processes the ERP by directly adjusting the sampling grid locations of convolution filters to achieve the distortion invariance and can be trained end-to-end, as depicted in Fig. 4(b). This is conceptually similar to those in [22], [23], as shown in Fig. 4(c) and (d). In particular, before ODIs are widely applied, Cohen et al. [29] have discussed the spatially varying distortions introduced by ERP and proposed a rotation-invariant spherical CNN approach to learn an SO3 representation. By contrast, KTN [31, 32] learns a transfer function to achieve that the convolution kernel, which is learnt from the conventional planar images, can be directly applied on ERP without retraining. In [33], the ERP is represented as a weighted graph, and a novel graph construction method is introduced by incorporating the geometry of the omnidirectional cameras into the graph structure to mitigate the distortions. [19, 34] focused on directly applying traditional 2D CNNs on CP and tangent projection, which are distortion-less.

Some methods have explored the special convolution filters in the spherical domain. Esteves et al. [36] proposed the first spherical CNN architecture, which considers the convolution filters in the spherical harmonic domain, to address the problem of 3D rotation equivariance in standard CNNs. Unlike [36], Yang et al. [35] proposed a representative framework to map spherical images into the rotation-equivariant representations based on the geometry of spherical surfaces. As shown in Fig. 5(a), SGCN [35] represents the input spherical image as a graph based on the GICOPix [35]. Moreover, it explores the isometric transformation equivariance of the graph through GCN layers. A similar strategy is proposed in [37] and [38]. In [37], the gauge equivariant CNNs are proposed to learn spherical representations from the icosahedron. By contrast, Shakerinava et al. [38] extended the icosahedron to all the pixelizations of platonic solids and generalized the gauge equivariant CNNs on the pixelized spheres. Due to a trade-off between efficiency and rotation equivariance, DeepSphere [39] models the sampled sphere as a graph of connected pixels and designs a novel graph convolution network (GCN) to balance the computational efficiency and sampling flexibility by adjusting the neighboring pixel numbers of the pixels on the graph. Compared with the methods above, another representative ODI representation is proposed in SpherePHD [27]. As shown in Fig. 5(b), SpherePHD represents the spherical image as the spherical polyhedron and provides specific convolution and pooling methods.

For image generation on ODI, there exist four popular research directions: (i) panoramic depth map completion; (ii) ODI completion; (iii) panoramic semantic map completion; (iv) view synthesis on ODI. In this subsection, we provide a comprehensive analysis of some representative works.

Ground-view, a.k.a., street-view images are usually panoramic to provide complete surrounding information, while satellite views are planar images captured to cover almost every corner of the world. There exist a few methods to synthesize ground-view images from satellite-view images. Lu et al. [54] proposed a representative work including three stages: satellite stage, geo-transformation stage, and street-view stage. The satellite stage predicts depth maps and segmentation maps from satellite images. The geo-transformation stage transforms the output of the satellite stage into the panoramas. Finally, the street-view stage predicts the street-view panoramas from the segmentation maps via a GAN. Sat2Vid [55], the first work for cross-view video synthesis, also employs three stages to generate street-view ODVs using voxel grids with semantics and depth cues transformed from satellite images with trajectory. This is conceptually similar to that in [54].

In general, the framework for geo-localization consists of two modules: cross-synthesis module and retrieval module. Shi et al. [59] proposed a representative contrastive learning pipeline to calculate the distance between the ground-view ODIs and satellite-view images in the embedding space, similar to [58, 57]. In particular, in [58], a ground-view ODI is synthesized from the polar transformation of the satellite view via a GAN, supervised by the corresponding ground-view ground truth. Meanwhile, an extra retrieval branch is applied to constrain the latent representations of two domains. Using conditional GANs, Regmi et al. [57] skillfully synthesized the satellite-view image from the ground-view ODI. To learn a robust satellite query representation, they fused the features from the satellite-view synthesis and ground-view ODI, and then matched the query feature with satellite-view features in the embedding space. As the latest work, TransGeo [60] is the first ViT-based framework to extract the position information from the satellite images and ground-view ODIs. With an attention mechanism, TransGeo removes uninformative patches in the satellite-view images and surpasses previous CNN-based methods.

Compared with conventional perspective images, omnidirectional data records richer geometrical information with a higher resolution and wider FoV, making it more challenging to achieve effective compression. The early approaches for ODI compression directly utilize the existing perspective methods to compress the perspective projections of the ODIs. For instance, Simone et al. [67] proposed an adaptive quantization method to solve the frequency shift in the viewport image blocks when projecting the ODI to the ERP. By contrast, OmniJPEG [68] first estimates the region of interest in the ODI and then encodes the ODI based on the geometrical transformation of the region content with a novel format called OmniJPEG, which is an extension of JPEG format [69] and can be viewable on legacy JPEG decoders. Considering the ERP distortion, a graph-based coder is proposed by [70] to adapt the sphere surface. To make the coding progress computationally feasible, the graph partitioning algorithm based on rate distortion optimization [71] is introduced to achieve a trade-off between the distortion of reconstructed signals, the signal smoothness on each sub-graph, and the coding cost of partitioning description. As a representative CNN-based ODI compression work, OSLO [72] applies HEALPix [73] to define a convolution operation directly on the sphere and adapt the standard CNN techniques to the spherical domain. The proposed on-the-sphere representation outperforms the similar learnable compression models on the ERP.

For ODV compression, Li et al. [74] proposed a representative work aiming to optimize the ODV encoding progress. They analyzed the distortion impacts of restoring spherical domain signals from the different planar projection types and then applied the rate distortion optimization based on the distortion of signal in spherical domain. Similarly, Wang et al. [75] proposed a spherical coordinates transform-based motion model to address the distortion problem in projections. Another representative method  [76] maps the ODV to the rhombic dodecahedron (RD) map and directly applies the planar perspective videos encoding methods on the RD map. Specifically, the rate control-based algorithms are proposed to achieve better qualities and smaller bitrate errors for ODV compression [77], [78]. Zhao et al. [78] utilized game theory to find optimal inter/intra-frame bitrate allocations while Li et al. [77] proposed a novel bit allocation algorithm for ERP with the coding tree unit (CTU) level. Similar to [20], the CTUs in the same row have the same weight to reduce the distortion influence.

Illumination recovery is widely employed in many real-world tasks ranging from scene understanding, reconstruction to editing. Hold-Geoffroy et al. [79] proposed a representative framework for outdoor illumination estimation. They first trained a CNN model to predict the sky parameters from viewports of outdoor ODIs, e.g., sun position and atmospheric conditions. They then reconstructed illumination environment maps for the given test images according to the predicted illumination parameters. Similarly, in [80], a CNN model is leveraged to predict the location of lights in the viewports, and the CNN is fine-tuned to predict the light intensities, i.e., environment maps, from the ODIs. In [81], geometric and photometric parameters of indoor lighting are regressed from the viewports of ODI, and the intermediate latent vectors are used to reconstruct the environment maps. Another representative method, called EMLight [82], consists of a regression network and a neural projector. The regression network outputs the light parameters, and the neural projector converts the light parameters into the illumination map. In particular, the ground truths of the light parameters are decomposed by a Gaussian map generated from the illumination via a spherical Gaussian function.

Existing Head-Mounted Display (HMD) devices [83] require at least the ODI with 21600$\times$10800 pixels for immersive experience, which can not be directly captured by current camera systems [84]. One alternative way is to capture low resolution (LR) ODIs and super-resolve them into high resolution (HR) ODIs efficiently. LAU-Net [85], as the first work to consider the latitude difference for ODI SR, introduces a multi-level latitude adaptive network. It splits an ODI into different latitude bands and hierarchically upscales these bands with different adaptive factors, which are learned via a reinforcement learning scheme. Beyond considering SR on the ERP, Yoon et al. [28] proposed a representative work, SphereSR, to learn a unified continuous spherical local implicit image function and generate an arbitrary projection with arbitrary resolution according to the spherical coordinate queries. For ODV SR, SMFN [86] is the first DNN-based framework, including a single-frame and multi-frame joint network and a dual network. The single-frame and multi-frame joint network fuses the features from adjacent frames, and the dual network constrains the solution space to find a better answer.

The standard approach of upright adjustment follows two steps: (i) estimating the position of the pole of the ODI; (ii) applying a rotation matrix to align the estimated north pole. The early representative work [87] estimates the camera rotation according to the geometric structures in the panoramas, e.g., curving straight lines and vanishing points. However, these methods are limited to the Manhattan [88] or Atlanta world [89] assumption and rely on necessary prior knowledge of geometric structures. Recently, DL-based upright adjustment has been widely studied. Without any specific assumption on the scene structure, DeepUA [90] proposes a representative CNN-based framework to estimate the 2D rotations of multiple NFoV images sampled from the ODI and then estimate the 3D camera rotation through the geometric relationship between 3D and 2D rotations. By contrast, Deep360Up [91] directly takes ERP image as the input and synthesizes the upright version according to the estimated up-vector orientation. In particular, Jung et al. [92] proposed a two-stage pipeline for ODI upright adjustment. First, the feature map is extracted by a CNN model from the rotated ERP image. The feature map is then mapped into a spherical graph. Finally, a GCN is applied to estimate the 3D camera rotation, which is the location of the point on the spherical surface corresponding to the north pole.

Due to the ultra-high resolution and sphere representation of omnidirectional data, visual quality assessment (V-QA) is valuable for the optimization of exiting image/video processing algorithms. We next introduce some representative works on ODI-QA and ODV-QA, respectively.

For the ODI-QA, according to the availability of the reference images, it can be further classified into two categories: full-reference (FR) ODI-QA and no-reference (NR) ODI-QA. In exiting methods on FR ODI-QA, some works focus on extending the conventional FR image quality assessment metrics, e.g., PSNR and SSIM, to the omnidirectional domain, e.g., [95], [96]. These works introduce special geometric structures of the ODI and its projection representations to traditional quality assessment metrics and measure the objective quality more accurately. In addition, there are a few DL-based approaches for FR ODI-QA. As the representative work shown in Fig. 6(a), Lim et al. [93, 97] proposed a novel adversarial learning framework, consisting of a quality score predictor and a human perception guider, to automatically assess the image quality following the human perception. NR ODI-QA, also called blind ODI-QA, predicts the ODI quality without expensive reference ODIs. Considering multi-viewport images in the ERP format, Xu et al. [98] applied a novel viewport-oriented GCN to process the distortion-less viewports in ERP images and aggregated these features to estimate the quality score via an image quality regressor. A similar strategy is applied in [99, 100]. By contrast, [2] extracted the features from CP images and their corresponding eye movement (EM) and head movement (HM) hotspot maps and provided a good projection-based potential, that is extracting the features from the multiple projection formats and fusing the features to improve the performance on blind ODI-QA, as shown in Fig. 6(b).

For the ODV-QA, Li et al. [94] proposed a representative viewport-based CNN approach, including a viewport proposal network and a viewport quality network, as shown in Fig. 6(c). The viewport proposal network generates several potential viewports and their error maps, and viewport quality network rates the V-QA score for each proposed viewport. The final V-QA score is calculated by the weighted average of all viewport V-QA scores. [101] is another representative that considers the temporal changes of spatial distortions in ODVs and fuses a set of spatio-temporal objective quality metrics from multiple viewports to learn a subjective quality score. Similarly, Gao et al. [102] modeled the spatial-temporal distortions of ODVs and proposed a novel FR objective metric by integrating three existing ODI-QA objective metrics.

Compared with the perspective images, DL-based object detection on ODIs remains two main difficulties: (i) traditional convolutional kernels are weak to process the irregular planar grid structures in the ODI projections; (ii) the criterias adopted in conventional 2D object detection do not fit well to the spherical images. To address the first difficulty, distortion-aware structures are proposed, e.g., multi-scale feature pyramid network in [105], multi-kernel layers in [106]. However, the detection flows of these two methods are similar to the methods for 2D domain, which take the whole ERP image as input and predict the regions of interest (ROIs) to obtain the final bounding boxes. Considering the wide FoV of ERP, Yang et al. [107] proposed a representative framework, which can leverage the conventional 2D images to train a panoramic detector. The detecting progress consists of three sub-steps: stereo-projection, YOLO detectors, and bounding box post processing. Especially, they generated four stereographic projections with a $180^{\circ}\times 180^{\circ}$ FoV from an ERP and the four result maps predicted by the YOLO detectors. Finally, the sub-window detected bounding boxes are re-projected to the ERP and re-aligned into the final distortion-less bounding boxes.

To tackle the second difficulty, a novel kind of spherical bounding boxe (SphBB) and spherical Intersection over Union (SphIoU) for ODI object detection are introduced in [103], as shown in the first row of Fig. 7. SphBB is represented by the coordinates $\theta$, $\phi$ of the object centers and the unbiased FoVs $\alpha$, $\beta$ of the objective occupation. SphIoU is similar to planar IoU and calculated by the IoU between two SphBBs. Concretely, FoVBBs are moved to the equator that is undistorted. Similarly, Cao et al. [104] proposed a novel IoU calculation method without any extra movement, called FoV-IoU. As shown in the second row of Fig. 7, FoV-IoU better approximates the exact computation of IoU between two FoV-BBs compared with the SphIoU.

DL-based omnidirectional semantic segmentation has been widely studied because ODI can encompass exhaustive information about the surrounding space. There are many practically remaining challenges, e.g., distortions in the planar projections, object deformations, computed complexity, and scarce labeled data. We next introduce some representative methods for ODI semantic segmentation via supervised learning and unsupervised learning.

Due to the lack of real-world datasets, Deng et al. [114] firstly generated ODIs from an existing dataset of urban traffic scenes and then designed an approach, called zoom augmentation, to transform the conventional images into fisheye images. Meanwhile, they proposed a CNN-based framework with a special pooling module to integrate the local and global context information and handle complex scenes in the ODIs. Considering that CNNs have inherently limited ability to handle the distortions in ODIs, Deng et al. [115] proposed a method, called Restricted Deformable Convolution, to model geometric transformations and learn a convolutional filter size from the input feature map. Zoom augmentation was also applied to [115] for enriching the train data. As the first framework to conduct semantic segmentation on the real-world outdoor ODIs, SemanticSO [116] builds a distortion-aware CNN model using the equirectangular convolutions [117].

Due to the time-consuming and expensive cost of ground truth annotations for ODIs, endeavours have been made to synthesize ODI datasets from the conventional images and utilize knowledge transfer to adopt models directly trained with the perspective images. PASS [118] is the first work to bypass fully dense panoramic annotations and aggregate features represented by conventional perspective images to fulfill the pixel-wise segmentation in panoramic imagery. Based on the PASS, DS-PASS [119] further re-uses the knowledge learned from perspective images and adapts the model learned from the 2D domain to panoramic domain. Meanwhile, in DS-PASS, the sensitivity to spatial details is enhanced by implementing attention-based lateral connections to perform segmentation accurately. To reduce the domain gap between the ODI and perspective image, Yang et al. [111] proposed a representative cross-domain transfer framework that designs an efficient concurrent attention network to capture the long-range dependencies in ODI imagery and integrates the unlabeled ODIs and labeled perspective images into training. A similar strategy was applied in [120], [121] and [109]. Particularly, in [109], a shared attention module is used to extract features from the 2D domain and panoramic domain, and two domain adaption modules are used to ”teach” the panoramic branch by the perspective branch. For unsupervised semantic segmentation, there also exist some works considering the geometric structure of ODI [108]. For instance, Zhang et al. [45] proposed an orientation-aware CNN framework based on the icosahedron mesh representation of ODI and introduced an efficient interpolation approach of the north-aligned kernel convolutions for features on the sphere.

Thanks to the emergence of large-scale panoramic depth datasets, monocular depth estimation has evolved rapidly. As shown in Fig. 9, there are several trends: (i) Tailored networks, e.g., distortion-aware convolution filters [108] and robust representations [123]; (ii) Different projection types of ODIs [19], [130], [25], as depicted in Fig. 9(a), (b); (iii) Inherent geometric priors [131], [126], as shown in Fig. 9(c); (iv) Multiple views [18] or pose estimation [128], as shown in Fig. 9 (d), (e), respectively.

It has been demonstrated that directly applying the DL-based methods for 2D optical flow estimation on ODI will obtain the unsatisfactory results [137]. To this end, Xie et al. [138] introduced a small diagnostic dataset FlowCLEVR and evaluated the performance of three kinds of tailored convolution filters, namely the correlation, coordinate and deformable convolutions, for estimating the omnidirectional optical flow. The domain adaptation frameworks [139, 140] benefit from the development of optical flow estimation in the perspective domain. Similar to [137], OmniFlowNet [139] is built on FlowNet2 and the convolution operation is inspired by [117]. Especially, as the extension of [141], LiteFlowNet360 [140] uses kernel transformation techniques to solve the inherent distortion problem caused by the sphere-to-plane projection. A representative pipeline is proposed by [142], consisting of a data augmentation method and a flow estimation module. The data augmentation method overcomes the distortions introduced by ERP, and the flow estimation module exploits the cyclicity of spherical boundaries to convert long-distance estimation into a relatively short-distance estimation.

Compared with the methods for 2D video summarization, only a few works have been proposed for ODV summarization. Pano2Vid [143] is the representative framework that contains two sub-steps: detecting candidate events of interest in the entire ODV frames and applying dynamic programming to link detected events. However, Pano2Vid requires observing the whole video and is less capable for video streaming applications. Deep360Pilot [42] is the first framework to design a human-like online agent for automatic ODV navigation of viewers. Deep360pilot consists of three steps: object detection to obtain the candidate objects of interest, training RNN to choose the important object, and capturing exciting moments in ODV. AutoCam [144] generates the normal NFoV videos from the ODVs following human behavior understanding. An similar strategy was applied by Yu et al. [145]. They built a deep ranking model for spatial summarization to select NFOV shots from each frame in the ODV and generated a spatio-temporal highlight video by extending the same model to the temporal domain. Moreover, Lee [146] proposed a novel deep ranking neural network model for summarizing ODV both spatially and temporally.

As the indoor panoramas can cover wider surrounding environment and capture more context cues than conventional perspective images, they are beneficial to scene understanding and widely applied into room layout estimation and reconstruction. Zou et al. [13] summarized that the general procedure of layout estimation and reconstruction contains three sub-steps: edge-based alignment, layout elements prediction, and 3D layout elements recovery, as shown in Fig. 10. The representative work, proposed by Zhang et al. [147], conducts the first DL-based pipeline for holistic 3D scene understanding that recovers 3D room layout and detailed information, e.g., shape, pose, and location of objects from a single ODI. In [147], a context-based GNN is designed to predict the relationships across the objects and room layout and achieves the SoTA performance on both geometry accuracy of room layout and 3D object arrangement.

For the alignment, this pre-possessing step provides indoor geometric information as the prior knowledge to ease the network training. Several SoTA approaches [148, 149, 150] follow the ”Manhattan world” assumption, in which all walls are aligned with a canonical coordinate system, and the floor plane direction is estimated by selecting long line segments and voting for the three mutually orthogonal vanishing directions. In contrast, AtlantaNet [151] predicts the 3D layout from less restrictive scenes that are not limited to ”Manhattan World” assumption. AtlantaNet follows ”Atlanta World” assumption and projects an gravity-aligned ODI into two horizontal planes to predict a 2D room footprint on the floor plan and a room height to recover the 3D layout.

For the layout element prediction, the primary task is to estimate layout boundaries and corner positions. On the one hand, the related methods usually choose different projections of ODIs as the input. For instance, some methods [149, 152, 153] predict the layout only from ERP images. Besides the ERP, Yang et al. [148] added a perspective ceiling-view image, which is obtained from the ERP through an equirectangular-to-perspective (E2P) conversion, as an extra input. They then extracted the features from the two formats by a two-branch network and fused the two-modal features to predict the layout elements. The advantage of [148] is that it directly uses the multi-projection model to jointly predict a Manhattan-world floor plan instead of estimating the number of corners. On the other hand, recent methods varied in their ways of feature representation. For instance, HorizonNet [152] represents the room layout of the ODI as three 1D embedding vectors and recovers 3D room layouts from 1D predictions with low computation cost. Differently, Wang et al. [154] converted the layout into ’horizon-depth’ through ray casting of a few points. This transformation maintains the simplicity of layout estimation and improves the generalization capacity to unseen room layouts.

For final recovery, the general strategy [149, 148, 152] is to reconstruct the layout by the optimization of mapping each pixel between walls and corners. In particular, it defines the weighted loss of probability maps of floors, ceilings, and corners. The major difficulty is the layout boundary occlusions when the camera position is not ideal for the entire display. To address this problem, HorizonNet [152] observes the occlusions by examining the orientation of the first Principal Component Analysis (PCA) component of adjacent walls and recovers occluded parts according to the long-term dependencies of global geometry.

Human binocular disparity depends on the difference between the projections on the retina, that is, a sphere projection rather than a planar projection. Therefore, stereo matching on the ODIs is more similar to the human vision system. In  [155], they discussed the influence of omnidirectional distortion on the CNN-based methods and compared the quality of disparity maps predicted from the perspective and omnidirectional stereo images. The experimental results show that stereo matching based on the ODIs is more advantageous for numerous applications, e.g., robotics, AR/VR, and several other applications. General stereo matching algorithms follow four steps: (i) matching cost computation, (ii) cost aggregation, (iii) disparity computation with optimization, and (iv) disparity refinement. As the first DNN-based omnidirectional stereo framework, SweepNet [156] proposes a wide-baseline stereo system to compute the matching cost map from a pair of images captured by cameras with ultra-wide FoV lenses and uses a global sphere sweep at the rig coordinate system to generate an omnidirectional depth map directly. By contrast, OmniMVS [157] takes four 220^∘ FoV fisheye views as the input to train an end-to-end DNN model and uses a 3D encoder-decoder block to regularize the cost volume. The method proposed in [158], as the extension of OmniMVS, provides a novel regularization of cost volume based on the uncertainty of prior guidance. Another representative work, 360SD-Net [159], is the first end-to-end trainable network for omnidirectional stereo depth estimation with the top-bottom ODI pairs as the input. It mitigates the distortion in the ERP images through an additional polar angle coordinate input and a learnable cost volume.

SLAM is an intricate system that adopts multiple cameras, e.g., monocular, stereo, or RGB-D, combined with sensors onboard a mobile agent to reconstruct the environment and estimate the agent pose in real-time. SLAM is often used in real-time navigation and reality augmentation, e.g., Google Earth. The stereo information, such as key points [160] and dense or semi-dense depth maps[161], is indispensable to build an accurate modern SLAM system. Specifically, compared with traditional monocular SLAM [162] or multi-view SLAM [163], the omnidirectional data can provide the richer texture and structure information due to a large FoV, and the omnidirectional SLAM avoids the influence of discontinued frames in the surrounding environment and enjoys the technical advantage of complete positioning and mapping. Caruso et al. [164] proposed a representative monocular SLAM method for omnidirectional cameras in which the direct image alignment and pixel-wise distance filtering are directly formulated. Zachary et al. [165] proposed a general framework that accepts multiple types of sensor data and is capable of iterative updates of camera pose and pixel-wise depth. DeepFactors [166] performs joint optimization of the pose and depth variables to detect the loop closure. As the omnidirectional data has rich geometry and texture information, further works may consider how to cultivate the full potential of DL and utilize these imaging advantages to construct a fast and accurate SLAM system.

Recently, there have been several research trends in ODI saliency prediction, building on DL progress: (i) From 2D traditional convolutions to 3D specific convolutions; (ii) From single feature to multiple features; (iii) From single ERP input to multi-type inputs; (iv) From normal CNN-based learning to novel learning strategies. In Table. VI, numerous DL-based methods have been proposed for ODI saliency prediction. In the following, we introduce and analyze some representative networks, as shown in Fig. 11.

(i) To directly apply 2D deep saliency predictors on ODIs and reduce the unsatisfactory distortion in ODIs, many works [177, 167] convert ODIs into 2D projection format. As the first attempt of DNNs on ODI saliency prediction, SalNet360 [177] subdivides an ERP into a set of six CP patches as the input because CP avoids the heavy distortions near the poles like ERP. Then SalNet360 combines predicted saliency maps and per-pixel spherical coordinates of these patches to output a resulting saliency map in ERP format. Differently, a few works [178, 168] propose the ODI-aware convolution filters for saliency prediction, and learn the relationships between the features from a non-distorted space. The representative work, SalGCN [168], transfers the ERP image to a spherical graph signal representation, generates the spherical graph signal representation of the saliency map and finally reconstructs the ERP format saliency map through the spherical crown-based interpolation. SalGFCN [179] proposes a SoTA method that is composed of a residual U-Net architecture based on the dilated graph convolutions and attention mechanism in the bottleneck. (ii) The viewports are the rectangular windows on ERP with different narrow FoVs caused by observers’ head movement. Due to less distortions in viewports, some works [180, 181] choose a set of viewports on ERP as the input and extract the multiple independent features from these viewports. The final omnidirectional saliency map is generated by a set of viewport saliency maps and refined via an equator biased post-processing. Different from most prior multi-feature works extracting the low-level geometric features, Mazumdar et al. [180] introduced a 2D detector to find important objects first, and this kind of local information can improve the performance of the overall saliency map. Recently, Chao et al. [169] utilized three different FoVs in each viewport to extract rich salient features and better combined the local and global information. Furthermore, stretch weighted maps are applied in the loss function to avoid the disproportionate impact of stretching in the north and south poles of the ERP image.

(iii) ODI saliency prediction methods with multi-type inputs focus on the projection transformations of the ODIs, which has been mentioned in the Sec 2.1. These methods aim to utilize the properties of different projection formats to achieve the better performance than the single ERP input [177, 182, 183]. Due to the geometric distortions in the poles of ERP format, Djemai et al. [182] introduced a set of CP images, which are projected by five different rotational ERP images into the CNN-based approach. However, boundary-distortion and discontinuity in CP images cause the lack of global information in the extracted features. To address the problem, SalBiNet360 [183] simultaneously takes ERP and CP images as the input. It constructs a bifurcated network to predict global and local saliency maps, respectively. The final saliency output is the fusion of the global and local saliency maps. Furthermore, Zhu [184] provided a groundbreaking multi-domain model, which decomposes the ERP image using spherical harmonics in the frequency domain and combines frequency components with multiple viewports of the ERP images in the spatial domain to extract features.

(iv) As the first to use GAN to predict the saliency maps for ODIs, SalGAN360 [185] provides a new generator loss, which is designed according to three evaluation metrics to fine-tune the SalGAN [186]. SalGAN360 constructs a different branch with the Multiple Cubic Projection (MCP) as input to simulate undistorted contents. For the attention-based learning on ODI saliency prediction, Zhu et al. proposed RANSP [187] and AAFFN [188]. Both methods contain the part-guided attention (PA) module, which is a normalized part confidence map that can highlight specific regions in the image. Moreover, an attention-aware module is introduced to refine the final saliency map. Especially, RANSP predicts the head fixations while AAFFN predicts the eye fixations.

Gaze following, also called gaze estimation, is related to detecting what people in the scene look at and are absorbed in. As normal perspective images are NFoV captured, gaze targets are always out of the scene. ODI gaze following is proposed to solve this problem because ODIs have a great ability to capture the entire viewing surroundings. Previous 3D gaze following methods can directly detect the gaze target of a human subject in the sphere space but ignore scene information of ODIs, which performs gaze following not well. Gaze360 [191] collects a large-scale gaze dataset using fish-eye lens rectification to pre-process the images. However, due to the distortion caused by the sphere-to-plane projection, the gaze target maybe not be in the 2D sightline of the human subject in long-distance gaze, which is no longer the same in 2D images. Li et al. [192] proposed the first framework for ODI gaze following and also collected the first ODI gaze following dataset, called GazeFollow360. They detected the gaze target within a local region and a distant region. For ODI gaze prediction, Xu et al. [193] built a large-scale eye-tracking dataset for dynamic 360^∘ immersive videos and gave a detailed analysis of gaze prediction. They utilized the temporal saliency, spatial saliency and history gaze path for gaze prediction with a combination of CNN and LSTM, which is similar to the architecture proposed by [194].

Because ODVs can provide the observers with an immersive understanding of the entire surrounding environments, recent research focuses on audio-visual scene understanding on ODVs. Due to its enabling viewers to experience sound in all directions, the spatial radio of ODV is an essential cue for full scene awareness. As the first work on the omnidirectional spatialization problem, Morgado et al. [195] designed a four-block architecture applying self-supervised learning to generate the spatial radio, given the mono audio and ODV as the joint inputs. They also proposed a representative self-supervised framework [196] for learning representations from the audio-visual spatial content of ODVs. In [197], ODIs combined with the multichannel audio signals are applied to localize sound source object within the visual observation. The self-supervised training method includes two DNN models: one for visual object detection and another for sound source estimation. Both DNN models are trained based on variational inference. Vasudevan et al. [198] simultaneously achieved an audio task, spatial sound super-resolution, and two visual tasks, dense depth prediction, and semantic labeling of the scene. They proposed a cross-modal distillation framework, including a shared encoder and three task-specific decoders, to transfer knowledge from vision to audio. For the audio-visual saliency prediction on ODVs, AVS360 [199] is the first end-to-end framework with two branches to understand audio and visual cues. Especially, AVS360 considers geometric distortion in ODV and extracts the spherical representation from the cube map images. Furthermore, as the first user behavior analysis for audio-visual content in ODV, Chao et al. [200] designed the comparative studies using ODVs with three different audio modalities and demonstrated that audio cues can improve the audio-visual attention in ODV.

Visual question answering (VQA) is a comprehensive and interesting task that combines computer vision (CV), natural language processing (NLP), and knowledge representation $\&$ reasoning (KR). Wider FoV ODIs and ODVs are more valuable and challenging for the VQA research because they can provide stereoscopic spatial information similar to the human visual system. VQA 360^∘, proposed in [201], is the first VQA framework on ODI. It introduces a CP-based model with multi-level fusion and attention diffusion to reduce spatial distortion. Meanwhile, the collected VQA 360^∘ dataset provides a benchmark for future developments. Furthermore, Yun et al. [6] proposed the first ODV-based VQA work, Pano-AVQA, which combines information from three modalities: language, audio, and ODV frames. The fused multi-modal representations extracted by a transformer network provide a holistic semantic understanding of omnidirectional surroundings. They also provided the first spatial and audio-VQA dataset on ODVs.