Epipolar Focus Spectrum: A Novel Light Field Representation and Application in Dense-view Reconstruction

Let $L(u,v,x,y)$ represent the distribution of rays in 3D space, where $(u,v)$ and $(x,y)$ denote the intersections between the ray with angular/camera and spatial/image planes, respectively [2, 3]. To better model the contents, several representations have been proposed in the literature. In the spatial domain, the sub-aperture image and EPI are the two most common representations. The former emphasizes the spatial information per view. The latter focuses on the disparity among views, where the slope of the EPI lines is associated with the disparity/depth. In the Fourier domain, by exploring the equivalence between the 3D focal stack and a 4D light field, Ng et al. [29] claim the 2D spectrum of a refocused image could be obtained by slicing the corresponding 4D spectrum. Dansereau et al. [30] propose the hyper-cone and hyper-fan representations, which extend the focal range of each focal slice. Le Pendu et al. [31] analyse the sparsity of light field spectrum and propose the Fourier disparity layer (FDL) representation considering the spectrum energy concentrates on several slices.

However, since these representations are highly correlated to the scene depth, the features extracted or learned from the light fields with small disparity range are unsuitable for the light fields with large disparity range, which might leads to wrong inference. In contrast, the proposed EFS is depth-invariant and thus enables an operation or processing covering the whole depth range.

Anti-aliasing requires either abundant angular samples or appropriate filters. This problem has been widely studied in both the spatial and frequency domains. In the spatial domain, Levoy and Hanrahan [2] propose a prefilter to reduce the spatial artifacts. Chang et al. [32] propose an anti-aliasing method that compensates the effect of undersampling by utilizing depth information. The technique attempts to interpolate more angular samples within each sampling interval using the depth map. Xiao et al. [33] further analyze the angular aliasing model in the spatial domain. They first detect the aliasing contents and then use lower-frequency terms of the decomposition to remove the angular aliasing at the refocusing stage. In the frequency domain, Isaksen et al. [34] first propose a frequency-planar light field filtering. Chai et al. [19] propose a comprehensive analysis on the trade-off between sampling density and depth resolution. Based on focal stacks and sparse collections of viewpoints, Levin and Durand [35] employ the focal manifold in derivations of 2D deconvolution kernels [31]. After that, Lumsdaine and Georgiev [36] discuss the aliasing in terms of the focal manifold and conclude by rendering wide depth-of-field images. By deriving the frequency domain of support of the light field, Dansereau et al. [30] present a simple, linear single-step filter to achieve volumetric focus effects.

All these methods consider the 3D focal stack as multiple individual slices and remove slice-wise aliasing contents, thus the consistency between neighboring focal slices is not preserved. Different from these methods, the proposed reconstruction method handles aliasing contents of all slices at the same time by treating the 3D focal stack as a whole, so the aliasing could be better removed and meanwhile the consistency is well maintained.

As mentioned above, the anti-aliasing operation essentially corresponds to a super-resolution operation in the angular domain [14, 13]. Existing angular super-resolution methods could be mainly divided into two categories.

The first category is based on depth estimation [7, 9, 10, 12]. Wanner et al. [11] reconstruct novel viewpoints by combining input viewpoints and estimated depth information. In order to synthesize novel views from a sparse set of input views, Kalantari et al. [14] propose two convolutional neural networks to estimate the depth and color of each viewpoint sequentially. Srinivasan et al. [12] present a machine learning algorithm that takes a 2D RGB image as input and synthesizes a 4D RGBD light field. Specifically, their pipeline consists of a CNN that estimates scene geometry, and a second CNN that predicts occluded rays and non-Lambertian effects. Subsequently, Srinivasan et al. [17] propose to utilize the MPI representation to synthesize the viewpoint from a narrow baseline stereo pair.

The second category focuses on modeling the consistency of EPI texture [22, 27, 37, 38] or the sparsity of Fourier spectrum [35, 25, 26]. Wu et al. [22] convert the angular domain reconstruction of a 4D light field into a one-dimensional super-resolution of the 2D EPI. Zhu et al. [38] further improve the super-resolution performance on EPI in large disparity areas by introducing a long-short term memory module. Considering the special 2D mesh sampling structure of the 4D light field, Levin et al. [35] utilize the 3D focus stack to complement the spectrum of the 4D light field and achieve the dense reconstruction for sparse light fields. Shi et al. [25] exploit the sparsity of the 4D light field in the continuous Fourier domain and perform the dense light field reconstruction by adopting the sparse Fourier transform. Vagharshakyan et al. [26] utilize a sparse representation of the underlying EPIs in shearlet domain and employ an iterative regularized reconstruction.

Nevertheless, these methods either rely on accurate depth calculations (especially near occlusion boundaries) or are inappropriate for large disparity scenes since the required texture lines or sparse spectrum features are not available. Differently, based on the depth-independent EFS representation, the proposed light field reconstruction method avoids the challenging depth estimation and optimize the contents at various depths with the same strategy.

For better understanding the definition of EFS, we first list the notations used in this work, as listed in Tab.I.

Given a 4D light field $L(u,v,x,y)$, $E(u,x)$ denotes its EPI when fixed $v=v^{*}$ and $y=y^{*}$ and $F(f,x)$ denotes the 2D focal stack integrated by $E_{d}(u,x)|_{d=f}$, where $(u,v)$ and $(x,y)$ refer to the angular and spatial dimensions respectively. $FT_{*}(\cdot)$ and $FT_{2D}(\cdot)$ denote the 1D and 2D Fourier transform respectively (* means specific variable). $u_{ref}$ is the reference view.

In brief, we have the following two equations,

1. 1.

   $\mathcal{E}(\omega_{u},\omega_{x})\triangleq FT_{2D}(E(u,x))$.

2. 2.

   $EFS(\omega_{f},\omega_{x})\triangleq FT_{f}(\mathcal{F}(f,\omega_{x}))$.

In this section, we first give the definition of EFS and then present two methods for constructing EFS.

According to Eq.1, $\mathcal{F}(f,\omega_{x})$ contains all spectrum contents of the EPI when $d_{min}\rightarrow-\infty$ and $d_{max}\rightarrow+\infty$. In other words, the EFS could represent the EPI losslessly in this case.

Alternatively, the EFS could also be obtained in the spatial domain according to the Fourier slice photography theorem [29]. Firstly, $E_{d}(u,x)$ denotes the sheared EPI at a specific disparity $d$,

After that, once the sheared EPI with an arbitrary disparity $f\in\left[d_{min},d_{max}\right]$ is integrated over all the views, the focal stack $F(f,x)$ is formed,

Then by performing the 2D Fourier transform on the focal stack, we can obtain the corresponding EFS representation,

Noting that, the lossless EFS could only be constructed in two cases, i.e., $\{d_{min},d_{max}\}\rightarrow\{-\infty,+\infty\}$ or the infinite aperture size [35]. However, since these conditions are practically impossible to achieve, the EFS is actually a lossy representation in practice. To minimize the effects of the missing spectrum in $\mathcal{E}(-f\omega_{u},\omega_{x})$, it is suggested to set $d_{min}$ and $d_{max}$ as the minimum and maximum disparities of the scene respectively. A detailed analysis on this issue will be given in Sec.V-C .

According to the second construction method, the EFS is equal to the Fourier spectrum of the focal stack. In this section, we first analyze the characteristics of the focal stack under different values of $f$, then discuss the attributes of EFS in the Fourier domain.

There are mainly two reasons for aliasing in the EFS, one is sparse angular sampling (Fig.4(b)(c)), the other is insufficient refocus layers (Fig.5). Since the EFS structure basically remains in the first case, here we focus on the second case caused by insufficient refocus layers, and analyze the lower bound of focal stack layers required for non-aliasing EFS recovery.

As shown in Fig.5, when the refocus range of the focal stack is equal to the depth range for all the objects in the scene, aliasing appears when the disparity gap $\Delta\alpha$ between neighboring focal layers increases. To eliminate aliasing, all lines formed from different views in the focal stack ought to be continuous instead of being discrete, i.e., $\Delta\alpha(u_{i}-u_{ref})\leq 1,~{}\forall i\in[1,N_{u}]$. All the inequalities for the middle views hold if the inequalities for two marginal views hold. In other words, the disparity gap $\Delta\alpha$ should meet the following inequality,

The refocus range $[d_{min},d_{max}]$ is set according to the scene depth range $[Z_{min},Z_{max}]$,

where $k$ is the focal length and $B$ is the baseline. Thus, the number of refocus layers $N_{f}$ in the focal stack is calculated by

Combining Eqns.9 and 11, we have

When the depth is discontinuous, it is essential to take the scene distribution into consideration. The minimum number of focal layers is estimated as

where $S(Z,O,T)$ denotes the scene distribution function, determined by the depth $Z$, occlusion $O$ and texture $T$.

In summary, the lower bound of focal stack layers is determined by the relative depth variation ($Z_{min},Z_{max}$), scene distribution $S(Z,O,T)$ and the baseline $kB$ of the synthetic aperture optical system. Fig.6 illustrates an example of choosing $N_{fmin}$, where $N_{u}=200$, $S(Z,O,T)=1$ (a continuous depth distribution), $kB=9$, $Z_{min}\in[2,10]$. The depth takes value in the interval $[1,100]$. Particularly, the black line curve shows the varying trend of $N_{fmin}$ with $Z_{min}=4$. Refer to Sec.V-C for more experimental analysis.

Specifically, We first perform shearing on an undersampled EPI to get the focal stack by Eq.4, then apply the Fourier transform on the aliased focal stack to get the aliased EFS (Eq.5). Subsequently, a CNN $\phi$ parameterized by $\sigma$ is proposed to complete the spectrum in the aliased $EFS_{ali}(\omega_{f},\omega_{x})$. The parameter $\sigma$ is optimized by

The loss function is,

where the scalar $\lambda$ is set to 1.5 for balancing the contributions of two loss terms. The second term ${loss}_{s}$ constrains the conjugate symmetry of the reconstructed EFS,

where $\!|\!\cdot\!|\!$ refers to the norm of a complex number and $*$ indicates the standard conjugate operation. $N_{f}$ and $W$ are the number of refocus layers and the width of sub-aperture image respectively.

As shown in Fig.8, a novel dual-stream U-Net is designed to deal with complex number inputs. The power spectrum and phase angle are firstly fed into two sub-networks respectively. Then the features are combined using the Euler’s formula to obtain the real and imaginary parts, which are concatenated and passed into several CNN layers for optimization. The detail of dual-stream U-Net is shown in the supplemental materials.

The Fourier Slice Imaging Theorem [29] tells a 2D slice through the origin of a 4D light field spectrum corresponds to a refocused image at a certain depth in the frequency domain. Based on this, we first apply the 1D inverse Fourier transform (IFT) of EFS along the $f$-axis,

The projection relationship between $\mathcal{F}(f,{\omega_{x}})$ and $\mathcal{E}(\omega_{u},\omega_{x})$ can be obtained via Eq.1. Thus the Fourier spectrum $\mathcal{E}_{efs}(\omega_{u},\omega_{x})$ of the reconstructed EPI could be calculated by performing a reverse projection,

Fig.9 shows the diagram of this reverse projection. Due to the limitation of disparity range, only the spectrum labeled as purple of Fig.9(b) can be reconstructed with this operation.

The next step is to apply the 2D IFT to get $E_{efs}(u,x)$ from $\mathcal{E}_{efs}(\omega_{u},\omega_{x})$. Since the interpolation operation is used during the focal stack construction, the “tailing” effects appear after the inverse Fourier slice operation, especially for the marginal viewpoints which are far away from the reference view. Hence we use the U-Net $\Psi_{\mu}$ with a perceptual loss to optimize the reconstructed EPIs ${E}_{efs}$,

where ${E}_{gt}$ represents the ground truth EPI.

The loss function for optimization is defined as follow,

where $loss_{MAE}$ is the Mean Absolute Error loss, $loss_{SSIM}$ is the Structural Similarity loss [41], and $loss_{VGG}$ is the perceptual loss[42] which is based on the VGG19 network trained on ImageNet. The scalars $\gamma_{i}$ $(i=1,2)$ are set to 3 and 5 for balancing the effects of different loss terms. The detailed structure of this network is illustrated in the supplementary materials.

The complete dense-view light field reconstruction algorithm is given in Algorithm 1. $H$ represents the height of the sub-aperture image.

In the training process, both the synthetic and real light fields are used. For the synthetic data, 12 light fields containing complex textured structures are rendered using the automatic light field generator [38, 45], of which 7 are for training and 5 for testing. Real light fields are taken from the high-resolution light field dataset provided by Guo et al. [37], of which 20 are for training and 6 for testing. In order to show the relationship between viewpoints and EFS lines, we utilize the first 200 viewpoints for experiments. Additionally, the light fields from the Disney [46] dataset are used to verify the performance of the proposed method on unseen scenes captured by a camera array. Tab.II displays the parameters of all the datasets. In the dense-view light fields, the disparity between two adjacent views for most scenes is less than one pixel, while in several scenarios, the disparity reaches two pixels.

At present, only the disparity along one single direction is concerned so that the 2D EPIs can represent the input light field. The details of several light fields under 10$\times$ and 15$\times$ downsampling scales is shown in the supplementary materials. For each light field in the experiment, by considering the disparity and the scene distribution, the EFS is constructed by performing the refocus operation for 200 times ($N_{f}=200$) where $\Delta\alpha=0.01$. The range of scene disparity determines the range of refocusing operation $f$. For details regarding the disparity, refer to our datasets and codes released later.

EPI spectrum loss analysis. In our experiments, the disparity range $d_{range}=[d_{min},d_{max}]$ is also the focal stack range. With the infinite aperture assumption, the spectrum energy could be regarded as zero for other focus layers beyond the constructed focal stack[35]. However, since the infinite aperture camera is currently not available, the EFS representation for a light field has a certain loss. Fig.10 shows the difference between the whole dense-view EPI spectrum and the EPI spectrum within a certain disparity range obtained from the EFS. Although partial contents are missing in the latter case (about 3.32$\%$), the structural information on the EPI can still be reconstructed.

To evaluate the upper bound of the EPI spectrum loss in a dense-view light field, we have counted the spectral energy loss over 10,000 EPIs which are reconstructed from the EFS containing all the scene depth ranges for several light fields and summarized the results in Fig.11. In these EPI spectra, the maximum loss is 9.03$\%$, the minimum loss is 0.31$\%$, and the average loss is 1.94$\%$. It can be seen that the EPI spectrum loss generally has a sparse ($\leq$ 5$\%$) distribution. In addition, the distribution of the loss on Disney LFs[46] is flatter and more spread out than that on synthetic[45] and real LFs[37]. We attribute this to: 1) the background texture of Synthetic LFs is relatively simple, while the texture of Disney LFs is more complex, 2) the scene depth of Synthetic data is primarily concentrated within a limited interval, while the depth distribution of Disney LFs is more divergent.

EFS reconstruction loss analysis. We try to analyze the loss in the EFS reconstruction. Some results are displayed in Fig.12, which consists of reconstructed EFS, the error map, and corresponding anti-aliasing focal stack.

By comparing Fig.4 (b)(c) with Fig.12, our method achieves a preferable performance on the EFS reconstruction for different sampling rates. The average EFS energy loss with different sampling rates on several test scenarios is list in Tab.III.

We empirically validate the influences of the range of refocus operation and the EFS sampling on the reconstructed EPI by performing the following parameter analysis. In these experiments, we use our synthetic light fields with 15$\times$ downsampling.

The range of refocus operation. The refocus range is set to 0.8$d_{range}$, $d_{range}$ and 1.2$d_{range}$, respectively. As mentioned in Sec.V-B, with 15$\times$ downsampling of the original light field, $d_{range}$ is the scene disparity range ($[15d_{min},15d_{max}]$). Fig.13 (a)(b)(c) show the qualitative comparisons with different refocus ranges on the tree root dataset. Quantitative analysis, in terms of average PSNR and SSIM, is summarized in the 2nd, 3rd and 4th rows of Tab.IV. As shown in Fig.13 (b), partial tree root has not been reconstructed. The scene structure can not be reconstructed completely when the range of refocus operation is too narrow.

The number of EFS layers $N_{f}$. The number of refocus layers $N_{f}$ is set to 104, 200 and 296, respectively. Fig.13 (a)(d)(e) show the qualitative comparisons with different numbers of EFS layers. The 2nd, 5th, and 6th rows of Tab.IV show the quantitative comparison on reconstructed light fields. It is noticed that insufficient focal layers, of which the count is smaller than the minimum focal layer count $N_{fmin}$ (see Sec.III-D), i.e., $N_{f}$=104, will cause performance degradation ($32.92/0.823$ ${vs}$ $37.54/0.952$ in terms of PSNR and SSIM respectively). In this case, aliasing appears in the focal stack, which leads to over-smooth textures in Fig.13(d). In addition, when $N_{f}$ is increased from 200 to 296, only a slight improvement in the performance is reported ($37.54/0.952$ ${vs}$ $37.98/0.961$). Hence once the number of EFS layers $N_{f}$ meets the minimum sampling rate requirement, continuously increasing EFS layers would not bring obvious improvements in the performance.

We compare our method against Wu et al. [22] and LLFF [44]. Tab.V shows the average PSNR/SSIM/LPIPS[47] measurements with different downsampling rates on both synthetic and real light fields. Qualitative comparisons between different methods on several test scenes under 15$\times$ downsampling rate are shown in Fig.14, Fig.15 and Fig.16 respectively.

As shown in Fig.17 and Fig.18, the reconstruction performance for some edge views by our method (only one center reference) has been slightly below than that of the SOTAs. This is because the center reference view does not contain enough side-view information when the baseline is large. In this section, we use multi-reference views to build Multi-EFSs to provide more information for marginal view reconstruction.

Fig.19 show the reconstructed Multi-EFSs, which is built using the 30th view ahead of the center view ($u_{ref}=u_{cen}-30$), the center reference ($u_{ref}=u_{cen}$) and the 30th view behind the center view($u_{ref}=u_{cen}+30$). The bottom row of Fig.17 and Fig.18 show the PSNR and SSIM measurements for each reconstructed view of Fig.14 and Fig.15. It can be seen that the Multi-EFSs significantly improve the overall performance of our EFS-based method, especially for edge views.

The sequential solution for full parallax is reconstructing the views containing vertical disparities after all the views containing horizontal disparities have been reconstructed, which is illustrated using an array of $17\times 17$ full parallax reconstructed from the $9\times 9$ input views (marked in black) in Fig.20. As shown in Fig.20(b), the views marked in green are firstly reconstructed in the horizontal parallax reconstruction step, and the views marked in blue are later reconstructed in the vertical parallax reconstruction step. We validate this solution on the Lego dataset[50] and show the qualitative and quantitative results in Fig.21.

Due to the lager disparity $([-9,7])$ of the Lego dataset, there exist obvious artifacts near boundary regions and discontinuous EPIs in both the results by Wu et al. [22] and LLFF [44]. Additionally, LLFF [44] usually requires a large dataset for model training, while our method is capable of learning the relationship between viewpoints and spectrum lines in the frequency domain from a relatively small training dataset. The experimental results show that our proposed method can generate clear edges and preserve cross-view consistency. Tab.VI shows the average PSNR and SSIM on the Lego dataset.

At present, the energy loss of the spectrum still exists, which may lead to unevenly colored reconstructed views. A possible solution to deal with this issue could be the space-frequency combined method, or adding a visual channel to compensate for the energy loss in the frequency domain. In addition, although the proposed method outperforms the SOTA methods on both view reconstruction quality and cross-view consistency preservation, the shearing operation may cause a moderate decline in the reconstruction performance for the marginal views (Sec.V-D). This could be addressed by introducing more reference views during the construction of the focal stack to provide more required information on the scene.