Light Field Image Quality Assessment with Auxiliary Learning based on Depthwise and Anglewise Separable Convolutions

Unlike traditional photography, which records the 2D projection of light rays, an LFI describes light rays from multiple directions. This technology allows capturing richer visual information due to the higher dimensional representation of the data [12]. The theoretical root of LFI is interpreted by a plenoptic function, which is usually denoted as $l_{\lambda}(x,y,z,\theta,\phi,\lambda,t)$ where $l_{\lambda}[W/m^{2}/sr/nm/s]$ represents spectral radiance per unit time, with $(x,y,z)$ for the spatial position, $(\theta,\phi)$ for light ray directions, $\lambda$ for wavelength, and $t$ is a temporal instance [12]. With a series of background assumptions, each point in a light field can be denoted as a 4D coordinate $(u,v,x,y)$ where $(x,y)$ represents the spatial position and $(u,v)$ indicates the angular position [12]. From the digital imaging perspective, each pixel in an LFI can be located with two 2D coordinates: an angular coordinate identifying the located subview in the LFI collection and a spatial coordinate indicating the exact pixel position in that subview similar to a traditional 2D image.

In summary, although many IQA metrics have been proposed for the NR-LFIQA in last few decades, the majority of them are not intended for LFI and thus they often ignore to take advantage of the LFI’s angular dimension. Several new NR-LFIQA metrics, including BELIEF and NF-LFQA, have yielded generally acceptable results. They are, however, inept in dealing with some forms of severe distortions such as EPICNN, resulting in instability in IQA prediction with large variations. As stated in Section II-C1, DSC has been shown to be a high-performance and computationally efficient feature extractor even when the input dimension is limited to three dimensions (i.e., the pixel coordinates and a color channel). Thus, it is interesting to investigate the adaptation of DSC from conventional 2D images to LFI in order to substantially decrease the computational cost associated with the high dimensionality. Additionally, the idea of separable convolution may potentially be extended to the angular dimension in the context of LFI for further improvement in feature extraction. Moreover, as mentioned in Section II-C2, the inclusion of auxiliary tasks may assist the model in mastering the primary task, which is NR-LFIQA in our case. Based on the above observations and analysis, we are inspired to present an auxiliary learning based model with effective LF-ASC and LF-DSC feature extractors, which is introduced in next section.

The ALAS-DADS framework is visualised in Fig. 1 (a). The framework consists of two stride-1/stride-2 LF-DSC blocks and three LF-ASC blocks that are trained via auxiliary learning. The walkthrough of the proposed model can be summarised into the following process:

1. 1.

   Data augmentation is conducted to increase the dataset’s size by eightfold. (see Section IV-A2).

2. 2.

   Normalization, i.e., for an LFI $n$:

   $$\hat{p_{n}}(D)=\frac{p_{n}(D)-\mu(D)}{\sigma(D)+1},\forall D=(u,v,x,y,c)$$

   (1)

   where $(u,v)$ identifies a subview and $(x,y,c)$ represents a pixel and its color channel. $\mu$ and $\sigma$ are mean and standard deviation of the corresponding pixel channel.

3. 3.

   A subview-wise stride-2 2D convolution is performed to reduce the spatial size.

4. 4.

   Three LF-ASCs are performed to extract angular as well as spatial features efficiently (see Section III-C).

5. 5.

   A stride-4 max pooling is applied to decrease the spatial size.

6. 6.

   Two stride-1/stride-2 LF-DSCs are performed for efficient spatial and angular feature extraction (see Section III-B).

7. 7.

   A pointwise convolution with the following stride-2 max pooling is applied to combine the channel features.

8. 8.

   The model outputs the NR-LFIQA score with auxiliary learning after fully connected neural networks (FC-NN) (see Section III-D).

To give more model details, TABLE I lists the filter shapes as well as the input size of each model layer. The input shape is $7\times 7\times 434\times 434\times 3$, and the model body will extract $7\times 7\times 3\times 3\times 1024$ features to feed in the prediction module. The module includes an FC-NN to predict the quality scores as the primary task, and two double 256-node layers to estimate spatial features and angular features, respectively.

As stated in Section II-C1, the DSC was initially developed for 2D images, which handle not only spatial dimensions but also depth dimensions (i.e., the number of channels). It converts a normal convolution to a depthwise convolution, then performs a $1\times 1$ convolution (i.e., pointwise convolution) to combine the outputs from the preceding layers. The primary difference between DSC and regular convolution is that the DSC separates the filtering features and combines the outputs into two layers instead of one step. Due to the fact that this factorization lowers the computing dimensions involved, it significantly reduces the computational complexity and the model size. To achieve these benefits, we expand the DSC into the spatial domain of the LFI to resolve the curse of dimensionality in LFI. Specifically, each subview in an LFI is treated as a 2D image that is fed into a depthwise convolution followed by a pointwise convolution. Fig. 1 (c) illustrates the internal structure of the LF-DSC. It includes a pointwise convolution, an LF-DSC, and a second pointwise convolution for combining the channel features. Suppose a subview-wise 2D convolution takes an $u_{i}\times v_{i}\times x_{i}\times y_{i}\times c_{i}$ input tensor $T_{i}$ with kernel $K_{j}\in R^{k_{j}\times k_{j}\times c_{i}\times c_{j}}$, we can calculate the computational cost as:

The proposed LF-DSC can achieve similar performance but only takes:

Therefore, by adopting the LF-DSC, we can save the computational cost of:

Lastly, we have two types of LF-DCS, including the stride-2 LF-DSC, which reduces the spatial dimensions and enriches the channel dimensions; and the stride-1 LF-DSC, which remains the input dimensions but applies residual shortcuts for more efficient training.

LF-DSC is mainly designed to concentrate on spatial feature extraction. To supplement the angular quality information missed by LF-DSC, we theoretically extend LF-DSC to the angular domain of LFIs and introduce the novel concept of LF-ASC. Since LF-ASC is designed for the distinctive structure of LFIs, it makes extensive use of the extra angular information to achieve high-performance feature extraction at a low computational cost. Fig. 1 (b) illustrates the structure of an LF-ASC module. LF-ASC divides a one-step 4D convolution with kernel $K_{j}\in R^{a_{j}\times a_{j}\times k_{j}\times k_{j}\times c_{i}\times c_{j}}$ into two lower-dimensional convolutions: a horizontal $H_{j}\in R^{a_{j}\times k_{j}\times k_{j}\times c_{i}\times c_{j}}$ and a vertical $V_{j}\in R^{a_{j}\times k_{j}\times k_{j}\times c_{i}\times c_{j}}$. Hence, we succeed in reducing multiplications from one 4D convolution with $a_{j}^{2}$ to two 3D convolutions with $2a_{j}$ in total to achieve comparable feature extraction performance.

Suppose a standard 4D convolution takes a $u_{i}\times v_{i}\times x_{i}\times y_{i}\times c_{i}$ input tensor $T_{i}$ with kernel $K_{j}\in R^{a_{j}\times a_{j}\times k_{j}\times k_{j}\times c_{i}\times c_{j}}$, we can calculate the computational cost as:

The proposed LF-ASC can obtain similar performance but reduce the computational cost significantly with:

The proposed LF-DSC and LF-ASC together can perform efficient feature extraction only at a cost of:

Therefore, by adopting the combination of the LF-DSC and LF-ASC instead of a 4D convolution, we can significantly reduce the computational cost by:

To further improve the performance, we configure spatial feature estimation and angular feature estimation as two auxiliary tasks that support the primary NR-LFIQA task. The primary task shares the same hidden layers and weights with the two auxiliary tasks to take both spatial and angular quality into consideration. Nonetheless, each task has its own FC-NN to be optimised before the output.

For the labels of the auxiliary tasks, natural scene statistics (NSS) features from BRISQUE are adopted as the spatial quality features [22] due to their reliability (as discussed in Section II-B1), while the global direction distribution (GDD) features are used as the angular quality features according to the state-of-the-art NR-LFQA (as discussed in Section II-B2). Since BRISQUE is designed for 2D images, we take the average NSS features of all subviews as the labels for an LFI. The primary task is output with a single FC-NN while each auxiliary task uses a global average pooling to shrink the dimensions, followed by a double 256-node FC-NN to optimize the loss function.

We leverage the mean squared error (MSE) as the loss function for each task. The loss function of the primary task is formulated as:

where $N$ is the total number of input LFIs, $y_{i}$ denotes the true quality score, and $\hat{y_{i}}$ represents the predicted score.

The loss function for the auxiliary task of spatial feature estimation can be formulated as:

where $s_{ij}$ stands for each element from a spatial feature vector $S_{i}$ while $\hat{s_{ij}}$ is the corresponding predicted value. Since we adopted a $36\times 1$ NSS feature vector from BRISQUE, the dimension of the spatial feature $D_{s}$ is $36$.

The loss function for the auxiliary task of angular feature estimation can be formulated as:

where $a_{ij}$ captures each element from an angular feature vector $A_{i}$ while $\hat{a_{ij}}$ is the corresponding predicted value. Since we adopted an $8\times 1$ GDD feature vector from the state-of-the-art NR-LFQA, the dimension of angular feature $D_{a}$ is 8.

The total training loss of the ALAS-DADS is defined as:

where $\lambda$ is the balancing factor between the main and the auxiliary tasks. In related work such as [33, 34, 35, 36], the typical values for $\lambda$ are 1, 0.1 and 0.01. In practice, we trained our framework with 1, 0.1, 0.01, and 0.001, and we chose 0.01 according to the experimental results. Additionally, we normalised the labels of the quality features for the auxiliary activities in order to scale the loss from three sources (i.e., quality score, spatial quality features, and angular quality features) to a comparable level and guarantee the effectiveness of the factor $\lambda$.

Therefore, the framework outputs a predicted quality score, the estimated spatial quality features, and the estimated angular quality features in response to an LFI. The determination of the quality scores is dependent on the labelling of the dataset on which the model is trained (see Section IV-A1). Moreover, we use natural scene statistics (NSS) (i.e., a $36\times 1$ vector) from BRISQUE as spatial quality features and global direction distribution (GDD) (i.e., an $8\times 1$ vector) from the state-of-the-art NR-LFQA as angular quality features.

We believe that our proposed techniques, such as LF-DSC and LF-ASC, and the related auxiliary learning scheme, can be easily adapted to other LFI processing tasks, such as LFI super-resolution, LFI image classification, and LFI depth estimation. First, similar to how DSC is used in a variety of 2D computer vision applications [38, 39, 40], the LF-DSC and LF-ASC can be used as highly efficient generative LFI feature extractors for these tasks. Second, the ALAS-DADS architecture as a whole can be easily modified to perform various LFI processing tasks. For example, Lu et al. [41] developed an LFI classification model that contains a VGGNet with an interleaved CNN refiner. One potential adaptation is to substitute the interleaved CNN in this classification model with LF-DSC and LF-ASC for more efficient feature extraction. Besides, another potential adaptation is to adopt ALAS-DADS as the framework replacing the VGGNet since VGGNet is designed for 2D images. In this instance, the primary task would be LFI classification, while the auxiliary tasks could include estimating the saliency map, which may provide useful hints to the primary task.

We implemented the proposed models with Python Tensorflow, and trained them with the hardware configurations of an AMD Ryzen 7 3700X CPU, two NVIDIA RTX2070S GPUs, and 64 GB RAM. Since LFI data is high-dimensional, it is infeasible to load many LFIs simultaneously for training due to limited memory resources. Instead, we design a dedicated training scheme for efficiently processing the high dimensional LFI data. The datasets are firstly split into training and testing segments at the ratio of $0.8$ and $0.2$. We then use stochastic learning for training and validation. For each tiny training batch:

• •

  $m$ training images are randomly drawn from the training dataset without replacement.

• •

  $n$ validation images are randomly drawn from the training dataset with replacement.

• •

  For each epoch, the model monitors the validation performance and will early stop and switch to the next batch if:

  • –

    The performance did not improve for several epochs $p$ (i.e., patience).

  • –

    Or it reaches the epoch limit $l$ of a tiny training batch.

Under this training scheme, it is possible to tune the training parameters to control the training process. For example, the epoch limit $l$ can be configured as a relatively small number (e.g., 5) to avoid overfitting. For optimization, Adam with AMS gradient [47] is applied, which stores “long-term memory” of past gradients. A new variant of the Adam algorithm is performed to solve the convergence issues and improve the performance. Since we have two GPUs, we implemented a parallel training scheme to improve the training efficiency. Due to the communication loss, Tensorflow’s default distributed framework does not take advantage of multi-GPU computing. As a result, we train a model independently on two GPUs and update both with the model with the lowest validation loss in every 100 batches. Finally, as shown in TABLE II, it took 6343 seconds for training the model on Win5-LID and 14881 seconds on SMART.

The deep learning-based methods are often questioned with overfitting problems. To address this issue, we use residual shortcut and dropout for regularization of the CNN in addition to our data augmentation for more generalization. More importantly, we split the dataset by training and testing at the beginning. During the whole training and validation process, only the training segment is involved, while all the experimental results are based on the independent test segment without retraining. The statistics of the training process from TABLE II prove that the ALAS-DADS model gains an excellent trade-off between overfitting and underfitting: the training loss is slightly less than the validation loss while either of them is less than the test errors.

We performed an ablation study prior to the benchmarking experiments to demonstrate the efficacy and learning efficiency of LF-DSC and LF-ASC in comparison to 4D convolution. We developed four testing models: 10-4D-Conv, 10-LF-DSC, 10-LF-ASC, and 10-LF-DSC-ASC. Except for the backbone, the construction of the four models is identical. For example, ten four-dimensional convolution layers form the basis of 10-4D-Conv. Similarly, the 10-LF-DSC, 10-LF-ASC, and 10-LF-DSC-ASC configurations include ten LF-DSCs, ten LF-ASCs, and ten LF-DSC LF-ASC combinations, respectively. The models are then evenly trained on Win5-LID using 100 batches of 5 epochs each. Each batch contains two LFIs for training and two LFIs for validation. TABLE III summarises the results of the ablation study. There are two things to be noted: 1) the training times in the table include data loading but not the evaluation time; and 2) the performance measures (i.e., RMSE, SROCC, and PLCC) are computed using the testing data. The number of trainable parameters in 10-4D-Conv is much larger than those in the other three models. While 10-LF-DSC-ASC includes a total of twenty layers, including ten LF-DSCs and ten LF-ASCs, it nevertheless has a much smaller number of trainable parameters and training duration than 10-4D-Conv. Additionally, despite the shorter learning period, the 10-LF-DSC and 10-LF-ASC beat the 10-4D-Conv, with the 10-LF-DSC-ASC achieving the best performance. Thus, the results of this ablation study confirms the theoretical conclusion we made in Section III that LF-DSC and LF-ASC can attain higher NR-LFIQA performance at a reduced computational cost.

To validate the proposed metric’s performance, we conduct an experiment comparing it to traditional metrics (i.e., PSNR, SSIM, and BRISQUE) and the state-of-the-art LFI metric (i.e., NR-LFQA). It is worth to mention that we acquire the source code of NR-LFQA from the original authors and retrain the model using identical experimental conditions. Because PSNR and SSIM are both FR-IQA metrics, we use the original undistorted images in both datasets. Apart from that, we use the average quality score of each subview for the predictions of BRISQUE.