Sub-Aperture Feature Adaptation in Single Image Super-resolution Model for Light Field Imaging

Well-developed research has been achieved in the single-image super-resolution domain, leading to extraordinary performance in a single image. Images captured by the LF camera can be up-scaled by pre-trained SISR models by considering each sub-aperture image as a single image. However, it fails to exploit the angular and disparity information present in multiple SAIs. Our main objective is to propose an LF domain-specific module on top of the SISR model to achieve LFSR. Recently developed SISR models, $F_{SR}$ can be divided into two parts. One is feature extraction module $F_{feat}$, and another one is upscaling cum reconstruction module $F_{up}$. $F_{feat}$ extracts the salient features from a single image that is up-scaled by $F_{up}$. Our main objective is to introduce a module that modulates the extracted features from $F_{feat}$ by exploiting angular information across SAIs.

Using our proposed adaptation module, the pre-trained SISR model adapts the spatial and angular information present in LF images which eventually improves the super-resolution performance further, as shown in Fig. 1. The feature extraction and upscale module are from any pre-trained SISR model, and we place the adaptation module between them. Fig. 2 shows the proposed light-field domain information adaptation module. The light-field camera captures multiple SA images of the same scene. In this work, the rich information present in those angular SA images is utilized to enhance each SA image’s quality. $f_{i}$ is extracted feature set from a sub-aperture image $I_{i}$ using pre-trained feature extraction module $F_{feat}$ of a SISR model. $f_{1},f_{2},...f_{n}$ are extracted features from $n$ different sub-aperture images. Each sub-aperture feature set is processed through its corresponding Sub-Aperture Shift (SAS) module. Each SAS module is expected to process the features in such a way that it can improve the performance on the sub-aperture image $I_{i}$ task in hand. The term shift in SAS is used as it is desired that the SAS module is expected to align the features from different sub-aperture images in the direction of the SA image feature set in hand to improve the performance. Fusion block $F_{s}$ fuses the processed SA features and feeds the fused features $f_{i}^{{}^{\prime}}$ into the upscale module. During training, the weights of the pre-trained feature extraction and upscale modules are not updated during gradient back-propagation. We only update the weights of the adaptation module, as shown in Fig. 1.

SAS Module: If we consider a light-field image of $a\times a$ angular resolution, there will be $n=a^{2}$ number of SA images. Therefore, we have $n$ SAS modules for $n$ SA images. Each module takes its corresponding extracted SA features concatenated with a difference feature map. The difference feature map represents the difference between the SA feature set in hand and the SA feature set of that corresponding SAS module. The difference feature map helps to shift or modify a SA feature set in such a way that it will improve the performance of the SA feature set in hand. The difference map acts as a modulator that decides how much shift is required for a SA feature for pixel-wise mapping. The output of a SAS module can be mathematically represented as

$f_{i}$ is the extracted features of $i^{th}$ angular SA image, which will be super-resolved, and $f_{j}$ is the extracted features of $j^{th}$ angular SA image. Both $f_{j}$ and $f_{i}-f_{j}$ are concatenated before feeding into $SAS_{j}$ module. This module is expected shift features $f_{j}$ and align with $f_{i}$ using the difference map $f_{i}-f_{j}$. All the SAS modules will align the features with $f_{i}$ and feed them into the fusion module.

Fusion Module: All the modulated SA features are fused together. It is mathematically expressed as

$F_{s}$ is the fusion module and $f_{i}^{{}^{\prime}}$ is the sub-aperture informative fused feature of $i^{th}$ SA feature $f_{i}$.

Each SAS module consists of one convolution block and three consecutive residual blocks, as shown in Fig. 2. Fusion block contains two convolution layers. The first convolution layer blends all the SA features using $1\times 1$ convolution, and the second convolution layer processes the fused features. We consider the popular RDN [9] and EDSR [8] architecture of SISR for experimental purposes. In both cases, there are $64$ features in the feature set that are extracted from the feature extraction block. Therefore, we consider $C_{i}=64$ and $C_{x}=32$ in our experimental setup, as shown in Fig. 2.

We used images from $5$ publicly available LF datasets, namely EPFL [22], HCInew [25], HCIold [26], INRIA [23], and STFgantry [24]. We follow the same train-test split as given by [14]. There are a total of $144$ training images in the dataset and consider standard $5\times 5$ angular resolution for benchmark analysis. For testing, we use real-world images from EPFL [22], INRIA [23], and STFgantry [24] datasets which consists of $10$, $5$, and $2$ test images, respectively. EDSR [8] and RDN [9] are the base SISR models, where we insert our proposed LFSAFA module to adapt these models for light-field imaging. Bicubic downsampling generates low-resolution (LR) images from its high-resolution (HR) counterpart. We extract random $32\times 32$ crop from LR images during training and augment the patch using random $90^{\circ}$ rotation with a random horizontal and vertical flip. We train the LFSAFA module for $250$ epochs, and each epoch consists of $\sim 1000$ batch updates with a batch size of $4$. Adam optimizer with a learning rate $10^{-4}$ is used for updating the weights, and the learning rate is reduced by a factor of $0.5$ after every $50$ epochs. The mean absolute difference between output reconstruction and HR image is employed as a loss function. Peak signal-to-noise ratio (PSNR) and Structural Similarity Index (SSIM) are calculated on all the sub-aperture views for comparative performance analysis. Larger values on those metrics imply better reconstruction performance. Following the trend in the LFSR domain, PSNR and SSIM are calculated on the luminance channel Y of an image in YCbCr space. The code is available at https://aupendu.github.io/LFSAFA-SR.

We compare the performance of our proposed approach with state-of-the-art single image super-resolution and light field super-resolution models. VDSR [7], RCAN [10] are the SISR models, and the rest of the methods in Table 1 are the LFSR models. All the SISR models are trained on light field training images for a fair comparison. We can observe from Table 1 that both the SISR models with our proposed modification outperform all the existing techniques in terms of PSNR on EPFL and INRIA datasets. The proposed method cannot outperform other methods in the SSIM metric. However, the SSIM values are very close to the best. Fig. 3 shows the qualitative comparison of our proposed algorithm with existing LFSR approaches. We observe that our proposed algorithm achieves a more satisfactory reconstruction of numbers in the first-row image, excellently reconstructs the round holes in the second-row image, and adequately preserves the line structure, which is on the left side of the third-row image.

Table 2 shows the model ablation studies of our proposed LFSAFA module. Along with the model ablation, a study on the effect of the angular resolution is given in that table. All the components are the same for both Ablation-4 and Ablation-5 except the angular resolution of the input LF image. We can observe from the table that the network’s performance in terms of PSNR and SSIM increases as we increase the angular resolution. The proposed module LFSAFA gets more angular information as we increase the angular resolution. Therefore, it leads to better performance. This phenomenon also supports our claim that our module has the potential to explore sub-aperture angular information. Other ablations show the effectiveness of three different parts of the LFSAFA module. The first one is the residual connection between the input and output, the second one is the inclusion of difference features into each SAS module, and the third one is the contribution of the whole proposed adaptation module, LFSAFA. The only difference between Ablation-3 and Ablation-4 is the residual connection, and it shows that performance improves slightly with that connection, and we also observe that the network converges faster. Ablation-1 is basically the SISR model without the LFSAFA module. The performance does not improve much even if we add the proposed adaptation module without the difference feature, as shown in Ablation-2. Therefore, the difference feature plays a key role, and we can observe that by comparing the Ablation-2 and Ablation-4. There is a significant jump in performance metrics. Therefore, we can say that the difference feature plays a crucial role in utilizing the sup-aperture information in a controlled manner. Table 3 shows the summarized main contribution of this paper. LFSAFA module helps both the SISR models to adopt the LF domain-specific extra angular information. We can observe a significant $\sim 1dB$ improvement in PSNR metric across all the datasets and models for $2\times$ SR, and $\sim 0.6-1dB$ improvement for $4\times$ SR.