Detail-Preserving Transformer for Light Field Image Super-Resolution

Convolutional Feature Extraction. Given the light field input $\bm{L}^{\text{LR}}$ as well as its gradient field $\bm{G}^{\text{LR}}$, two CNNs $\mathcal{E}^{\text{Cont}}$ and $\mathcal{E}^{\text{Grad}}$ are leveraged to separately extract

per-SAI embeddings $\bm{F}^{\text{Cont}}$ and $\bm{F}^{\text{Grad}}$ at spatial resolution $H\times W$ and with embedding channel $C$.

Content and Gradient Transformer. The convolutional feature embeddings (i.e., $\bm{F}^{\text{Cont}}$ and $\bm{F}^{\text{Grad}}$) capture local context within each SAI independently but lack global context across different SAIs. We use Transformers (Dosovitskiy et al. 2020) to enrich the embeddings with sequence-level context. We start by learning unimodal contextualized representations $\bm{T}^{\text{Cont}}$ and $\bm{T}^{\text{Grad}}$ using a content Transformer $\mathcal{T}^{\text{Cont}}$ and a gradient Transformer $\mathcal{T}^{\text{Grad}}$:

Note that the two transformers share the same network structure. For simplicity, we only describe $\mathcal{T}^{\text{Cont}}$ in the following.

Cross-Attention Fusion Transformer. While the content and gradient Transformers process each modality separately, we design a fusion Transformer to aggregate together their representations. To obtain more comprehensive unimodal representations, we obtain the inputs of fusion Transformer by combining all intermediate features in the content or gradient Transformers:

SAI Reconstruction. Finally, we leverages a reconstruction module $\mathcal{D}^{\text{Reco}}$ over $\bm{Z}$ to obtain a high-resolved LF $\bm{L}^{\text{HR}}$:

Here, $\mathcal{D}^{\text{Reco}}$ is separately applied to each SAI.

Remark. Our DPT employs Transformers to enrich convolutional features $\bm{F}^{\text{Cont}}$ and $\bm{F}^{\text{Grad}}$ (Eq. 2) into richer representations $\bm{H}^{\text{Cont}}$ and $\bm{H}^{\text{Grad}}$, respectively, which are further aggregated together via a fusion Transformer. In this manner, our network is able to collect informative non-local contexts within each SAI and across different SAIs, allowing for higher-quality reconstruction.

Spatial-Angular Convolutional Tokenization. ViT-like architectures (Dosovitskiy et al. 2020) leverage a linear projection layer to achieve input tokens at a very early stage. In contrast, our proposed convolutional tokenization mechanism obtains the tokens in the self-attention layer, yielding a multi-stage hierarchy like CNNs. This allows our Transformer to capture local contexts from low-level features to high-level semantic representations.

Spatial-Angular Self-Attention. The fully-connected self-attention layer is applied on $\bm{X}$ to explore the non-local spatial-angular relations among the tokens as follows:

where Attention denotes a standard self-attention layer as in (Vaswani et al. 2017; Dosovitskiy et al. 2020).

Spatial-Angular Convolutional De-Tokenization. To enable the application of convolutional tokenization in each self-attention layer, we further de-tokenize the attended feature sequence $\bm{X}^{\prime}$ to fold the patches into a large feature map $\hat{\bm{F}}$ with the same dimension as $\bm{F}$:

where $\mathcal{P}$ denotes the de-tokenization operation, and $f_{P}$ is a convolutional layer as Eq. 9. A residual layer is also used to avoid the loss of important features. Here, $\hat{\bm{F}}$ well encodes the local (in the spatial dimension) and global (in both spatial and angular dimensions) context of the input angular sequence, which could be expected to help produce better reconstruction results.

Convolutional Feature Extraction. The convolutional modules $\mathcal{E}^{\text{Cont}}$ and $\mathcal{E}^{\text{Grad}}$ (Eq. 2) share a similar network structure. Following (Wang et al. 2021d), each of them consists of two residual blocks and two residual atrous spatial pyramid pooling blocks organized in an intertwine manner, following by an angular alignment module without using deformable convolution to align the convolutional features of different views.

Transformer Networks. In the content and gradient Transformers, we set the number of attention blocks $K$ to $2$ by default. For $\mathcal{T}^{\text{Fuse}}$, Eq. 9 is slightly modified to enable the exploration of cross-attention by generating query from the content representation, while key and value from the gradient representation.

Reconstruction Module. The reconstruction module $\mathcal{D}^{\text{Reco}}$ (Eq. 8) is implemented as a cascaded of five information multi-distillation blocks, following with a upsampling layer, which consists of a pixel shuffle operation combined with two $1\times 1$ convolution layers, to generate the super-resolved SAIs (Wang et al. 2021d).

Quantitative Results. The quantitative performance comparisons are reported in Table 1. As seen, DPT achieves a promising performance in comparison with other methods. Single image super-resolution methods (i.e., VDSR (Kim, Lee, and Lee 2016), EDSR (Lim et al. 2017) and RCAN (Zhang et al. 2018)) super-resolve each view separately. DPT outperforms all of them, which can be attributed to its outstanding capability to capture the complementary information of different views.

Qualitative Results. Fig. 3 depicts some representative visual results of different approaches for $\times 4$ SR. As seen, LF-InterNet (Wang et al. 2020) and LF-DFNet (Wang et al. 2021d) produce distorted results for the structures of the telescope stand in the Bicycle scene and the flowerbed edge in the Sculpture scene. In contrast, our DPT yields much better results, with all above regions being well preserved.

Computational Analysis. Table 2 provides a detailed analysis of the LFSR models in terms of parameters, FLOPs, and reconstruct accuracy on EPFL. Following (Wang et al. 2021d), we set the size of the input LF image as $5\times 5\times 32\times 32$ for the computation of the FLOPs. As can be seen, DPT shows the best trade-off between reconstruction accuracy and model efficiency. For $\times 4$ SR, DPT has fewer parameters while better accuracy, in comparison with LF-DFNet (Wang et al. 2021d). This result further confirms the effectiveness of DPT, not only in better performance but also its efficiency.

Content Transformer. We first investigate the effect of the content Transformer by constructing a network which is implemented with the convolution feature extraction module $\mathcal{E}^{\text{Cont}}$, content Transformer $\mathcal{T}^{\text{Cont}}$ and the image reconstruction module $\mathcal{D}^{\text{Reco}}$. Its results are given in the 2nd row. We can see that the content Transformer itself can lead to a similar performance as the baseline model, however, our content Transformer has fewer parameters. This confirms that global relations among different views brought by the content Transformer are benefical to LFSR.

Gradient Transformer. Furthermore, we combine the gradient Transformer into the content Transformer, with the results being shown in the $3$rd row. As we can see, by introducing the gradient Transformer, the performance of the content Transformer (the 2nd row) improves by 0.09dB in terms of PSNR and 0.0011 improvement in terms of SSIM, respectively. Moreover, the models with dual transformers (the 4th, the 6th and the 7th rows) outperform the model with only a content Transformer (the 2nd row), which further confirms the effectiveness of gradient Transformer. Finally, we replace the content Transformer and gradient Transformer of DPT with residual blocks while maintaining the network parameters almost unchanged. Its results are given in the 5th row. The dual branches in DPT are equivalent to two convolutional neural networks for feature extraction. As reported in Table 3, DPT has a 0.21 dB PSNR drop, demonstrating the effectiveness of the proposed Transformers.

Impact of Fusion Mechanism. To explore the effect of fusion mechanism for content and gradient feature aggregation, we construct two model variants, which generates the detailed-preserved representation $\bm{Z}_{i}$ in Eq. 7 by the element-wise summation (the 3rd row) and the Transformer for a single view non-local dependencies exploration (the 4th row), respectively. As can be observed, our cross-attention fusion Transformer (the 7th row) brings a promising performance improvement over the results of 3rd row and the 4th row in PSNR and SSIM, which is attributed to its effectiveness for the complementary non-local information exploration of the content and gradient features.

Efficacy of SA-LSA. We study the effect of the SA-LSA layers by replacing them with the vanilla fully-connected self-attention layers in DPT (the 6th row). As seen, the model with SA-LSA layers (the 7th row) obtains a better performance with fewer parameters, proving the effectiveness of local spatial context for SAI reconstruction.

Number of Attention Blocks $K$. At last, we study the performance of DPT with respect to the number of spatial-angular attention blocks in the content Transformer (or the graidient Transformer). Table 4 reports the comparison results. As seen, DPT achieves the best results at $K\!=\!2$. Thus, we choose $K\!=\!2$ as the default number of spatial-angular attention blocks in DPT.