Multi-Level Feature Fusion Network for Lightweight Stereo Image Super-Resolution

Recovering HR images from LR images is the aim of image SR, deep learning-based methods achieve this goal by learning the mapping relationship between a large number of LR images and their corresponding HR images. Since Dong et al. [9] initially suggested using CNN to achieve image SR task, many deep learning-based methods have emerged, demonstrating excellent performance. Kim et al. [16] further improved the effectiveness of CNN in image SR by increasing the depth of the network. By adding dense [41, 36] and residual [19, 40] connections, researchers have optimized the information flow and feature reuse amongst deep neural networks, thus increasing models’ robustness and training speed. However, images contain rich multi-level information, and different information contributes differentially to the image SR task. In order to focus more on the important features and structural information in the image, Zhang et al. [40] proposed the channel attention mechanism. Since then, various attention mechanisms [8, 26, 7, 24, 23, 32, 22] have been proposed as effective means to enhance the performance of image SR.

Recently, Transformer has achieved significant success in the field of computer vision, with Transformer-based SR models achieving state-of-the-art (SOTA) results. However, these models often have a large number of parameters, which limits their practical use. Additionally, single image super-resolution (SISR) can only utilize information within the image itself, without fully exploiting complementary information from other images, thereby restricting further enhancement in potency.

Stereo image SR can make use of information across views to further enhance the resolution effect. Jeon et al. [14] proposed the first deep learning-based stereo image SR algorithm, StereoSR, which uses parallax prior and a two-stage joint network enhance the spatial resolution of stereo images. Song et al. [29] suggested a Self and Parallax Attention Mechanism (SPAM) to recover HR features while preserve stereo consistency between image pairs. In order to leverage texture-rich single image datasets, Ying et al. [38] developed a generic Stereo Attention Module (SAM) to extend SISR network to a stereo image SR network. Xu et al. [37] incorporated the idea of bilateral grid processing into a CNN framework to effectively utilize cross-view information. Wang et al. [35] utilized a Bi-directional Parallax Attention Module (BiPAM) to simultaneously interact with information from both left and right perspective images. Additionally, they addressed the issue of inconsistent illumination between left and right perspective images in real-world scenes by improving the loss function. Chu et al. [5] used a stack of NAFBlock [2] for intra-view feature extraction and combined it with stereo cross-attention modules for cross-view feature interaction, resulting in excellent performance. Zou et al. [44] improved upon their work by designing the CVHSSR, which effectively conveys mutual information between different views. In addition, Transformer-based methods [4, 20, 1] have begun to be applied in the field of stereo image SR, achieving impressive outcomes.

However, the above methods often focus solely on performance while neglecting the potential for application in downstream tasks. To address this issue, we design a lightweight stereo image SR network, redefining the processes of intra-view feature extraction and cross-view feature interaction to enhance the efficiency of the network.

The network proposed by us is illustrated in Figure 2. MFFSSR employs a dual-branch network with shared weights to restore images from both left and right perspectives. It consists of three parts: shallow feature extraction, deep feature extraction and interaction, and stereo image reconstruction. The Multi-Level Feature Fusion Block (MFF Block) is the core component of the deep feature extraction and interaction, which consists of two Hybrid Attention Feature Extraction Blocks (HAFEBs) and an embedded Cross-View Interaction Module (CVIM). Detailed information about HAFEB and CVIM will be presented in Section 3.2 and Section 3.3, respectively. Specifically, the operation process of MFFSSR is as follows.

Firstly, given a pair of LR stereo images $I_{L}^{LR},I_{R}^{LR}\in{R^{H\times W\times 3}}$, a simple convolutional operation is used for them to extract the shallow features $F_{L}^{S},F_{R}^{S}\in{R^{H\times W\times C}}$, where H, W, and C represent the image’s height, width, and number of channels, respectively. This process can be described as:

$$F_{L,R}^{S}={H_{{\rm{conv}}}}(I_{L}^{LR},I_{R}^{LR})$$

(1)

where ${H_{{\rm{conv}}}}$ denotes $3\times 3$ convolution operation.

Next, we perform deep feature extraction and interactive fusion using MFF Block on the acquired features. The number of MFF Blocks, denoted by N, is flexible and can be adjusted. This process can be described as:

$$F_{L,R}^{D}=H_{{\rm{MFF}}}^{N}(H_{{\rm{MFF}}}^{N-1}(\cdot\cdot\cdot(H_{{\rm{MFF}}}^{1}(F_{L,R}^{S}))))$$

(2)

$$F_{L,R}^{i+1}=H_{{\rm{MFF}}}(F_{L,R}^{i})$$

(3)

where ${H_{{\rm{MFF}}}}$ denotes MFF Block. $F_{L,R}^{D},F_{L,R}^{i+1}$ denote the features after deep extraction and interactive fusion and the features obtained after processing by the $i$-th MFF Block, respectively.

Finally, we utilize the pixel shuffling operation to upsample the output features to the HR size. Furthermore, a global residual structure is used to maintain input image features and increase the performance of SR. This process can be described as:

$$I_{L}^{SR}=H_{up}(F_{L}^{D})+H_{up}(I_{L}^{LR})$$

(4)

$$I_{R}^{SR}=H_{up}(F_{R}^{D})+H_{up}(I_{R}^{LR})$$

(5)

where $H_{up}$ denotes upsampling operation. $I_{L}^{SR}$ and $I_{R}^{SR}$ represent the final left and right perspective images after SR, respectively.

Stereo images contain information spanning global, local, and cross-view ranges. Intra-view feature extraction serves as the foundation for cross-view interaction. To efficiently capture and fuse these multi-level features, we introduce the Hybrid Attention Feature Extraction Block (HAFEB).

As shown in Figure 3, the HAFEB consists of two components: (1) Multi-level Feature Extraction and Fusion (MFEF) and (2) Information Refinement Feedforward (IRF). In addition to channel attention and large kernel attention mechanisms, we also employ reparameterized convolution (RepConv) in MFEF. Their comprehensive use significantly enhances the capability and flexibility of HAFEB in feature extraction. Furthermore, we use branch structures to interact partial intra-view features with the embedded Cross-View Interaction Module (CVIM), therefore reducing computational complexity and substantially improving the efficiency of cross-view information fusion.

Given an input tensor $F{\rm{in}}\in{R^{H\times W\times C}}$, the working process of MFEF can be described as follows:

	$\displaystyle{F_{{\rm{MFEF}}}}$	$\displaystyle=H_{{\rm{pconv}}}^{2}({H_{{\rm{LKA}}}}(H_{\rm{f}}(\kappa(H_{{\rm{pconv}}}^{1}(LN({F_{{\rm{in}}}}))))$		(6)
		$\displaystyle+\kappa(H_{{\rm{pconv}}}^{1}(LN({F_{{\rm{in}}}})))))+{F_{{\rm{in}}}}$

where $LN(\cdot)$ denotes layer normalization. $H_{{\rm{pconv}}}^{(\cdot)}$, $H_{{\rm{f}}}$, $H_{{\rm{LKA}}}$ represent $1\times 1$ point-wise convolution, feature fusion operation, and large kernel attention, respectively. ${F_{{\rm{MFEF}}}}$ is the output feature of MFEF. We use notation $\kappa(\cdot)$ to represent hybrid feature fusion extraction operation. Specifically, given the input feature $X\in{R^{H\times W\times C}}$ , it is firstly split into two parts ${X_{1}}\in{R^{H\times W\times\theta}}$, $X_{2}\in{R^{H\times W\times(1-\theta)}}(\theta\in[0,1])$ on channel dimension. $\lambda$ is a hyperparameter that controls the ratio. We set $\theta=0.75$ to balance between the parameters and efficiency. More detailed information about it can be found in the ablation study in Section 4.3. Then, $X_{1}$ and $X_{2}$ are used for further feature extraction and cross-view information interaction, respectively. This process can be described as:

	$\displaystyle\kappa(X)$	$\displaystyle={\delta_{{\rm{SG}}}}({H_{{\rm{rconv}}}}(X_{1}))\odot{H_{{\rm{CA}}}}({\delta_{{\rm{SG}}}}({H_{{\rm{rconv}}}}(X_{1})))$		(7)
		$\displaystyle+{F_{{\rm{CVIM}}}}(X_{2})$

where ${H_{{\rm{rconv}}}}$ and ${F_{{\rm{CVIM}}}}$ represent the RepConv and the output feature of CVIM, respectively. ${\delta_{{\rm{SG}}}}$ means SimpleGate function and $\odot$ denotes element-wise multiplication.

The internal computation process of large kernel attention and channel attention can be described as follows:

$${H_{{\rm{LKA}}}}(X)=X\odot({H_{{\rm{pconv}}}}({H_{{\rm{dd7}}}}({H_{{\rm{d5}}}}(X))))$$

(8)

$${H_{{\rm{CA}}}}(X)=X\odot({H_{{\rm{pconv}}}}({H_{{\rm{Avg}}}}(X)))$$

(9)

where ${H_{{\rm{Avg}}}}$, ${H_{{\rm{d5}}}}$ and ${H_{{\rm{dd7}}}}$ represent average pooling operation, $5\times 5$ depth-wise convolution, and $7\times 7$ depth-wise dilation convolution respectively.

Then, IRF employs a non-linear gate mechanism to focus on complementary details across different levels. The working process of IRF can be described as follows:

	$\displaystyle{F_{{\rm{out}}}}$	$\displaystyle=H_{{\rm{pconv}}}^{4}({\delta_{{\rm{NG}}}}(H_{{\rm{d3}}}^{1}(H_{{\rm{pconv}}}^{3}(LN({F_{{\rm{MFEF}}}})))))$		(10)
		$\displaystyle+{F_{{\rm{MFEF}}}}$

where $H_{{\rm{d3}}}^{\left(\cdot\right)}$ and ${\delta_{{\rm{NG}}}}$ represent $3\times 3$ depth-wise convolution and non-linear gate function, respectively. The ${F_{{\rm{out}}}}$ denotes the output feature of HAFEB.

We refine the Cross-View Interaction Module (CVIM) proposed in CVHSSR [44]. Redundant cross-view feature interaction has little contribution to SR performance improvement but can lead to a significant increase in computational complexity. To improve cross-view interaction efficiency and reduce parameters, we constrain the input feature in dimension. Additionally, layer normalization is removed for better integration into the Hybrid Attention Feature Extraction Blocks. The details of CVIM is as shown in Figure 4. It combines Scaled DotProduct Attention [30], which utilizes queries and keys to generate corresponding weights:

$$Attention({\bf{Q}},{\bf{K}},{\bf{V}})=softmax({\bf{Q}}{{\bf{K}}^{T}}/\sqrt{C}){\bf{V}}$$

(11)

where ${\bf{Q}}\in{R^{H\times W\times C}}$ is the query matrix from one view, and ${\bf{K}},{\bf{V}}\in{R^{H\times W\times C}}$ are key and query matrices to another view.

CVIM efficiently facilitates the interaction between left and right view information. Given the input stereo partial intra-view features $X_{2L}^{i},X_{2R}^{i}\in{R^{H\times W\times C}}$, we can get the cross-view fusion features $X_{2L\to R}$ through the following process:

$${{\bf{Q}}_{\rm{L}}}=H_{d3}^{{Q_{L}}}(H_{{\rm{pconv}}}^{{Q_{L}}}(X_{2L}^{i}))$$

(12)

$${{\bf{K}}_{\rm{R}}}=H_{d3}^{{K_{R}}}(H_{{\rm{pconv}}}^{K_{R}}(X_{2R}^{i}))$$

(13)

$${{\bf{V}}_{\rm{R}}}=H_{d3}^{{V_{R}}}(H_{{\rm{pconv}}}^{V_{R}}(X_{2R}^{i}))$$

(14)

$$X_{2L\to R}=H_{{\rm{pconv}}}^{R}Attentio{n_{{\rm{L}}\to{\rm{R}}}}({{\bf{Q}}_{\rm{L}}},{{\bf{K}}_{\rm{R}}},{{\bf{V}}_{\rm{R}}})$$

(15)

$X_{2R\to L}$ can be obtained through the similar process:

$${{\bf{Q}}_{\rm{R}}}=H_{d3}^{{Q_{R}}}(H_{{\rm{pconv}}}^{{Q_{R}}}(X_{2R}^{i}))$$

(16)

$${{\bf{K}}_{\rm{L}}}=H_{d3}^{{K_{L}}}(H_{{\rm{pconv}}}^{{K_{L}}}(X_{2L}^{i}))$$

(17)

$${{\bf{V}}_{\rm{L}}}=H_{d3}^{{V_{L}}}(H_{{\rm{pconv}}}^{{V_{L}}}(X_{2L}^{i}))$$

(18)

$$X_{2R\to L}=H_{{\rm{pconv}}}^{L}Attentio{n_{{\rm{R}}\to{\rm{L}}}}({{\bf{Q}}_{\rm{R}}},{{\bf{K}}_{\rm{L}}},{{\bf{V}}_{\rm{L}}})$$

(19)

The cross and intra view features are finally fused to generate the output features $X_{2L}^{i+1}$ and $X_{2R}^{i+1}$:

$$X_{2L}^{i+1}={\gamma_{L}}X_{2L\to R}+X_{2L}^{i}$$

(20)

$$X_{2R}^{i+1}={\gamma_{R}}X_{2R\to L}+X_{2R}^{i}$$

(21)

where ${\gamma_{L}}$ and ${\gamma_{R}}$ are trainable channel-wise scales and initialized with zeros for stabilizing training.

The loss function defines the optimization objective of the SR network and plays a crucial role in determining how well it performs. Zou et al. have already demonstrated the effectiveness of utilizing spatial and frequency domain losses to jointly guide the SR network for image restoration [44].

Specifically, we use the MSE loss to measure the spatial structural difference between the SR images $I_{L,R}^{SR}$ and the HR images $I_{L,R}^{HR}$, which can be described as:

$${L_{MSE}}={1\over N}{\sum\limits_{i=1}^{N}{\left\|{I_{L,R}^{HR}-I_{L,R}^{SR}}\right\|}^{2}}$$

(22)

Additionally, frequency Charbonnier loss is introduced to guide the learning of high-frequency information in SR images, aiding in better preservation of details and textures. It can be defined as:

$${L_{FC}}={1\over N}\sum\limits_{i=1}^{N}{\sqrt{{{\left\|{FFT(I_{L,R}^{HR})-FFT(I_{L,R}^{SR})}\right\|}^{2}}+{\varepsilon^{2}}}}$$

(23)

where $\varepsilon$ is a constant and is set to ${10^{-3}}$. $FFT\left(\cdot\right)$ denotes the fast Fourier transform.

In conclusion, the overall loss function can be expressed as:

$${L_{Total}}={L_{MSE}}(I_{L,R}^{HR},I_{L,R}^{SR})+\lambda{L_{FC}}(I_{L,R}^{HR},I_{L,R}^{SR})$$

(24)

where $\lambda$ is a hyperparameter, it is set to 0.01 to control the proportion of the frequency Charbonnier loss function.

In this section, we provide a detailed overview of the datasets, evaluation metrics, and model configurations.

Datasets. Following previous works [38, 37, 35, 6], we used publicly available stereo image datasets for our training and testing. Specifically, we used 800 pairs of images from the Flickr1024 [34] dataset and 60 pairs of images from the Middlebury [28] dataset for training. We use four benchmark test sets: KITTI 2012 [11], KITTI 2015 [25], Middlebury [28], and Flickr 1024 [34] , to fully verify the effectiveness of our model.

Evaluation metrics. We evaluate our model using Peak Signal-To-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) on the RGB color space.

Model Setting. The number of MFF Block and feature channels is flexible and can be changed. In this paper, we adjust the settings in NTIRE 2024 competition to further reduce the number of parameters and improve efficiency. In the ×2 SR model, the number of blocks and the number of channels are set to 16 and 64, respectively. In the ×4 SR model, the number of blocks and the number of channels are set to 24 and 48, respectively.

Training Setting. We augment the training data using random horizontal flipping, rotation, and RGB channel shuffling. We use Lion [3] optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.99$. MFFSSR is training in PyTorch on a server with eight Nvidia A100 GPUs. The learning rate is initially set to $5\times{10^{-4}}$ and decay the learning rate with the cosine strategy. The total iterations for the model is set to 200,000.

In this section, we compare our proposed MFFSSR with existing image SR methods. These methods include VDSR [16], EDSR [19], RDN[42], RCAN[40], StereoSR [14], PASSRnet[31], IMSSRnet [17], iPASSSR [35], SSRDE-FNet [6], PFT-SSR [12], SwinFIR [39], NAFSSR [5] and CVHSSR [44] and so on. All models are trained on the same datasets, and these results are from [44].

Quantitative Evaluations. We summarize the SR results of MFFSSR and other SR methods at both ×2 and ×4 upsampling factors in Table 1. Our MFFSSR achieves the best performance with low parameters.

Efficiency Evaluations. We compare the parameters and computational complexity with the state-of-the-art methods from the NTIRE Stereo Image SR Challenge in 2022 and 2023. We achieve better performance than NAFSSR-S [5] while utilizing only 60% of the parameters and 75% of the FLOPs. For fair comparison, we reduce the parameter of SwinFIRSSR [39] and SCGLANet [43] to a level equivalent to that of MFFSSR, resulting in two variant networks: SwinFIRSSR-v and SCGLANet-v. As shown in Table 2, even in the condition that the parameters are slightly greater than SwinFIRSSR-v and SCGLANet-v, our computational cost remains lower. The evaluations are tested on 128×128 size as inputs.

Visual Comparison. Figure 5. and Figure 6. display the ×4 SR visualization results of different methods, our model is more visually realistic in perception and provides a clearer restoration of textures and features compared with existing methods. It notably achieves a better restoration effect on fences and buildings.

In this section, we conduct various ablation experiments to validate the effectiveness of the proposed structures and parameter settings. All ablation results are obtained using the Flickr1024 [34] test set.

Effectiveness of Hybrid Attention Feature Extraction Block. To further validate the effectiveness of the proposed structures, we investigate the roles of different elements in HAFEB, resulting in four network variants: Net-A (without LKA), Net-B (without RepConv), Net-C (without LKA and RepConv) and Net-D (replace IRF with the simple FFN in NAFSSR [5]). The comparison results are presented in Table 3. LKA plays the most important role in feature extraction process, which offers a wider receptive field. RepConv enhances the network’s generalization capability and perceptual capacity for details. The HAFEB leverages a residual structure to fully preserve the high and low-level features obtained from different elements, and integrates them with the cross-view features obtained by CVIM. We can observe that the performance of MFFSSR deteriorates when LKA and RepConv are removed. Removing both results in a decrease in PSNR of 0.07 dB. In addition, IRF more effectively regulate the information flow, achieving the 0.01 dB improvement in PSNR compared to the original FFN. These results demonstrate the crucial role these elements play in our network, indicating their indispensability.

Performance and Efficiency Trade-offs. As shown in Table 4, we compare the effects of (a) different branch weights $\theta$ in the multi-level extraction and fusion structure and (b) different cross-attention modules on network performance and efficiency. As the intra-view feature extraction branch weights $\theta$ gradually increase, the parameters and FLOPs decrease. However, $\theta$ too big leads to insufficient utilization of complementing information from cross-views, causing bad performance. When $\theta=0.750$, we achieve the best performance, but the increase in parameters and FLOPs is minimal compared to $\theta=0.875$. Additionally, we compare CVIM with NAFSSR’s cross-attention module, SCAM [5]. We obtain a 0.03 dB improvement in PSNR at the cost of 0.04M parameters and 0.651G FLOPs. We believe this trade-off is worthwhile.

We have submitted the results of our original model to the NTIRE 2024 Stereo Image Super-Resolution Challenge [33]. Our final scores are 23.53 dB PSNR and 21.50 dB PSNR in Track 1 Constrained SR & Bicubic Degradation and Track 2 Constrained SR & Realistic Degradation, respectively, ranking 7th and 9th.