Cross-View Hierarchy Network for Stereo Image Super-Resolution

Image super-resolution is a regression problem that maps a low-resolution image to its corresponding high-resolution image. Since Dong et al. [8] introduce CNN into the SISR field with their pioneering work SRCNN, CNN-based methods have been proven to achieve impressive performance in SISR tasks. On this basis, Lim et al. [13] further improved network performance by deepening the network and increasing the dimension of intermediate features. With the development of deep learning technology, the researchers use residual and dense connections [40, 18, 14, 36] to control network information flow, thereby obtaining better image reconstruction performance. However, these methods did not consider the importance of different features, leading to redundant network design. Therefore, RCAN [39] introduced channel attention mechanisms to model the interdependence between feature channels and adaptively rescale the features of each channel. Subsequently, various attention mechanisms were proposed to enhance network expression ability, including spatial attention [7, 23], second-order attention[6], non-local attention[21, 20], and large kernel convolution attention [30].

Recently, the Transformer has achieved great success in the field of vision. Therefore, many researchers [16, 17, 38, 19, 33] introduce the Transformer into SISR tasks. With the powerful learning ability of Transformers, Transformer-based methods have achieved state-of-the-art performance in the field of single-image super-resolution. Despite the consistently improved performance, SISR cannot utilize complementary information from different views in stereo image pairs, which limits the performance of stereo image super-resolution.

Instead of SISR method, which only has access to context information from intro, stereo image SR can leverage the additional information provided by the cross-view information to enhance SR performance. However, the presence of binocular disparity between the left and right views in a stereo image pair can pose a significant challenge to the fusion of information across views. Thus, Jeon et al. [12] proposed the first deep learning-based model for stereo image SR (namely, StereoSR). This approach addresses the challenge of fusing complementary features of the left and right views by concatenating the left image and a stack of right images with predefined shifts. On this basis, Wang et al. [28, 29] introduced a parallax attention module (PAM) to model stereo correspondence by effectively capturing global contextual information along the epipolar line. These methods outperforms StereoSR and exhibits greater flexibility in accommodating disparity variation. In pursuit of more refined stereo correspondence, Song et al. [25] extended the parallax-attention mechanism to propose SPAM, which aggregates information from both the primary and cross views to generate stereo-consistent image pairs. Yan et al. [35] proposed a domain adaptive stereo SR network (DASSR) to achieve both stereo image SR and stereo image deblurring tasks. Xu et al. [34] introduced the concept of bilateral grid processing within a CNN framework, thereby proposing a bilateral stereo SR network. Then, Wang et al. [32] enhanced PASSRNet (ie. iPASSRNet) by leveraging the symmetry cues present in stereo image pairs. Ma et al. proposed a perception-oriented StereoSR framework, which aims to restore stereo images with improved subjective quality. More recently, Chu et al. [4] developed the NAFSSR network by utilizing nonlinear activation-free blocks [1] for intra-view feature extraction and PAM for cross-view feature interaction, which achieves the champion in the NTIRE 2022 Stereo Image SR Challenge [27].

Although existing methods have achieved superior performance in stereo image SR, they typically focus on modeling cross-view information while neglecting the hierarchical similarity relationships from intra-view. To address this limitation, we propose to leverage both local and global hierarchical feature representations to further improve the performance of state-of-the-art stereo image SR methods.

To avoid complex network designs that require a large number of parameters and computational effort, we adopt a simple weight-sharing two-branch network structure to recover the left and right view images, as illustrated in Figure 2. Our CVHSSR mainly consists of four components: the shallow feature extraction, the cross-hierarchy information mining block (CHIMB), the cross-view interaction modules (CVIM), and stereo image reconstruction. Specifically, the CHIMB is designed to extract similar features both locally and globally from the intra-view image, effectively restoring accurate texture details. The CVIM is mainly used to fuse features from two viewpoints. More details of CHIMB and CVIM are described in Sections 3.2 and 3.3.

Firstly, given an input stereo low-resolution images $I^{L}_{LR},I^{R}_{LR}\in\mathcal{R}^{H\times W\times 3}$, CVHSSR first applies a convolution to obtain two-views shallow feature $F^{L}_{0},F^{R}_{0}\in\mathcal{R}^{H\times W\times C}$, where $H\times W$ denotes the spatial dimension and $C$ is the number of channels. It can be formulated as:

$$F^{L,R}_{0}=H_{conv}(I^{L}_{LR},I^{R}_{LR}),$$

(1)

where $H_{conv}$ denotes $3\times 3$ convolution operation.

Next, we integrate CHIMB and CVIM into a cross-view hierarchy (CVH) block, which not only extracts deep intra-view features but also fuses information from different view images. Therefore, we stack $N$ CVH blocks to obtain output features that incorporate information from multiple perspectives. It can be expressed as:

	$\displaystyle F^{L,R}_{out}=H_{CVH}^{N}(H_{CVH}^{n-1}(\cdots H_{CVH}^{1}(F^{L,R}_{0}))\cdots),$		(2)
	$\displaystyle F^{L,R}_{i+1}=H_{CVH}(F^{L,R}_{i})=H^{i}_{CV}(H^{i}_{CH}(F^{L,R}_{i})),$

where $H_{CVH},H_{CV}$, and $H_{CH}$ denote CVH block, CVIM, and CHIMB, respectively. $F^{L,R}_{out},F^{L,R}_{i}$ denote the output of the $N$-th CVH Block and $i$-th CVHBlock, respectively.

Finally, we upsample the output features to the HR size using the pixel-shuffle operation. Additionally, we incorporate a global residual path to leverage the input stereo image information to further improve the super-resolution performance. It can be expressed as:

	$\displaystyle I^{L}_{SR}=H_{up}(F^{L}_{out})+H_{up}(I^{L}_{LR})=H^{L}_{CVHSSR}(I^{L}_{LR}),$		(3)
	$\displaystyle I^{R}_{SR}=H_{up}(F^{R}_{out})+H_{up}(I^{R}_{LR})=H^{R}_{CVHSSR}(I^{R}_{LR}),$

where $H_{up}$ denotes the upsampling operation. $H_{CVHSSR}$ denotes the proposed CVHSSR network. $I^{L}_{SR},I^{R}_{SR}$ denote the final restored left-view and right-view images, respectively.

Many existing methods for stereo image SR focus mainly on modeling cross-view information and do not adequately exploit the hierarchy information from the intra-view image. This leads to difficulties in recovering clear texture details. To address this issue, we have proposed the CHIMB as depicted in Figure 3, which is capable of effectively extracting different hierarchy information features from images.

The CHIMB consists of two parts: (1) The cross-hierarchy information extractor (CHIE); (2) The information refinement feedforward network (IRFFN). The CHIMB incorporates both channel attention and large kernel convolution attention to capture both global and local similarity relationships. The channel attention calculates global statistics of the feature map to enhance the focus on important features. Meanwhile, the large kernel convolution attention utilizes larger kernel convolution to capture long-range dependencies in the intra-view image, thereby enhancing the attention to local information. These two attention mechanisms in combination enable the CHIE to effectively model the hierarchy information contained in the input images and accurately recover texture details.

Given an input tensor $F_{in}\in\mathcal{R}^{H\times W\times C}$, the CHIE is formulated is:

$$F_{\text{CHIE}}=W_{p}^{0}(\mathcal{H}(\delta_{SG}(W^{1}_{d3}W^{1}_{p}(\text{LN}(F_{in})))))+F_{in},$$

(4)

where $W_{p}^{(\cdot)}$ is the $1\times 1$ point-wise convolution and $W_{d3}^{(\cdot)}$ is the $3\times 3$ depth-wise convolution. $F_{\text{CHIE}}$ denotes the output feature of CHIE. The $\text{LN}(\cdot)$ denotes layer normalization. We use the notation $\delta_{SG}(\cdot)$ and $\mathcal{H}(\cdot)$ to represent the SimpleGate function and the hybrid attention operation, respectively. Specifically, the SimpleGate first split the input into two features $\textbf{X}_{1},\textbf{X}_{2}\in\mathcal{R}^{H\times W\times C/2}$ along channel dimension. Then, it computes the output with the linear gate as $\delta_{SG}(\textbf{X})=\textbf{X}_{1}\odot\textbf{X}_{2}$, where $\odot$ denotes element-wise multiplication. The hybrid attention $\mathcal{H}(\cdot)$ consists of two components: channel attention and large kernel convolution attention. It can be described as follows:

	$\displaystyle\mathcal{H}(\textbf{X})$	$\displaystyle=\text{LKA}(\textbf{X})+\text{CA}(\textbf{X}),$		(5)
	$\displaystyle\text{CA}(\textbf{X})$	$\displaystyle=\textbf{X}\odot(W_{p}H_{Avg}(\textbf{X})),$
	$\displaystyle\text{LKA}(\textbf{X})$	$\displaystyle=\textbf{X}\odot(W_{p}W_{dd7}W_{d5}(\textbf{X})),$

where $\text{LKA}(\cdot)$, $\text{CA}(\cdot)$ and $H_{Avg}$ denote the large kernel convolution attention, the channel attention, and the average pooling operation, respectively. $\odot$ denotes element-wise multiplication. The $W_{d5}$ and $W_{dd7}$ represent the $5\times 5$ depth-wise convolution and the $7\times 7$ depth-wise dilation convolution.

The IRFFN in our pipeline utilizes a non-linear gate mechanism to control the flow of information, allowing each channel to focus on fine details complementary to the other levels. The IRFFN process is formulated as:

$$F_{out}=W^{3}_{p}(\delta_{NG}(W^{2}_{d3}W^{2}_{p}(\text{LN}(F_{\text{CVIE}}))))+F_{\text{CVIE}}$$

(6)

where $\delta_{NG}(\cdot)$ is the function of non-linear gate mechanism. Similar to SimpleGate, the non-linear gate mechanism divides the input along the channel dimension into two features $\textbf{X}_{1},\textbf{X}_{2}\in\mathcal{R}^{H\times W\times C/2}$. The output is then calculated by non-linear gating as $\delta_{NG}(\textbf{X})=GELU(\textbf{X}_{1})\odot\textbf{X}_{2}$, where $GELU(\cdot)$ denotes the activation function. The $F_{out}$ denote the output of CHIMB.

The details of the proposed CVIM as shown in Figure 4. This method employs Scaled DotProduct Attention [26], which involves calculating the dot product between the query and keys, followed by the application of a softmax function to generate weights assigned to the corresponding values:

$$Attention(\textbf{Q},\textbf{K},\textbf{V})=\text{Softmax}(\textbf{Q}\textbf{K}^{T}/{\sqrt{C}})\textbf{V},$$

(7)

where $\textbf{Q}\in\mathcal{R}^{H\times W\times C}$ is query matrix projected by source intra-view feature (e.g. left-view), and $\textbf{K},\textbf{V}\in\mathcal{R}^{H\times W\times C}$ are key, value matrices projected by target intra-view feature (e.g. right-view). Here, $H,W,C$ represents the height, width, and number of channels of the feature map.

The CVIM adopts a novel cross-view attention mechanism that integrates information from both left and right view images to generate cross-view attention maps. This approach allows for the exploitation of the distinctive information present in each view, leading to more effective feature fusion and better restoration results. In detail, given the input stereo intra-view features $F_{L}^{i},F_{R}^{i}\in\mathcal{R}^{H\times W\times C}$, we can get the cross-view fusion features $F_{L\rightarrow R}$ as follows:

	$\displaystyle\textbf{Q}_{L}$	$\displaystyle=W_{d}^{Q_{L}}W_{p}^{Q_{L}}(LN(F_{L}^{i})),$		(8)
	$\displaystyle\textbf{K}_{R}$	$\displaystyle=W_{d}^{K_{R}}W_{p}^{K_{R}}(LN(F_{R}^{i})),$		(9)
	$\displaystyle\textbf{V}_{R}$	$\displaystyle=W_{d}^{V_{R}}W_{p}^{V_{R}}F_{R}^{i},$		(10)
	$\displaystyle F_{L\rightarrow R}$	$\displaystyle=W_{p}^{R}Attention_{L\rightarrow R}(\textbf{Q}_{L},\textbf{K}_{R},\textbf{V}_{R}),$		(11)

where $W_{p}^{(\cdot)}$ is the $1\times 1$ point-wise convolution. $W_{d}^{(\cdot)}$ is the $3\times 3$ depth-wise convolution. It offers feature refinement from both channel and spatial perspectives.

With an analogous argument, we can obtain the cross-view fusion features $F_{R\rightarrow L}$ as follows:

	$\displaystyle\textbf{Q}_{R}$	$\displaystyle=W_{d}^{Q_{R}}W_{p}^{Q_{R}}(LN(F_{R}^{i})),$		(12)
	$\displaystyle\textbf{K}_{L}$	$\displaystyle=W_{d}^{K_{L}}W_{p}^{K_{L}}(LN(F_{L}^{i})),$		(13)
	$\displaystyle\textbf{V}_{L}$	$\displaystyle=W_{d}^{V_{L}}W_{p}^{V_{L}}F_{L}^{i},$		(14)
	$\displaystyle F_{R\rightarrow L}$	$\displaystyle=W_{p}^{L}Attention_{R\rightarrow L}(\textbf{Q}_{R},\textbf{K}_{L},\textbf{V}_{L}),$		(15)

Finally, the interacted cross-view information $F_{L\rightarrow R}$, $F_{R\rightarrow L}$ and intra-view information $F_{L}^{i}$, $F_{R}^{i}$ are fused by element-wise addition:

	$\displaystyle F_{L}^{i+1}=\gamma_{L}F_{L\rightarrow R}+F_{L}^{i},$		(16)
	$\displaystyle F_{R}^{i+1}=\gamma_{R}F_{R\rightarrow L}+F_{R}^{i},$		(17)

where $\gamma_{L}$ and $\gamma_{R}$ are trainable channel-wise scales and initialized with zeros for stabilizing training.

As image super-resolution tasks are primarily focused on restoring high-frequency details, we leverage both spatial and frequency domain losses to jointly guide our network in effectively recovering clear and sharp high-frequency textures. Specifically, given an input two view LR image $I_{LR}^{L,R}$, the proposed model predicts HR stereo image denote $I_{SR}^{L,R}$. We optimize our CVHSSR with the following loss function:

$$\mathcal{L}_{total}=\mathcal{L}_{MSE}(I_{SR}^{L,R},I_{HR}^{L,R})+\lambda*\mathcal{L}_{FC}(I_{SR}^{L,R},I_{HR}^{L,R}),$$

(18)

where $I_{HR}^{L,R}$ represents the left-view and right-view HR image, and $\mathcal{L}_{MSE}$ is the MSE loss:

$$\mathcal{L}_{MSE}=\dfrac{1}{N}\sum_{i=1}^{N}||I_{HR}^{L,R}-I_{SR}^{L,R}||^{2},$$

(19)

In addition, $\mathcal{L}_{FC}$ is the frequency Charbonnier loss, defined as:

$$\mathcal{L}_{FC}=\dfrac{1}{N}\sum_{i=1}^{N}\sqrt{||FFT(I_{HR}^{L,R})-FFT(I_{SR}^{L,R})||^{2}+\epsilon^{2}},$$

(20)

with constant $\epsilon$ emiprically set to $10^{-3}$ for all the experiments. $FFT(\cdot)$ denotes a fast Fourier transform. The parameter $\lambda$ in Eq. (18) is a hyper-parameter used to control the composition of the frequency Charbonnier loss function. The parameter $\lambda$ is set to $0.01$ for all the experiments. More training details of our method are presented in Section 4.

In this section, we provide a detailed description of the experimental setting, including the datasets, the evaluation metrics, and the training configurations.

Dataset. Following the previous methods [32, 4, 38], we employ the training and validation datasets provided by the Flickr1024 [31]. To be specific, we employ 800 stereo images as the training data, and 112 stereo images as the validation data. We augment the training data with random horizontal, flips, rotations, and RGB channel shuffle. For testing, we use four benchmark datasets: KITTI 2012 [9], KITTI 2015 [22], Middlebury [24], and Flickr1024 [31].

Evaluation metrics. We adopted peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as quantitative metrics for evaluation, which are calculated in the RGB color space between a pair of stereo images (i.e.,$(Left+Right)/2$).

Model Setting. The numbers of the CHIMB blocks and feature channels are flexible and configurable. We construct two CVHSSR networks of varying sizes, which we named CVHSSR-T (Tiny) and CVHSSR-S (Small) by adjusting the number of channels and blocks. Specifically, the number of channels and blocks for CVHSSR-T are set to 48 and 16 respectively. The number of channels and blocks for CVHSSR-S are set to 64 and 32 respectively.

Training Settings. Our network was optimized using the Lion method [2] with $\beta_{1}$=0.9, $\beta_{2}$=0.999, and a batch size of 8. Our CVHSSR was implemented in PyTorch on a PC with four Nvidia RTX 3090 GPUs. The learning rate was initially set to $5\times 10^{-4}$ and decayed the learning rate with the cosine strategy. We trained this model for 200,000 iterations. To alleviate the overfitting issue, we use stochastic depth [11] with 0.1 and 0.2 probability for CVHSSR-S and CVHSSR-B, respectively. Moreover, we also use Test-time Local Converter (TLC) [3] to further improve the model performance. The TLC method mainly aims to reduce the discrepancy between the distribution of global information during training and inference by converting global operations into local operations at inference.

In this section, we conduct a comparative analysis between our CVHSSR (with 2 different variations) and existing super-resolution (SR) methods. The comparison involved SISR methods such as VDSR [13], EDSR [18], RDN [40], and RCAN [39], as well as stereo image SR methods like StereoSR [12], PASSRnet [29], SRRes+SAM [37], IMSSRnet [15], iPASSR [32], SSRDE-FNet [5], NAFSSR [4], SwinFIR [38], and PFT-SSR [10]. All these methods were trained on the same datasets as ours, and their PSNR and SSIM scores were evaluated and reported by [4].

Quantitative Evaluations. We present a comparative evaluation of our proposed CVHSSR against existing stereo SR methods at $\times 2$ and $\times 4$ upscaling factors, as summarized in Table 1. Notably, even our smallest CVHSSR-T model outperforms the NAFSSR-S method on all datasets while utilizing 60% fewer parameters. Moreover, our CVHSSR-S model achieves better results than the NAFSSR-B method while requiring 70% fewer parameters. Specifically, our CVHSSR-S method is 0.19 dB and 0.48 dB higher than NAFSSR-S and SRRDE-FNet, respectively, on the $\times 4$ Flickr1024 dataset for the same amount of parameters. These results demonstrate the effectiveness of our proposed method and its superiority over existing methods in stereo image SR tasks.

Visual Comparison. In Figures 5 and 6, we present comparative results of $\times 4$ stereo SR obtained by different stereo SR methods on the Flickr1024 [31], KITTI2012 [9], and KITTI2015 [22] datasets. The visual comparison of reconstructed images demonstrates that our proposed CVHSSR-S method achieves sharper and more accurate texture details compared to the NAFSSR-B method which still suffers from over-smoothing fine textures. This validates the superiority and effectiveness of our proposed CVHSSR method.

In this section, we conduct a set of ablation experiments to evaluate the performance of each proposed module. The evaluation is performed on the Flickr1024 [31] validation dataset.

Effectiveness of Each Operation. To further substantiate the effectiveness of our proposed module, a series of ablation experiments were conducted and the results are presented in Table 2. Initially, the NAFSSR was used as the baseline, and subsequently, the corresponding module was modified continuously to verify the efficacy of the proposed module. As depicted in the table, LKA provided a performance improvement of 0.08 dB to the baseline due to its larger receptive field. However, merely enlarging the receptive field of the network does not fully exploit the hierarchical relationship of the intra-view. Hence, our proposed CHIE provided a superior performance improvement of 0.1 dB to the network. Compared to the simple FFN in NAFSSR [4], our proposed IRFFN more effectively regulated the information flow. Additionally, our proposed CVIM demonstrated a superior ability to fuse similar information from various perspectives and enhance network performance, compared to the traditional PAM [32]. These comparisons unequivocally underscore the effectiveness of our proposed methods.

Effectiveness of $\lambda$ in loss function. To evaluate the influence of different values of $\lambda$ in the loss function, a series of experiments are conducted in this section. The hyperparameter $\lambda$ is utilized to balance the trade-off between the MSE loss function and the frequency Charbonnier loss function. We conducted a range of empirical experiments with six different $\lambda$ values within the range of [0,1], based on previous experience. The impact of different $\lambda$ values on the model performance is illustrated in Figure 7. The results demonstrate that the network achieves optimal performance at $\lambda=0.01$.

We have submitted the results obtained from our proposed approach to the NTIRE 2023 Stereo Image Super-Resolution Challenge. To enhance the potential performance of our method, we have increased the depth and width of the CVHSSR-Based model. During test-time, we employed self-ensemble and model ensemble strategies. The final submission achieves 24.114 dB PSNR on the validation set and achieves 23.742 dB PSNR on the test set. We won 8th, 5th, and 4th on track 1 Fidelity & Bicubic, track 2 Perceptual & Bicubic, and tack 3 Fidelity & Realistic, respectively.