Cross-Scope Spatial-Spectral Information Aggregation for Hyperspectral Image Super-Resolution

Single hyperspectral image super-resolution has attracted considerable attention owing to its diverse application prospects. Due to the excellent performance of CNNs for natural image SR [20, 21, 22, 37], a large number of CNN-based architectures have emerged for addressing hyperspectral image super-resolution. Yuan $et\ al.$ [38] integrated the transfer learning and collaborative non-negative matrix factorization to guide the reconstruction of hyperspectral image from the natural images. To utilize the high-dimensional spectral characteristics, Mei $et\ al.$ [11] proposed a 3D fully CNN network (3DFCNN) to directly extract the spatial-spectral features by the 3D convolutions. To reduce the high computational and memory complexity of 3D convolutions, Li $et\ al.$ [39] decomposed the 3D kernel into the combination of 1D-2D convolutional kernels. Then, Li $et\ al.$ [24] designed a mixed convolutional network (MCNet) to extract the spatial and spectral information by mixing the 2D and separable 3D convolution. Furthermore, Li $et\ al.$ [40] alternately employed 2D and 3D units to boost the spatial-spectral representation capacity, which fully explored the relationship between 2D/3D convolutions (ERCSR). Then, replacing 3D Convolution with a grouping mechanism, Li $et\ al.$ [26] further designed a grouped deep recursive residual network (GDRRN) by introducing the group-wise convolution into the recursive residual module. Similarly, Jiang $et\ al.$ [14] proposed a progressive multi-branch network to learn a spatial-spectral prior of the grouping spectra (SSPSR). To maintain the spatial–spectral consistency, Wang $et\ al.$ [27] proposed a recurrent feedback network (RFSR) with the regularization strategy to strengthen the interaction between the adjacent spectral grouping. Then, Wang $et\ al.$ [28] explored the similar and complementary information within adjacent spectral groups to reconstruct more details. Li $et\ al.$ [41] designed a dual-stage network from the coarse stage to the fine stage to exploit the spatial-spectral similarity between adjacent bands. Nevertheless, these convolution-based methods were restricted to excavating the local spatial-spectral features, which overlooked the long-range spatial-spectral dependencies.

Transformer has received widespread application in SR tasks [42, 43, 32] due to its exceptional performance in exploring non-local similarity. Liu $et\ al.$ [33] designed a parallel branch network called Interactformer, which incorporated both Transformer modules and 3D convolutions. Hu $et\ al.$ [12] proposed a multilevel progressive network (MPNet) to learn the fine details by the progressive learning and non-local channel attention. Wang $et\ al.$ [44] also combined the spectrum-wise self-attention and 3D convolutions named 3D-THSR to learn the spatial-spectral features in global receptive field. However, there were several limitations with the aforementioned approach. Firstly, in order to reduce the computational complexity, these methods only explored long-range dependencies in the spectral dimension, thereby failing to fully harness the global spatial information. Secondly, based on the previous analysis, these methods ignored the redundancy when calculating self-attention from the deep features with expanded spectral dimension. Lastly, the utilization of 3D convolutions for extracting local information required substantial memory consumption. Hence, to efficiently capture long-range dependencies in both spectral and spatial dimensions with low complexity, we introduce a cross-scope method to leverage local-global spatial-spectral information.

Motivated by the limitations of current CNNs and Transformer methods in capturing long-range spatial-spectral dependencies [31, 33], this paper aims to address the shortcomings of the existing CNN and Transformer in revealing latent spatial-spectral features. While 2D convolutions are confined to local areas and neglect spectral features, 3D convolutions lack the capability to model long-range dependencies and escalate network complexity. Despite the rising popularity of Transformers in SR, few studies have focused on dedicated Transformers for global spatial-spectral dependencies in hyperspectral image SR. Current strategies for natural images are not readily adaptable to hyperspectral data [14]. The proposed approach seeks to overcome these challenges by introducing a novel cross-scope strategy that leverages local-global spatial-spectral information efficiently.

The overall architecture of the proposed CST is shown in Fig. 2, consisting of three parts: shallow feature extraction, deep feature extraction, and image reconstruction. Given the input LR hyperspectral image $I_{\rm LR}\in\mathbb{R}^{H\times W\times B}$ and the SR scale factor $s$, where $H$ and $W$ are the height and width in the spatial dimension and $B$ is the number of spectral bands, our network outputs the HR hyperspectral image $I_{\rm SR}\in\mathbb{R}^{sH\times sW\times B}$. The CST first extracts the shallow features from $I_{\rm LR}$ through a $3\times 3$ convolution, which expands the spectral dimensions to obtain more feature maps. The shallow feature extraction can be denoted as

where $f_{\rm s}(\cdot)$ is the shallow feature extraction layer, and $F_{0}$ is the extracted shallow feature. Then, the shallow features pass through a sequence of Transformer stages to extract the deep features. As shown in Fig. 2 (b), the designed cross-scope Transformer stage contains successive Transformer layers and one $3\times 3$ convolution layer. And each Transformer stage includes two parallel branches, Transformer layer and spectral attention module, to adaptively aggregate the features from the the non-local or local regions. The deep feature extraction can be denoted as

where $f_{\rm n}(\cdot)$ is the function of $n$-th Transformer stage and $F_{n}$ denotes the $n$-th spatial-spectral feature from the Transformer stage. Subsequently, the deep feature is obtained by the skip connection and one convolution layer, which can be denoted as

where $F_{d}$ represents the extracted deep feature. Finally, the deep feature is fed into the reconstruction layer to improve the spatial resolution, which can be denoted as

where $F_{\rm up}(\cdot)$ represents an upsampling operation implemented using pixelshuffle and $F_{up}$ denotes the upsampled feature. The final reconstruction SR image can be attained by long skip connection and one convolution layer, which can be denoted as

where $F_{bi}(\cdot)$ is the bicubic upsampling operation. Through the final residual structure, the network can directly learn the significant textures and details for HR hyperspectral image.

As shown in Fig. 2 (c), the designed Transformer layer adopts the common the architecture that consists of some essential blocks: LayerNorm (LN) modules, distinctive cross-scope multi-head self-attention modules, and concise feed-forward neural network. Let $x_{l}\in\mathbb{R}^{H\times W\times C}$ denote the input feature of the $l$-th Transformer layer. The outputs of Transformer layer can be expressed as

where $CMSA(\cdot)$ represents the cross-scope multi-head self-attention, $LN(\cdot)$ represents the function of layer normalization, and $FFN(\cdot)$ represents the designed concise feed-forward neural network. The details of CMSA are shown in Fig.1 (c), which contains a cross-Scope spatial attention and cross-scope spectral attention.

Traditional convolutional approaches struggle to capture extended spatial dependencies, while standard Transformers are constrained by computational complexity. Drawing from these factors, we further introduce the cross-scope spatial self-attention to address the inherent drawback of window-based attention mechanisms that can only account for information within local regions. As shown in Fig. 3, the spatial cross-attention integrates the local and global information within the window-based attention mechanism, aiming to establish comprehensive global spatial dependencies while maintaining linear computational complexity.

Following [36], for the input feature $X\in\mathbb{R}^{H\times W\times C}$ after layer normalization, CSA is first divide it into two segments along the spectral dimension, where $X_{1}\in\mathbb{R}^{H\times W\times\frac{C}{2}}$ and $X_{2}\in\mathbb{R}^{H\times W\times\frac{C}{2}}$. Then, $X_{1}$ and $X_{2}$ are fed into horizontal and vertical cross-scope spatial multi-head self-attention mechanisms, respectively. Finally, the output $X^{{}^{\prime}}$ of CSA is calculated by aggregating the results from the horizontal and vertical dimensions, which can be denoted as

where $H\text{-}CSA(\cdot)$ and $W\text{-}CSA(\cdot)$ represents the function of mutli-head self-attention along horizontal and vertical dimensions, respectively.

Specifically, we calculate the multi-head attention in parallel, and here we take the single head as an example for the sake of simplicity. When the size of rectangle window is $[h,\ w]$ and $h>w$, we partition the feature $X_{1}$ as $[X_{1}^{1},X_{1}^{2},...,X_{1}^{N}]$ vertically, where $X_{1}^{i}\in\mathbb{R}^{H\times W\times\frac{C}{2}}$ and $N=\frac{H\times W}{h\times w}$. Then, the local features are calculated in the vertical windows. To establish long-range spatial dependencies within linear complexity, we directly apply two pooling operations to the input features to obtain global spatial information. Subsequently, the local and global features are reshaped and linearly mapped as query ($Q\in\mathbb{R}^{hw\times\frac{C}{2}}$), key ($K\in\mathbb{R}^{hw\times\frac{C}{2}}$), and value ($V\in\mathbb{R}^{hw\times\frac{C}{2}}$) to compute the cross-attention, which can be denoted as

where $W_{Q}^{i}K\in\mathbb{R}^{\frac{C}{2}\times\frac{C}{2}}$ and $W_{V}^{i}\in\mathbb{R}^{\frac{C}{2}\times\frac{C}{2}}$ represent the learnable parameters, $Averagepool(\cdot)$ and $Maxpool(\cdot)$ represent the function of adaptive average and max poolings, $B$ is the relative position encoding, and $d$ is the feature dimension. Finally, the output of the vertical CSA is obtained as

Similarly, for the horizontal cross-scope spatial multi-head self-attention, the size of rectangle window is $[w,h]$ and we can obtain the horizontal output $X_{2}^{{}^{\prime}}$. Additionally, between consecutive cross-scope spatial self-attention, we introduce the shift operation along different directions to enhance interactions between different windows. The previous works have demonstrate that the rectangle windows are advantageous for capturing repetitive textures in various directions.

The computational complexity of CSA can be denoted as

where $h$ and $w$ are the constants. In summary, our CSA can calculate the spatial cross-attention with linear computational complexity on the spatial size of $(H\times W)$.

Traditional convolution-based methods adopt the 3D convolution to emphasize local spectral features, disregarding the potential long-range connections. Although some Transformer-based approaches have combined spectral self-attention mechanisms and 3D convolutions, they often suffer from increased memory costs and high complexity due to the high-dimensional feature space. To tackle these issues, we design a cross-scope spectral self-attention mechanism that efficiently captures characteristic spatial-spectral correlations and global interactions. As shown in Fig. 4, the spectral cross-attention models the global spectral dependencies in linear computational complexity.

For the input feature $X^{{}^{\prime}}\in\mathbb{R}^{H\times W\times C}$ from the CSA, CSE first normalizes and reshapes it as $Y\in\mathbb{R}^{C\times H\times W}$. Then, we sequentially employ the point-wise convolution and depth-wise convolution to extract the characteristic features containing comprehensive spatial-spectral information. We also achieve the multi-head attention mechanism, and here we take the single head as an example for the simplicity. To avoid excessive redundancy across high-dimensional channels, the characteristic features, (i.e. query ($Q\in\mathbb{R}^{H_{r}\times W_{r}\times C_{r}}$) and key ($K\in\mathbb{R}^{H_{r}\times W_{r}\times C_{r}}$)), are adaptively calculated through the reduction in both spatial and spectral dimensions. In this paper, we define $H_{r}=\frac{H}{2},W_{r}=\frac{W}{2}$, and $C_{r}=\frac{C}{2}$. We exploit the cross-attention between the characteristic features and the global feature to achieve the global spatial-spectral interactions, which can be denoted as

where $W_{p}^{(\cdot)}(\cdot)$ is the function of point-wise convolution, $W_{d}^{(\cdot)}(\cdot)$ is the function of depth-wise convolution, $W_{v}(\cdot)$ is the standard $1\times 1$ convolution, $d$ is the feature dimension, and $Y^{{}^{\prime}}\in\mathbb{R}^{C\times H\times W}$ is the output of CSE. Through CSE, we facilitate the interactions between essential representative features and global features, capturing long-range spectral similarities with low computational cost.

The standard computational complexity of self-attention on spectral dimension is $O(C^{2}HW)$. The computational complexity of the designed CSE can be denoted as

where $C_{r}<C$, $H_{r}<H$, and $W_{r}<W$. In summary, our CSA can calculate the spectral cross-attention with low computational cost compared with the original spectral self-attention.

Traditional feed-forward neural network treat the pixels at each position independently and equally without considering the relationships between them. Following the gating mechanism [32], we design a concise feed-forward neural network (CFN) to improve the representative captivity of the model. As shown in Fig. 5, CFN comprises two parallel branches to process the input spatial-spectral features. In detail, we sequentially utilize point-wise convolution and depth-wise convolution in two branches to extract details beneficial for image reconstruction. Additionally, a non-linear transformation using the GELU activation function is applied in one of the branches, enabling the network to flexibly focus on crucial information. Finally, the advantageous context is learned through the element-wise product of the linear and non-linear branches. Given an input feature $Y^{{}^{\prime}}\in\mathbb{R}^{C\times H\times W}$, the process of CFN can be denoted as

Compared with the gating structure in [32], our CFN focuses on exploring internal spatial correlations to uncover more spatial information conducive to image restoration.

Following the previous works, three losses, i.e. $l_{1}$ loss, spectral angle mapper (SAM) loss, and gradient loss in both spatial and spectral domains, are employed to optimize the proposed network.

Firstly, the $l_{1}$ loss is widely used in SR for natural image [22]. It calculates the absolute pixel-wise difference between the reconstructed SR image and the original HR image. While the $l_{2}$ loss tends to generate overly smoothed results, the $l_{1}$ loss provides a more equitable distribution of errors and enhances the convergence of training. This property encourages the model to reconstruct sharper and more detailed results. In our approach, we adopt the $l_{1}$ loss to measure the accuracy of the reconstructed hyperspectral images and preserve the spatial details during the super-resolution process. Secondly, The SAM loss is introduced to ensure the spectral consistency of the reconstruction images. Unlike traditional loss functions that focus solely on pixel-wise differences, the SAM loss takes into account the spectral characteristics of hyperspectral data. Finally, inspired by [27, 28], we integrate gradient information to further enhance the sharpness of the reconstructed images. The gradient loss focuses on the differences between adjacent pixels. In conclusion, the total loss for the proposed network can be formulated as

where $N$ is the number of the input in a training batch, $H_{hr}^{n}$ and $H_{sr}^{n}$ represents the $n$-th HR and SR hyperspectral images, and $M(\cdot)$ represents the horizontal, vertical, and spectral gradients of the input, respectively. $\lambda_{s}$ and $\lambda_{g}$ denote the hyper-parameters that control the balance between the different losses for excellent SR performance. The analysis of the loss functions is shown in the section of the experiment. In this paper, $\lambda_{s}$ is set to 0.3 and $\lambda_{g}$ is set to 0.1, empirically.

The experiments are conducted on three common real-scenario hyperspectral image datasets, i.e., Chikusei [45], Pavia Center [46], and Houston ¹¹1https://hyperspectral.ee.uh.edu/?page_id=1075.

In the proposed network CST, we set the kernel size of the standard convolution and depth-wise convolution as $3\times 3$, and the size of others for point-wise convolution and spectral attention is $1\times 1$. Zero padding is adopted to maintain the spatial size of feature maps. Following the settings in [22], we set the reduction ratio in CA as 16. We set the number of channels $C$ to $180$, the number of Transformer stages as $4$, and the number of Transformer layers as $4$. In general, the rectangle window size is set to $2\times 16$. In the image reconstruction, we adopt the progressive upsampling strategy by PixelShuffle [47] to upsample the input LR hyperspectral images for reducing the parameters (e.g., upsampling 3 times for scale factor $\times$8). For the loss function, we set $\lambda_{1}=0.3$ and $\lambda_{2}=0.1$. In the training procedure, the Adam optimizer with the default setting is employed to train the network for 500 epochs, and the size of the mini-batch is set to 32. The initial learning rate is set to $1e^{-4}$ and halves every 100 epochs until 300 epochs. The proposed model is implemented by Pytorch on NVIDIA RTX 3090 GPU.

We compare the proposed CST with other state-of-the-art methods at three scale factors, including one interpolation method, one Transformer-based SR SwinIR [31] for natural image and deep learning-based SHSR methods, i.e. Bicubic, GDRRN [26], SSPSR [14], RFSR [27], and GELIN [28]. For SwinIR, We directly feed the whole hyperspectral image into the network for training and modify the dimensions of the input and output. For the aforementioned methods, we strive to achieve their optimal performance. Six popular evaluation metrics are used for the comparison experiments to entirely demonstrate the performance of the models in both spatial and spectral dimensions, including peak signal-to-noise ratio (PSNR), structure similarity (SSIM), spectral angle mapper (SAM), cross correlation (CC), root-mean-squared error (RMSE), and erreur relative global adimensionnellede synthese (ERGAS). While PSNR, SSIM, and RMSE are typically employed to assess the quality of natural image restoration, the remaining three metrics, CC, SAM, and ERGAS, are common evaluation measures in hyperspectral image fusion tasks.

After removing the blurred edges, the size of the remaining central region in the Chikusei dataset is $2304\times 2048\times 128$. Following the previous work [14, 28], the four non-overlapping images with the size of $512\times 512\times 128$ are cropped from the top region for testing. The remaining area is cropped into overlapping HR images for training (10% of the training data is randomly selected for validation). Specifically, the spatial resolution of LR training patches is $32\times 32$, and the sizes of corresponding HR patches for scale factors $\times 2$, $\times 4$, and $\times 8$ are $64\times 64$, $128\times 128$, and $256\times 256$, respectively. All the LR patches from the hyperspectral image are generated through bicubic downsampling at different scales.

The quantitative results of our method and other compared methods at different scale factors on the Chikusei dataset are shown in Table I. The best results are indicated in bold, and the second-best results are underlined. We can observe that the interpolation-based method Bicubic exhibits ordinary SR performance, while learning-based methods achieve significant improvements. While the Transformer-based method SWINIR can slightly enhance spatial recovery results, this approach designed for natural image super-resolution does not consider the spectral characteristics of hyperspectral images, leading to spectral distortion. Grouping-based methods designed for hyperspectral images restore the HR images in both spatial and spectral dimensions but do not explore the non-local spatial-spectral similarities. Notably, through capturing the global spatial-spectral dependencies, our proposed CST achieves the best performance for all metrics at scale factors $\times 2$, $\times 4$, and $\times 8$, which demonstrates the effectiveness and superiority of our method.

As shown in Fig. 6, the qualitative results of all methods at scale factor $\times 4$ on the Chikusei dataset are presented. Especially, the 70th, 100th, and 36th bands of the test image are selected as the R-G-B channels for visual comparison. We can observe that while interpolation-based method Bicubic upsampling can achieve hyperspectral image SR, it often introduces blur and overly smooth details. Recent methods like SSPSR and GELIN have made progress in restoring overall structure, but they still suffer from unclear boundaries and severe artifacts. In particular, the visual results demonstrate that the proposed CST can reconstruct the HR hyperspectral images with clearer and sharper details compared with the other methods. Moreover, the advantages of our method can be better observed based on the enlarged view of the details in the red box. Additionally, we also visualize the mean error images among all spectra in Fig. 7 to provide an intuitive view of the accuracy of the reconstruction for individual pixels. In the error images, a bluer color indicates a higher accuracy for the reconstruction results. According to the highlighted red box region in error maps, our method exhibits minimal errors, which illustrates the superiority of the proposed method. Finally, the mean spectral difference curves of the test images at the scale factors $\times 4$ and $\times 8$ from the testing dataset are shown in Fig. 8 to evaluate the SR results from a spectral perspective. Compared to analyzing the reflectance of random pixels in the reconstructed image, the mean spectral difference offers a comprehensive measure of the accuracy of spectral reconstruction quality across the entire image. The lower the curve, the fewer spectral differences and distortion between the SR results and the ground-truth. Clearly, our method achieves the best spectral reconstruction results at different scale factors.

Following the above settings, the eight non-overlapping images from the Houston dataset with the size of $256\times 256\times 48$ are cropped from the top region for testing. The remaining area is cropped into overlapping HR patches for training (10% of the training data is randomly selected for validation). Specifically, the spatial resolution of LR training patches is $32\times 32$, and the sizes of corresponding HR patches for scale factors $\times 2$, $\times 4$, and $\times 8$ are $64\times 64$, $128\times 128$, and $256\times 256$, respectively.

In Table II, the quantitative results of all methods at different scale factors on the Houston dataset are shown. It can be observed that our CST method outperforms other methods for all metrics at different scale factors. Moreover, compared to the results on other datasets, our method shows greater improvements on the Houston dataset. This is attributed to the Transformer-based structure of our method, which requires a substantial amount of training data to enhance the capabilities of the model, and the Houston dataset offers more training samples. The results further substantiate the effectiveness of our method, and emphasize the significance of long-range spatial and spectral similarities in hyperspectral image SR.

Similarly, as shown in Fig. 9, we also present the qualitative results of all methods at scale factor $\times 4$ on the Houston dataset. Specifically, the 29th, 26th, and 19th bands of the test image are treated as the R–G–B channels for visual comparison. It can be observed that other methods inevitably produce many blurry edges and artificial artifacts, while our method is capable of recovering more low-frequency and high-frequency information in the SR hyperspectral images. Moreover, the enlarged view of the details in the red box presents the advantages of our method.

After removing the noise region without information, the size of the available part in the Pavia dataset is $1096\times 715\times 102$. For scale factors $\times 2$ and $\times 4$, the four non-overlapping images with the size of $256\times 256\times 102$ are cropped from the left region for testing. Additionally, for the scale factor $\times 8$, the non-overlapping images with the size of $128\times 128\times 102$ are cropped from the left region for testing. The remaining area is cropped into overlapping HR images for training (10% of the training data is randomly selected for validation). Specifically, the spatial resolution of LR training patches is $16\times 16$, and the sizes of corresponding HR patches for scale factors $\times 2$, $\times 4$, and $\times 8$ are $32\times 32$, $64\times 64$, and $128\times 128$, respectively.

In Table III, the quantitative results of all methods at different scale factors on the Houston dataset are shown. Compared to the other datasets, the Pavia dataset has a lower spatial resolution. Consequently, it is a challenge for our Transformer-based method CST due to the limited training samples. In addition, all methods perform poorly relatively on this dataset. Moreover, Nonetheless, our method still manages to outperform other methods across all metrics on this dataset. In summary, our method achieves competitive performance on the dataset with limited samples, which demonstrates the effectiveness of non-local spatial and spectral similarities.

As shown in Fig. 10, we visualize the qualitative results of all methods from the testing image at the scale factor ×4. Especially, the 100th, 30th, and 12th bands of the testing image are selected as the R–G–B channels for visual comparison. The observations for the previous dataset are also applicable to the Pavia dataset. Similarly, other methods generate unclear contours and textures, while our approach can restore more fine details. Based on the region in the red box, we can observe the superiority of our approach.

In this section, we provide the performance of different variants at the scale factor $\times 4$ on Chikusei dataset to verify the effectiveness of our method. We calculate the model parameters and the floating point operations (Flops) of all methods when the size of input is $32\times 32\times 128$.