Dual-Stage Approach Toward Hyperspectral Image Super-Resolution

Unlike natural images, hyperspectral images have multiple bands, which can effectively identify objects. As for hyperspectral image SR, whether CNN is built with 2D or 3D convolution, the existing methods [7, 25, 14, 4, 26] are mainly to input all bands into the model for study. The simplified structures are shown in Figs. 2(a)-2(c). It obviously requires more GPU memory than the RGB image SR task. Under the condition of limited hardware resources, GPU memory is still a major constraint to build a deep network.

To alleviate this trouble, a natural improvement is to change input mode. Concretely, all the bands of hyperspectral image are input into the model in band-by-band. Typical work is SFCSR [16]. The structure of its network is displayed in Fig. 2(d). Inspired by this new structure, in our paper, we adopt this manner to design the backbone in coarse stage.

In this section, we introduce several representative works using different types of convolution.

(a) SSPSR: For hyperspectral image SR and natural image SR, there is no difference except for the type of input image. Therefore, Jiang et al. [25] refer to single RGB image SR approaches, and introduce spatial-spectral prior network using 2D convolution to design model, whose simplified flowchart is shown in Fig. 2(a). Since only 2D convolution cannot discover spectral knowledge, this type of method is hardly employed.

(b) 3D-FCNN: The regular 3D convolution is first utilized to explore both spatial context and spectral correlation in neighboring bands [14]. The structure is shown in Fig. 2(b). As we mentioned previously, the regular 3D convolution produces large parameters, thus making the network not easy to learn. To relieve this drawback, further improvements include the use of separable 3D convolution [19, 5, 13, 6] and reducing the number of 3D convolution [4, 21].

(c) ERCSR: Li et al. state that MCNet [21] designs the main modules in parallel, which does not effectively combine 2D/3D convolution, resulting in structural redundancy. For this reason, a novel alternate structure with 2D/3D convolutions is developed [4] (see Fig. 2(c)). It not only solves module redundancy. Besides, it also improves the representation learning in spatial domain. Currently, ERCSR is by far the best performance among the existing techniques.

(d) SFCSR: The approach [16] first applies novel input mode to analyze the information from both single band and its two adjacent bands (see Fig. 2(d)). Although it availably saves memory footprint, it does not take into account more neighboring bands. Importantly, such single band SR inevitably leads to serious spectral distortion during recurrent strategy. Furthermore, the recurrent mode ignores the overall learning and optimization of hyperspectral image. Considering this issue, after hyperspectral image is reconstructed in band-by-band, we add an enhanced back-projection strategy as post-processing to further optimize result. The simplified overview is shown in Fig. 2(e). The strategy refines the result globally after hyperspectral image is reconstructed in band-by-band. It can maintain the spectral curve well. Meanwhile, the performance of other metrics has also been improved to a certain extent.

At present, there are few hyperspectral image SR models based on two-stage strategy. Now, we review existing coarse-to-fine architectures. For instance, Yuan et al. [27] propose a framework by transfer learning from natural image domain to hyperspectral image domain. Its aim is to learn the common properties of the two modes. Then collaborative nonnegative matrix factorization is introduce to enforce collaborations between the observed LR hyperspectral image and the transferred HR hyperspectral image. Although it can refine transferred results, it requires many iterations and is very time consuming to calculate. Wang et al. [28] first develop a recurrent feedback network to generate each band in hyperspectral image. Then the Pseudo-3D convolution is utilized to further optimize the initial result by supervised manner. Zhang et al. [29] propose a coarse-to-fine scheme, including fusion module and adaptation module. They are utilized to respectively general image prior and specific hyperspectral image. The adaptive module here is made up of multiple convolutional layers and requires many iterations to refine the results. Different from these works, to obtain better super-resolved result, in our paper, an enhanced back-projection method is designed to optimize by unsupervised manner. Importantly, the proposed approach does require iteration.

Previous works [5, 21, 12] almost adopt this way to build model, i.e., all the bands are processed synchronously. Actually, it is not necessary because the reconstruction of the current band is mainly influenced by adjacent bands. Inspired by the pipeline in [16], we propose a novel structure for hyperspectral image SR via adjacent spectral fusion, whose flowchart is shown in Fig. 4. The overview of coarse stage mainly covers three modules, involving neighboring band partition (NBP), adjacent spectral fusion mechanism (ASFM), and feature context fusion (FCF).

Despite our method utilizes five bands to restore single band in coarse stage, it still lacks information exploration of a larger perspective in spectral dimension. As a result, the super-resolved hyperspectral image inevitably suffers serious spectral distortion by such single band SR. A natural way is to input all bands into the model to optimize the results by supervised manner. Nevertheless, this approach violates the original intention. Considering this limitation, we apply unsupervised manner to further refine initial results.

Back-projection [35] optimizes the reconstruction error through an efficient iterative strategy. For this algorithm, it usually utilizes multiple upsampling descriptors to upsample LR image and iteratively calculate the reconstruction error. Currently, back-projection is widely introduced in natural image SR [36, 37, 38], which has been proved to develop the quality of SR image. Yang et al. [36] employ back-projection to refine results produced according to coupled dictionaries through unsupervised strategy. However, the initialization which leads to an optimal solution remains unknown. The main reason is that the algorithm involves predefined hyperparameters, such as number of iteration and convolution kernel.

To extend this algorithm, we further develop back-projection without predefined hyperparameters, which is shown in Fig. 6. Specifically, given a LR image $I_{LR}$, the coarse result $U$ is obtained by coarse model $f$ according to scale factor $s$, which is expressed as

Previous studies aim to resize SR image to the same size as LR image, and measure the reconstruction error between the two. As for large scale factor $s$, this approach discards too much useful information, resulting in very little performance gain for reconstruction. Therefore, in our paper, the coarse model is utilized to scale LR image $I_{LR}$ again to obtain another initial SR result $V$ for scale factor $\frac{s}{2}$, i.e.,

Then, the coarse result $U$ for scale factor $s$ is downsampled using Bicubic. After generating LR image $M$, the reconstruction error is reckoned by

To yield high spectral fidelity, we introduce Spectral Angle Mapper (SAM) [39] to constrain error, which is represented as

where $arccos(\cdot)$ is arccos function. $<\cdot,\cdot>$ is dot product. $||\cdot||_{2}$ denotes the L2 norm. The obtained error is upsampled using Bicubic and added to the coarse result $U$. Eventually, the fine result $I_{SR}$ is obtained by

Our method abandons the convolution operation executed in [36]. Importantly, the strategy does not require iteration. Since the refinement is conducted using unsupervised manner, it can be applied to any hyperspectral image SR algorithms to further optimize SR image, which indicates it belongs to a plug-and-play method. For the description of dual-stage SR algorithm, the procedures are summarized in Algorithm 1.

The CAVE dataset [40] was captured by a tunable filter and a cooled CCD camera. It contains 32 scenes for a wide variety of objects and materials, which is grouped into 5 parts. The range of full spectral resolution is from 400 $n$m to 700 $n$m at 10 $n$m steps, which means that each scene has 31 bands. The spatial resolution of scene is $512\times 512$ pixels. Unlike CAVE dataset, Harvard dataset [41] is more challenging. It was collected under daylight, both outdoors and indoors. The dataset contains 71 hyperspectral images. Each scene consists of $1392\times 1040$ pixels and 31 spectral bands in the range of 420 $n$m to 720 $n$m. In our work, we randomly select 80% of the data as the training set and the rest for testing. We augment the given training data by choosing 24 patches. With respect to each patch, its size is scaled 1, 0.75, and 0.5 times, respectively. We rotate these patches $90^{\circ}$ and flip them horizontally. Through various blur kernels, we then subsample these patches into LR hyperspectral images with the size of $L\times 32\times 32$.

For our network, the convolution kernel after concatenation is set to $1\times 1$, which reduces the number of channels. We adopt sub-pixel convolution layer to upscale the features into HR space in terms of upsampling operation. The kernel of other convolution operations involved in the network is fixed to $3\times 3$, and the number of convolution kernels is defined as 64. In the training phase, our network is trained using $L1$ loss function. The mini-batch is set to 64. We optimize our network using ADAM optimizer with $\beta_{1}$ = 0.9 and $\beta_{2}$ = 0.999 and initial learning rate $10^{-4}$. For learning rate, it is gradually updated by a half at every 30 epochs. Our model runs on PyTorch framework and is trained with NVIDIA GTX 1080 GPU.

To quantitatively evaluate the proposed method, we apply Peak Signal-to-Noise Ratio (PSNR), Structural SIMilarity (SSIM), and Spectral Angle Mapper (SAM). Among these metrics, PSNR and SSIM are to evaluate the performance of super-resolved hyperspectral image in spatial domain. Generally, the higher their values are, the better the performance is. SAM is to analyze the performance of restored image in spectral domain. The smaller the value is, the less the spectral distortion is.

To verify the effectiveness of key modules, we test variant versions of our model by removing each component, including inter-group fusion, intra-group fusion, and FCF.

Table I shows the ablation study for key modules. For clear show, we only provide the results evaluated by CoarSR. Specifically, with intra-group fusion, the network directly exploits 2D convolution to explore spatial information. In contrast with only the existence of inter-group fusion, the network exhibits better performance. It implies that the model should pay more attention to the exploration of spatial features and contribute to generating sharper results, which is consistent with the original intention of our design. When two kinds of group fusion are attached, the network tends to bring fine results. It indicates the combination of the two effectively integrates complementary information between different groups.

Finally, all three modules are involved in feature extraction. One can observe that this combination produces better performance than any other arbitrary module combination, except for SAM. It is concluded that these quantitative analyses revels the effectiveness and benefits of the proposed modules.

To further refine result obtained by CoarSR, we propose an enhanced back-projection under spectral angle constraint. In this section, we study the effectiveness of enhanced back-projection on bicubic SR kernel. Note that the proposed enhanced back-projection involves two different scale factors during fine stage. Therefore, the results are evaluated for scale factors $\times 4$ and $\times 8$ on two public datasets.

Using known bicubic downsampling condition, we compare our proposed DualSR with existing multiple approaches on CAVE and Harvard datasets, including 3D-FCNN [14], EDSR [42], SSPSR [25], MCNet [21], SFCSR [21], ERCSR [4]. Next, we investigate the performance by quantitative evaluation and qualitative comparison.

In this section, we show results on various blur kernel conditions to demonstrate generalizability. For fair comparison, each image is subsampled by different downsampling kernels using [43]. The LR images are input into the model to get the corresponding restored images, and the numerical results are obtained by calculating the average value. Note that the downscaling kernels in our work are cubic, lanczos, box, and linear.