SIPSA-Net: Shift-Invariant Pan Sharpening with Moving Object Alignment for Satellite Imagery

Our contributions can be summarized as follows:

• $\bullet$

  Alignment-aware pan-sharpening: Except for the global registration, none of the previous PS methods considered the extreme local misalignment induced by moving objects. We propose the first deep-learning-based PS method that can generate both locally and globally well-aligned PS images from misaligned PAN-MS image pairs.

• $\bullet$

  Feature alignment module: The newly proposed feature alignment module learns a probability map of offsets in the feature domain for the aligned MS pixels with respect to the pixel locations in the PAN image to cope with both global and local misalignment. The probability map and the features extracted from the MS image are then used to generate the aligned MS image with respect to its corresponding PAN image.

• $\bullet$

  Shift-invariant spectral (SiS) loss: The conventional deep-learning-based PS methods have optimized their networks by using the misaligned MS images as the pseudo-ground-truth. However, this approach leads to artifacts in the PS outputs such as double-edge and blur artifacts. To remedy this, a SiS loss is proposed, which is effective in transferring the spectral information from the misaligned MS images to match the corresponding details in PAN images. SiS loss is calculated as the minimum difference between the output PS image and each of the multiple-shifted versions of MS input images. In this way, the loss becomes shift-invariant in color registration of the misaligned MS image to the PAN image.

Many conventional PS methods have been developed over the last decades, such as multi-resolution analysis methods [15, 19], component substitution algorithms [5, 9, 18], and model-based algorithms [3, 4, 8].

The multi-resolution analysis (MRA) methods [15, 19] are based on the idea that the PAN images have high-frequency details which are not present in MS images. Such high-frequency details are separated from the PAN images using multi-scale decomposition techniques and then adaptively fused with the upsampled version of the MS images.

The component substitution (CS) algorithms separate the spatial and spectral information of the MS images using a specific transformation, and then the spatial component is substituted with the PAN image. The CS-based algorithms include pan-sharpening methods based on an intensity hue-saturation technique [5], principal component analysis [18], and Brovey transforms [9]. These methods are extremely fast because they only need to upscale the MS input image and apply some spectral transformation to separate and replace the spatial component.

The model-based algorithms [3, 4, 8] treat the PAN and MS images as the spectral and spatial degraded versions of the ideal PS images. Under such assumption, pan-sharpening becomes a restoration problem aiming to reconstruct the ideal PS images from their degraded observations (PAN and MS input). Model-based algorithms optimize an objective function designed with some prior knowledge that is independent of specific training data. The methods require high computational complexity compared to the previously mentioned methods due to the optimization process.

Recently, many deep-learning-based pan-sharpening methods [6, 12, 16, 25, 27] have been proposed. The architectures of such methods are based on convolutional neural networks (CNNs). Many types of CNN architectures for pan-sharpening have been proposed, and the proposed methods have shown a large margin of performance improvements over the conventional methods.

The goal of the pan-sharpening task is to obtain high-resolution PS images. Without the PAN images as guidance, this solely becomes a super-resolution (SR) task. Therefore, most previous deep-learning-based pan-sharpening methods have borrowed the CNN architectures of existing SR frameworks [7, 13, 26]. However, to make good use of the high-resolution PAN images, loss functions should be carefully designed to help pan-sharpening networks maintain the high-frequency details from PAN images and spectral information from MS images.

The first deep-learning-based pan-sharpening method is PNN [16]. The network of PNN is based on the SRCNN [7], the first CNN-based SR method. The network of PNN is composed of three convolutional layers. It is optimized to minimize the difference between lower-scaled pan-sharpened images and target MS images of the original-scale, which are given pseudo-ground-truths. Yang et al. proposed PanNet [25] which trains their network in the high-pass filtered domain rather than the image domain for better spatial structure preservation. DSen2 [12] shares a similar idea with PanNet in their network design and loss functions. The network architectures of both PanNet and DSen2 are based on an SR network called VDSR [11], which have improved the SR quality over SRCNN. Zhang et al. proposed BDPN [27] which utilizes a bi-directional network design to fuse PAN and MS images.

However, there are some difficulties when generating PS images from misaligned PAN-MS image pairs. S3 [6] is the first paper that considers the artifacts that come from such misalignment. S3 proposes spectral and spatial loss functions based on the correlation maps between MS and PAN inputs. The correlation maps are used as weights for the two-loss functions. They put more weights on the spectral loss function for the areas with higher correlation values, and those having lower correlation values have higher weights for the spatial loss function. This approach has improved the visual quality of PS outputs by reducing the artifacts from the misalignment. However, S3 has a critical drawback that the areas with lower correlation values (regions with large misalignment) are mostly affected by the spatial loss. Therefore the original colors from MS images are diminished. An appropriate way to handle the misalignment would be moving the color of an object on the MS image to match its shape on the corresponding PAN image.

As mentioned earlier, one of the main limitations of the previous CNN-based pan-sharpening methods is that their networks are simply trained with L1 or L2 loss functions for the misaligned MS-PAN image pairs without any registration a priori. This results in the PS output with various artifacts such as color bleed and double edges. However, direct alignment between the PAN and MS images are very difficult due to their domain discrepancy with different signal characteristics and resolutions. Instead, the alignment might be better solved in the feature domain. So, we propose a novel feature alignment module (FAM) that aligns each MS input image to its corresponding PAN image in the feature domain, yielding an aligned MS image that is later fed into the pan-sharpening module (PSM) of the SIPSA-Net. As shown in Fig. 2, the resulting aligned MS image $\textbf{I}^{align}_{MS}$ has the same size as the input MS image $\textbf{I}_{MS}$ of size W$\times$H$\times$3 and is aligned to its corresponding downscaled PAN image $\textbf{I}^{down}_{PAN}$ of size W$\times$H. The alignment in the FAM helps the PSM easily generate the robust and stable PS output $\textbf{I}_{PS}$ with a good alignment between colors and shapes, given an input MS-PAN pair with large misalignment.

As shown in Fig. 2, the FAM includes two feature extractors: alignment and MS feature extractors. The alignment feature extractor learns a pixel-wise offset probability map (PWOPM) of size W$\times$H$\times$81 to align the features $\textbf{F}_{MS}$ of MS input $\textbf{I}_{MS}$ to the features $\textbf{F}_{PAN}$ of the corresponding PAN input $\textbf{I}_{PAN}$ of size 4W$\times$4H$\times$1, thus yielding the aligned MS features $\textbf{F}^{align}_{MS}$. The offset value represents the amount of misalignment between PAN and MS features, which can be further utilized to align two features in following operations. The MS feature extractor converts $\textbf{I}_{MS}$ in pixel-domain to $\textbf{F}_{MS}$ in feature-domain. The output of the FAM for the concatenated input of $\textbf{I}_{MS}$ and the space-to-channel-rearranged PAN input $\textbf{I}^{s2c}_{PAN}$ of size W$\times$H$\times$16 is the PWOPM where a 9$\times$9-sized alignment offset probability map is positioned at each pixel location along the 81-depth channel dimension. Each 9$\times$9-sized alignment offset probability map is obtained via a softmax function, which is then convolved with the 9$\times$9-sized local region centered at the corresponding pixel location. Note that all the feature channels share the 9$\times$9-sized alignment offset probability map at the same pixel locations. By doing so, the aligned MS features $\textbf{F}^{align}_{MS}$ are obtained.

As illustrated in the right upper part of Fig. 2, an 81-dimensional feature vector in the PWOPM is shown in a rearranged 9$\times$9-sized offset probability map where the red circle indicate the center pixel location (0,0) and the green circle in the (-1, 2) location has the highest offset probability value. This indicates that the MS feature at the (-1, 2) location is likely to be moved to the center pixel location. The aforementioned FAM operation is expressed in Eq. 1. Let the PWOPM be $\textbf{M}=[M_{0,0},...,M_{W,H}]$, $M_{x,y}\in\mathcal{R}^{9\times 9}$. By the pixel-wise probabilistic alignment convolution (PWPAC), the extracted MS feature $\textbf{F}_{MS}$, $\textbf{F}_{MS}\in\mathcal{R}^{W\times H\times C}$ can be point-wise aligned to $\textbf{I}^{down}_{PAN}$. The resulting aligned c-th feature map can be expressed as

where $1\leq x\leq W,1\leq y\leq H$, and $1\leq c\leq C$. It should be noted that our PWPAC is distinguished from pixel-wise adaptive convolution (PAC) [20]. The PAC operates local convolution with learned filter values for up-sampling while the PWPAC performs pixel-wise alignment in a probabilistic sense, i.e., probabilistic alignment convolution, for the extracted features of an MS input with respect to the local features of the PAN input.

Fig. 3 shows the feature extraction for the MS input by the MS feature extractor. The MS feature extractor extracts a feature map $F^{\textit{pre}}_{MS}$ and expands it to a set of feature maps, $\textbf{F}_{MS}$, via point-wise patch vectorization operations that rearrange each 9$\times$9-sized patch in stride 1 into the channel dimension. $\textbf{F}_{MS}$ is a set of point-wise shifted versions of $F^{\textit{pre}}_{MS}$ from (-4,-4) to (4,4). Therefore, the shifted version by (0, 0), which is the middle channel, is identical to $F^{\textit{pre}}_{MS}$. In this way, Eq. 1 performs probabilistic alignment operation over the set of shifted versions of $F^{\textit{pre}}_{MS}$ by PWOPM M.

The pan-sharpening module (PSM) in Fig. 2 performs the pan-sharpening based on the aligned MS input $\textbf{I}^{align}_{MS}$ and the space-to-channel-rearranged PAN input $\textbf{I}^{s2c}_{PAN}$ of size W$\times$H$\times$16. After the first five ResBlocks, the intermediate output is up-scaled 4 times via the Pixel-Shuffle layer. The 2D gradient of $\textbf{I}_{PAN}$ is concatenated to the output of the second two ResBlocks, which are then fed as input into the last convolution layer. It is worthwhile to note that the 2D gradient helps maintain the details of the PAN input $\textbf{I}_{PAN}$ while producing the final PS output $\textbf{I}_{PS}$.

For the training of SIPSA-Net, two different types of loss functions are utilized to generate accurate pan-sharpened images. For this, the edge detail loss and the shift-invariant spectral loss are incorporated for spatial and spectral information preservation, respectively.

The results of the proposed SIPSA-Net are compared with (i) seven non-deep-learning PS methods including Brovey transform [9], affinity PS [21], guided-filtering-based PS [22], intensity-hue-saturation (IHS) PS [5], principal component analysis (PCA) PS [18], P+XS PS [4] and variational PS [8], and (ii) five deep-learning-based PS methods including PNN [16], PanNet [25], DSen2 [12], and their variants trained with S3 loss [6], called PanNet-S3 and DSen2-S3, respectively. For testing, we randomly selected 100 PAN-MS image pairs in the WorldView-3 test dataset.

Ablation studies have been conducted in three different conditions to show the effectiveness of key aspects of our proposed SIPSA-Net. Throughout the experiments, the SIPSA-Net has been inspected by removing each key component from its full version. The resulting different versions (models) have been evaluated in the full-scale using the original MS and PAN images as inputs for the networks. We use three metrics to measure the performance of the different versions: SCC_F[28], ERGAS[24], and JQM [17].

Table 2 provides the performance of the different versions with/without the SiS loss and FAM. Also, there is a version that applies PWPAC directly in pixel domain for MS input images, called ’SIPSA-Net w/ PAM (pixel alignment module)’. The ’SIPSA-Net w/o SiS loss’ version is trained without the SiS loss, and the FAM is excluded for the ’SIPSA-Net w/o FAM’ version that directly takes the original MS images as input without alignment. As can be seen in Table 2, the full version of the SIPSA-Net shows the best performance in terms of JQM. The ’SIPSA-Net w/o SiS loss’ version is superior to all the other versions in terms of SCC_F and ERGAS and the full version is the second best. Without the SiS loss, the SISPA-Net suffers from various artifacts such as comet tails, false color and misalignment as in Fig. 6-(n) where the big red and blue spots (color artifacts) nearby the car appear. It is also worthwhile to mention that for the ’SISPA-Net w/o FAM’ version that the FAM plays a crucial role in generating the PS images of high visual quality as shown in Table 2 (see also Fig. 3 in the Supplemental material). Lastly, the ’SIPSA-Net w/ PAM’ version is difficult to be well trained because the MS images and their corresponding PAN images have different signal characteristics and are difficult to be directly aligned in pixel domain rather than in feature domain.