Decomposition-based Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentation

Leveraging encoder-decoder-based networks, numerous methods of semantic segmentation have emerged in the field of remote sensing [36]. Most existing methods are developed based on CNNs with residual learning and attention mechanisms to boost the representation learning [37]. To address the limitations of CNNs in modeling global representations, many transformer-based methods [38, 39] have been integrated under the encoder-decoder framework, leveraging the global context modeling capability of the self-attention mechanism [26, 40]. Recently developed remote sensing foundation models attempt to achieve domain-generalized performance by leveraging large-scale datasets and vision models [41, 42, 43]. For downstream tasks on specific remote sensing datasets, transfer learning or fine-tuning is required.

In contrast to the straightforward semantic segmentation task, cross-domain semantic segmentation focuses on transferring knowledge from the source domain to the target domain. According to the transfer strategy, existing cross-domain semantic segmentation models can be divided into two main categories, namely the self-training approach [44, 45] and the adversarial training approach [14, 21]. The former generates pseudo-labels as the supervision information of the target domain using a model trained on the labeled source domain [46]. However, pseudo-labeling inevitably introduces noise into model training, typically requiring a pre-defined threshold to filter out low-confidence pseudo-labels. Alternatively, the adversarial training approach focuses on aligning distributions of two domains at the image feature or output levels. The key to this method lies in the stability of the training process. Recently, there has been an increase in methods that integrate self-training techniques with adversarial learning to address UDA challenges, using a multi-stage training strategy [44, 47, 48]. This work aims to develop a versatile and efficient framework based on adversarial training. Compared to these methods, this work develops an effective one-stage UDA framework with promising performance based on adversarial training.

GAN-based UDA algorithms have been proven effective in cross-domain semantic segmentation for remote sensing imagery. [49] proposed the first GAN-based model designed for cross-domain semantic segmentation from aerial imagery. On this basis, TriADA [44] introduced a triplet branch to improve the CNN-based feature extraction by simultaneously aligning features from the source and target domains. Recently, a self-training approach that employs high-confidence pseudo-labels was applied to incorporate category information [48, 33, 50]. Furthermore, SADA [31] and CS-DDA [20] developed subspace alignment methods to constitute a shared subspace with high-level features. To improve the high-level feature extraction, attention mechanisms were introduced into GAN-based UDA. More specifically, CAGAN [51] developed a covariance-based channel attention module to compute weighted feature maps, whereas CCAGAN [32] used a category-certainty attention module to align the category-level features. MASNet [52] stored domain-invariant prototypical representations by employing a feature memory module while MBATA-GAN [21] aligned high-level features by constructing a cross-attention-based transformer. Compared with the methods above, this work attempts to fully explore global-local context modeling in the generator and discriminator, as shown in Table I. Therefore, it can capture cross-domain global semantics and local detailed contexts while boosting the learning of domain-invariant features.

The core idea of decomposition-based UDA is to project features into different subspaces before performing domain alignment in the corresponding subspaces. For instance, ToMF-B [58] decomposed the cross-domain features into task-related features and task-irrelevant features by analyzing the gradients of the predicted score corresponding to the labels. Furthermore, ST-DASegNet [59] introduced a domain disentangled module to extract cross-domain universal features and improve single-domain distinct features in a self-training framework whereas DSSFNet [60] decoupled building features to learn domain-invariant semantic representations and domain-specific style information for building extraction. However, these techniques mainly concentrate on aligning high-level abstract features, overlooking the relationship between high/low-frequency details and the characteristics of ground objects. Therefore, it is crucial to address this shortfall by delving into frequency decomposition technology.

Before introducing our proposed UDA framework, we first revisit the basics of the multi-head self-attention (MHSA) design and the convolutional operation (Conv) from a frequency perspective. We employ the discrete Fourier transform (DFT) to analyze the feature maps regarding specific frequency components generated by MHSA and Conv, respectively. Let $\bm{X}\in\mathbb{R}^{H\times W}$ and $\bm{Y}\in\mathbb{R}^{C}$ represent an image in the spatial domain and its corresponding label, where $C$ denotes the number of classes. We transform $\bm{X}$ into the frequency spectrum using the DFT $\mathcal{F}$: $\mathbb{R}^{H\times W}\rightarrow\mathbb{C}^{H\times W}$ and revert the signals of the image from frequency back to the spatial domain using the inverse DFT $\mathcal{F}^{-1}$: $\mathbb{C}^{H\times W}\rightarrow\mathbb{R}^{H\times W}$. For a mask $m\in\{0,1\}^{H\times W}$, the low-pass filtering $\mathcal{M}^{S}_{\mathrm{low}}$ of size $S$ can be formally defined as [61]:

where

where $\odot$ stands for the Hadamard product, and $m_{i,j}$ denotes the value of $m$ at position $(i,j)$.

Similarly, the high-pass filtering $\mathcal{M}^{S}_{\mathrm{high}}$ is defined as

where

According to the definitions in Eq. (2) and Eq. (4), the low-frequency components are shifted to the center of the frequency spectrum while the high-frequency components are shifted away from the center of the frequency spectrum.

Fig. 2 illustrates the contrasting behaviors that MHSA and Conv demonstrate. As shown in Fig. 2(b) and (d), MHSA aggregates feature maps, whereas Conv disperses them. Furthermore, as depicted in Fig. 2(c) and (e), Fourier analysis of feature maps reveals that MHSA focuses on low-frequency components, while Conv amplifies high-frequency components. In other words, MHSAs function as low-pass filters, while Conv serves as high-pass filters. Consequently, MHSA and Conv complement each other and can be jointly employed for frequency decomposition. More details can be found in [62].

Based on the discussions above, we propose to utilize the existing GLTB for the frequency decomposition. In particular, the global-local attention in GLTB consists of two parallel branches, namely the global and local branches, as presented in Fig. 3(a). The former captures the global context between windows by exploiting the window-based multi-head self-attention (W-MHSA) [40] while maintaining the spatial consistency of ground objects through Average Pooling across windows, while the latter has a structure of two parallel convolutional layers with kernel sizes of $3$ and $1$, respectively. Each convolutional layer is followed by a batch normalization operation. The global-local attention leverages its two branches to investigate and incorporate global-local contextual information. Hereby, the outputs from the global and local branches are fused together before the resulting global-local context is further characterized by two convolution layers and a batch normalization operation.

It is observed that GLTB satisfies the requirements for frequency decomposition that needs a branch based on MHSA and a branch based on Conv. Therefore, we propose the HLFD module as depicted in Fig. 3(b) by exploiting the GLTB structure to decompose multiscale features. However, in sharp contrast to GLTB, which is utilized for global-local information aggregation, the HLFD module utilizes two parallel branches for the decomposition. More specifically, a low-frequency extractor is designed to capture the low-frequency components by utilizing the W-MHSA while a high-frequency extractor is employed to extract high-frequency components using Conv operation. The output low- and high-frequency components are then processed by a LayerNorm (LN) layer and a multilayer perceptron (MLP), respectively. Finally, a novel model called De-GLGAN is established by capitalizing on the GLTB and HLFD modules.

Typical UDA tasks involve a labeled source dataset $D_{s}=\left\{(\bm{X}_{s},\bm{Y}_{s})\right\}^{n_{s}}$ and an unlabeled target dataset $D_{t}=\left\{\bm{X}_{t}\right\}^{n_{t}}$ where $\bm{X}_{s}$ represents an image in the source domain with its corresponding label $\bm{Y}_{s}$ whereas $\bm{X}_{t}$ denotes the unlabeled image in the target domain. Furthermore, $n_{s}$ and $n_{t}$ stand for the sample size of the source domain and the target domain, respectively. As the source domain and target domain possess different marginal and conditional distributions, i.e., $p_{s}(\bm{X}_{s})\neq p_{t}(\bm{X}_{t})$, the domain shift problem occurs. We assume that certain inherent similarities exist between different domains, which is valid as both domains share the same segmentation output space [11, 21].

As depicted in Fig. 4, the proposed De-GLGAN is designed by capitalizing upon frequency decomposition and global-local context utilization. Specifically, multiscale HLFD modules are presented to align cross-domain representations by exploiting multiscale features generated by the encoder. Moreover, the global-local discriminator named GLDis further exploits the decomposed features to learn multiscale domain-invariant representations. In particular, the GLDis attempts to mimic the decoder to learn the domain-specific output space using the global-local representations.

As illustrated in Fig. 5, the encoder comprises a Patch Partition layer and four successive stages. More specifically, the first stage contains one embedding layer followed by the SwinTBs, whereas the following three stages are equipped with one merging layer followed by each SwinTB. We denote by $\bm{X}_{s}\in\mathbb{R}^{{H\times W\times Z}}$ and $\bm{X}_{t}\in\mathbb{R}^{{H\times W\times Z}}$ the training images from the source and target domains, respectively. Furthermore, $Z$ stands for the number of image channels, whereas $H$ and $W$ represent the height and width of the image, respectively. $\bm{X}_{s}$ and $\bm{X}_{t}$ are first split into non-overlapping patches of size $4\times 4$ by the Patch Partition layer. After that, the first-stage Embedding layer is applied to project these patches onto the encoding space before the SwinTBs gather image information on these patch tokens. As shown in Fig. 5, the SwinTB consists of four LN layers, two MLPs, one W-MHSA and one shifted window-based multi-head self-attention (SW-MHSA) [26, 40]. After the first stage, three more successive stages are applied to generate the multiscale features denoted as $\bm{F}_{s}^{\emph{1-4}}$ and $\bm{F}_{t}^{\emph{1-4}}$ both of size $d_{i}\times\frac{H}{2^{i+1}}\times\frac{W}{2^{i+1}}$ where $d_{i}$ is the encoding dimension with $i\in\left\{1,2,3,4\right\}$ being the stage index. After that, these multiscale features are then fed into the corresponding HLFD module for frequency decomposition. Notably, the images from the source and the target domain are fed into the generator successively, that is, $\bm{F}_{s}^{\emph{1-4}}$ and $\bm{F}_{t}^{\emph{1-4}}$ are generated and collected, respectively.

HLFD modules are employed to decompose the multiscale feature maps $\bm{F}_{s}^{\emph{1-4}}$ and $\bm{F}_{t}^{\emph{1-4}}$, as illustrated in Fig. 6. The feature map at each scale, including the features from the source domain and the target domain, will be decomposed into the high or low-frequency feature maps by the corresponding scale HLFD. After that, the final outputs of the HLFD modules denoted as $\bm{LF}_{s}^{\emph{*}}/\bm{LF}_{t}^{\emph{*}}$ and $\bm{HF}_{s}^{\emph{*}}/\bm{HF}_{t}^{\emph{*}}$ are used to compute the frequency alignment loss $L_{\rm{f}}$, which will be elaborated in Sec. III-F.

The decoder is developed based on GLTB [35] and designed to exploit the decomposed features by gradually extracting global and local information, as shown in Fig. 7(a). Specifically, the high-level feature maps derived from the encoder are first fed into an individual GLTB whose output is processed by two stages of weighted sum operation and GLTB. In particular, GLTB is able to capture the global context and local details simultaneously. After that, the weighted sum operation is employed to extract domain-invariant features by adaptively fusing the residual features and those from the previous GLTB. After the first three stages of the GLTB-based decoding, a weighted sum operation is used to fuse the residual connection from the first stage of the encoder and the deep global-local feature derived from previous GLTB before a classifier produces the final segmentation maps denoted by $\bm{P}_{s}$ and $\bm{P}_{t}$ of size $\mathbb{R}^{H\times W\times C}$ for the source and target domains, respectively, where $C$ stands for the number of ground object categories.

Finally, the supervised segmentation loss $L_{\rm{seg}}$ and the adversarial loss $L_{\rm{adv}}$ are also derived in the generator optimization stage. More specifically, $L_{\rm{seg}}$ is computed based on the source labels while $L_{\rm{adv}}$ is calculated using the adversarial strategy. More details about the loss functions will be provided in Sec. III-F.

In the proposed discriminator, GLDis, we employ GLTB to exploit the semanftic output space produced by the generator. As shown in Fig. 7, the GLDis and the decoder are similar in structure but different in function. Specifically, it first down-samples the weighted self-information (WSI) distribution $\bm{I}_{s}$ and $\bm{I}_{t}$ derived from $\bm{P}_{s}$ and $\bm{P}_{t}$ before feeding the down-sampled data into a GLTB. Given the output segmentation maps of the source domain $\bm{P}_{s}$ and the target domain $\bm{P}_{t}$, the corresponding self-information is defined as $-\mathrm{log}\bm{P}_{s}$ and $-\mathrm{log}\bm{P}_{t}$, respectively. Consequently, the WSI maps $\bm{I}_{s}$ and $\bm{I}_{t}$ are calculated as follows:

where the dimensions of $\bm{P}_{s}$, $\bm{P}_{t}$, $\bm{I}_{s}$ and $\bm{I}_{t}$ are $\mathbb{R}^{H\times W\times C}$, and $(h,w)$ represents the index position of one pixel. These information maps are regarded as the disentanglement of the Shannon Entropy. After the first GLTB, the resulting data is further processed by two successive stages of down-sampling, followed by the GLTB, which is designed to gradually gather the global and local information in the segmentation map-generated features. Finally, the resulting data is down-sampled and processed by a classifier, which predicts the domain to which the input segmentation maps belong. Despite its appearance similar to the decoder structure, GLDis is tailored to learn domain-specific representations using an adversarial strategy. More specifically, the decoder upsamples the abstract semantic features, whereas GLDis down-samples the structured output space to identify the domain to which each input belongs. The final output of GLDis is used to compute a cross-entropy loss denoted by $L_{\rm{d}}$ in the discriminator training stage. More details about the loss function will be presented in the next section.

The proposed De-GLGAN capitalizes on four loss functions, namely the segmentation loss $L_{\rm{seg}}$, the typical adversarial loss $L_{\rm{adv}}$, the frequency alignment loss $L_{\rm{f}}$, and the cross-entropy loss $L_{\rm{d}}$. These loss functions are defined as follows.

In this work, we adopt three metrics to quantitatively assess the segmentation performance of the cross-domain remote sensing images for all UDA methods, namely, the Overall Accuracy (OA), the F1 score (F1) and the Intersection over Union (IoU), which are defined as follows:

where $TP$, $FP$, $TN$ and $FN$ are true positives, false positives, true negatives and false negatives, respectively. Furthermore, the mean F1 score (mF1) and the mean IoU (mIoU) for the main five classes apart from Clutter/Background are calculated to measure the average performance of the UDA methods following the common practice reported in the literature [44, 21].

In addition, the computational complexity of the proposed GLGAN and De-GLGAN is evaluated using the following metrics. First, the giga floating point operation counts (GFLOPs) are employed to evaluate the model complexity. Second, the number of model parameters and the memory footprint are utilized to evaluate the memory requirement. Finally, the frames per second (FPS) is used to evaluate the execution speed. For a computationally efficient method, its first three metrics are usually small, while its FPS value should be large. Each method undergoes training for $50$ epochs, with each epoch consisting of $1000$ batches. During training, the performance of each method is evaluated every 100 batches and recorded. The best-performing model for each method is then selected for subsequent testing based on its highest segmentation performance.

In our work, the basic unit of the encoding block in the generator is adopted from the Swin Transformer [40] pretrained on ImageNet. In the experiments, two versions of Swin Transformers, namely Swin-Base and Swin-Large, were applied for image feature extraction. Both Swin-Base and Swin-Large have four stages with the stack of $\left\{2,2,18,2\right\}$ SwinTBs with the embedding dimensions of Swin-Base and Swin-Large being $\left\{128,256,512,1024\right\}$ and $\left\{196,384,768,1536\right\}$, respectively. Furthermore, the number of the heads in Swin-Base and Swin-Large are $\left\{4,8,16,32\right\}$ and $\left\{6,12,24,48\right\}$, respectively. In addition, the window size for W-MHSA and SW-MHSA is $16\times 16$. The dimension of both the decoder and GLDis is set to $256$, whereas the window size of the global-local attention is $8\times 8$.

To confirm the effectiveness of the proposed methods, some representative existing UDA methods were selected for quantitative comparison. Table III summarizes the nine methods evaluated in our following experiments. Notably, in addition to the first three representative methods in the field of computer vision, the other six methods are specifically designed for remote sensing images. Furthermore, we also performed comparison experiments on two non-DA methods for the first three UDA tasks on the ISPRS Potsdam and Vaihingen dataset, namely a deep model without domain adaptation (labeled as “Source-only”) and a supervised deep model (labeled as “Supervised”) trained with labeled data from the target domain. These two non-DA methods stand for the two extreme cases, with the “Supervised” model representing the optimal performance that any UDA method targets and the “Source-only” model standing for the performance lower bound. Furthermore, the classical ResUNet++ [65] is selected for these two non-DA methods by taking into account the balance between segmentation performance and computational complexity. For the aforementioned methods, the same network architectures, training parameters, and evaluation methods were applied to make a fair comparison.

The comparative results reveal a performance gap between our method and other methods. This is because the proposed framework improves both the generator and discriminator of the GAN framework. In summary, the extensive experiments demonstrated that the proposed GLGAN and De-GLGAN have the capability of overcoming domain shifts, leading to significantly improved semantic segmentation performance on different remote sensing scenarios.

Given that the proposed De-GLGAN incorporates both GLTB and HLFD, we conducted three sets of ablation studies to demonstrate the importance of each component.

In this section, the sensitivity of hyper-parameter $\lambda^{f}$ is analyzed. This parameter is designed to regulate the contribution derived from frequency decomposition. Inspection of the results presented in Fig. 15 indicates that a smaller value of $\lambda^{f}$ diminishes the importance of aligning high/low-frequency information, whereas a larger value may exaggerate the significance of the frequency alignment. Consequently, setting $\lambda^{f}>0.05$ results in notable performance degradation. Conversely, both performance demonstrates less sensitivity to $\lambda^{f}$ within the range $\lambda^{f}\in\left[0.008,0.02\right]$. Therefore, we set $\lambda^{f}=0.01$ in our experiments, which performs effectively across the five UDA tasks, suggesting its strong applicability.

In this section, we will investigate the characteristics of the proposed multiscale HLFD modules by visually inspecting the heatmaps before and after the frequency decomposition. Specifically, multiscale feature maps are first resized to a uniform image size, followed by averaging across all channels. Subsequently, pseudo-color processing is applied. The results are depicted in Fig. 16. Upon inspecting the feature maps before and after the decomposition in both the source and target domains, distinctions become apparent between low- and high-frequency information: low-frequency features possess global characteristics, whereas high-frequency features appear more discrete and localized. In particular, even the fourth scale (high-level) feature maps show significant differences. It fully demonstrates that our HLFD modules successfully decompose high- and low-frequency information.

Comparison on the feature maps between the source and target domains reveals that cross-domain high-frequency features share great similarities as well as the cross-domain low-frequency features. It suggests that the model’s capacity to decompose features acquired from the labeled source domain can be successfully applied to the target domain. In other words, the model has learned domain-invariant features from both high- and low-frequency, which correspond to the similarity of spatial details across domains that we emphasized in Sec. I and the global context similarity that the existing works explored. Our work has proved that both of these components are necessary for cross-domain alignment in the UDA method.

Table XII shows the computational complexity of all UDA methods discussed. Compared with the baseline Advent, the proposed general backbone GLGAN (Swin-Base) achieved greatly improved segmentation performance with significantly reduced computational complexity as its hierarchical window-based attention has linear computation complexity to input image size [40]. More specifically, the computational complexity of GLGAN (Swin-Base) is only half that of Advent. Furthermore, GLGAN (Swin-Large) has computational complexity comparable to Advent. However, GLGAN (Swin-Large) requires much more memory to store its parameters. Overall, the proposed GLGAN harvests better performance at the cost of moderately increased computational complexity. The model complexity of De-GLGAN is between GLGAN (Swin-Base) and GLGAN (Swin-Large), yet it exhibits superior performance. It proved that the HLFD module is of great significance in practical application.