Integrating Multiple Sources Knowledge for Class Asymmetry Domain Adaptation Segmentation of Remote Sensing Images

Accurately labeling RSIs is a very complicated and time-consuming work [10, 11, 12]. To improve the generalization ability of deep segmentation models, more and more attention has been paid to the researches of RSIs domain adaptation. From an intuitive visual perspective, the differences between RSIs mainly exist in color and other style attributes. Therefore, the initial exploration mainly focuses on RSIs translation, aiming to reduce the discrepancies between the source and target domains by unifying the styles of different RSIs. Tasar et al. present a color mapping GAN, which can generate fake images that are semantically exactly the same as real training RSIs [13]. Sokolov et al. focus on the semantic consistency and per-pixel quality [14], while Zhao et al. introduce the depth information to improve the quality of synthetic RSIs [15]. Only the implementation of style transfer in the input space often produces unstable domain adaptation performance. Therefore, obtaining the domain-invariant deep features through adversarial learning has gradually become the mainstream in RSIs domain adaptation segmentation. Cai et al. develop a novel multitask network based on the GAN structure, which possesses the better segmentation ability for low-resolution RSIs and small objects [16]. Zhu et al. embed an invariant feature memory module into the conventional adversarial learning framework, which can effectively store and memorize the domain-level context information in the training sample flow [17]. Zheng et al. improve the high-resolution network (HRNet) according to the RSIs characteristics, making it more suitable for RSIs domain adaptation. In addition, the attention mechanism [18, 19], contrastive learning [20], graph network [21] and consistency and diversity metric [22] are also integrated into the adversarial learning framework, further improving the segmentation accuracy of target RSIs. Recently, the self-supervised learning based on the mean teacher framework [23] has been gradually applied in RSIs domain adaptation due to its excellent knowledge transfer effect and stable training process. Yan et al. design a cross teacher-student network, and improve the domain adaptation performance on target RSIs through the cross consistency constraint loss [24]. Wang et al. focus on the problem of spatial resolution inconsistency in self-supervised adaptation, effectively improving the effect of knowledge transfer from airborne to spaceborne RSIs [25]. In addition, combining the above different types of methods is a common idea to further improve the performance of RSIs domain adaptation [26].

Although various UDA methods for RSI semantic segmentation have sprung up, they all follow a common ideal assumption of class symmetry. Under the condition that the source and the target class set are different, the existing methods often fail to achieve the satisfactory performance.

The effective utilization of multiple RSIs sources can provide more abundant and diverse knowledge for improving RSIs domain adaptation performance. However, most of the existing UDA methods of RSIs focus on the knowledge transfer from a single source to a single target domain, and there are relatively few researches tailored for multi-source RSIs domain adaptation. The existing methods can be divided into two categories: constructing a combined source and aligning each source-target pair separately. The former usually combines several different RSIs into a single source domain and performs cross-domain knowledge transfer according to the conventional UDA mode [27, 28, 29, 30]. The obvious shortcoming is that the RSIs sources with different distribution in the combined domain will interfere with each other during domain adaptation process, thus weakening the effect of knowledge transfer [31, 32]. In contrast, the latter typically aligns each source and target RSI separately, and merges the results from different sources as the final prediction of target RSIs [33, 34, 35]. The multiple feature spaces adaptation network proposed by Wang et al. is the closest to our work, which explores the performance of multi-source UDA in crop mapping by separately aligning each source-target RSI pair [35]. However, all the above work, including literature [35], require that each source and target RSI share exactly the same class space. Obviously, meeting this requirement is even more laborious and cumbersome than the conventional UDA methods in Section II-A.

Our work focuses on the problem of RSIs class asymmetry domain adaptation. Compared with existing multi-source UDA method, the proposed method can deal with the scenario closer to practical situation, that is, performing knowledge transfer using multiple RSIs sources with different class spaces.

In the field of computer vision, there are two research directions related to our work, namely incomplete domain adaptation (IDA) and partial domain adaptation (PDA). The problem setting of IDA is to utilize multiple sources with incomplete class for domain adaptation, and the existing few researches mainly focus on the image classification task [36, 37]. Lu et al. and Gong et al. introduce IDA into the remote sensing field and preliminarily explore the performance of RSIs cross-domain scene classification [38, 39], and Ngo et al. further deepen the research on this issue [40]. In addition, Li et al. propose to conduct the class-incomplete model adaptation without accessing source information, and design a deep model for street scene semantic segmentation [41]. However, the performance of this method on target domain is dissatisfactory, since it abandons the source knowledge in the domain adaptation process. Different from the IDA setting, PDA is a single-source to single-target domain adaptation task. However, in the PDA setting, the number of source classes is required to be greater than the number of target classes. In the field of computer vision, various PDA methods have been developed for the image classification task [42, 43, 44]. Meanwhile, the performance of PDA methods on RSIs scene classification has begun to be studied, and the typical methods include the weight aware domain adversarial network proposed by Zheng et al. [45].

The differences between the existing IDA and PDA methods and our work can be summarized into the two aspects. First, most of the existing methods are designed for the classification task of natural images, and the performance of these methods directly applied to the RSIs domain adaptation segmentation is greatly degraded. Secondly, our work focuses on the class asymmetry domain adaptation of RSIs with multiple sources, and can simultaneously cover the two scenarios where the source class union is equal to or includes the target class set. On the one hand, our work can directly perform RSIs domain adaptation with multiple class-incomplete sources; On the other hand, compared to the single-source PDA tasks, our work is more consistent with the practical situations where multiple sources with different class sets are available.

Formally, the problem setting of class asymmetry RSIs domain adaptation with multiple sources is first presented. There are $k$ labeled source domains $\{\mathcal{D}_{i}^{S}\}^{k}_{i=1}$ with class sets $\{\mathcal{C}_{i}^{S}\}^{k}_{i=1}$ and one unlabeled target domain $\mathcal{D}^{T}$ with the class set $\mathcal{C}^{T}$. The distribution $\mathcal{P}$ and class space $\mathcal{C}$ of each domain are known and distinct from each other, i.e., $\mathcal{P}^{S}_{1}\neq\mathcal{P}^{S}_{2}\ldots\neq\mathcal{P}^{S}_{k}\neq\mathcal{P}^{T}$, and $\mathcal{C}^{S}_{1}\neq\mathcal{C}^{S}_{2}\ldots\neq\mathcal{C}^{S}_{k}\neq\mathcal{C}^{T}$. In addition, the source class union is equal to or contains the target class set, i.e., $\mathcal{C}^{T}\subseteq\mathcal{C}^{S}_{1}\bigcup\mathcal{C}^{S}_{2}\ldots\bigcup\mathcal{C}^{S}_{k}$, and each source class set has an intersection with the target class set, i.e., $\mathcal{C}^{S}_{k}\cap\mathcal{C}^{T}\neq\emptyset$. Given source labelled samples $(x^{S_{i}},y^{S_{i}})$ and target unlabelled samples $x^{T}$, the class asymmetry domain adaptation aims to achieve knowledge transfer from multiple sources to target RSIs.

This paper proposes to integrate multiple sources knowledge for RSIs domain adaptation segmentation in the case of class asymmetry. To clearly articulate the proposed MS-CADA method, the two-source scenario is used as an example to describe its workflow, as shown in Fig. 2. The whole deep model $M$ consists of a feature extraction network $G$ and a multi-branch segmentation head. The former is shared by multiple domains for common feature extraction existing in different RSIs, while the latter includes two expert networks $E_{1}$ and $E_{2}$, two classifiers $F_{1}$ and $F_{2}$, and a knowledge integration module $H$. The expert networks are used to learn the source-specific deep features, and the module $H$ is responsible for integrating knowledge from different domains and inferring target RSIs. Overall, the proposed method follows the self-supervised adaptation mode, thus in each iteration, the teacher model $M^{\prime}$ is established based on the exponential moving average (EMA) algorithm. Specifically in each iteration, the proposed method performs three different learning tasks simultaneously, including supervised learning, collaborative learning, multi-domain knowledge transfer.

Firstly, the source RSIs and their corresponding true labels are used for model supervised training. Due to the discrepancies in class space and data distribution, different RSIs are respectively fed into the corresponding expert networks and classifiers for loss calculation after passing through the shared $G$. This step provides the basic supervision information for model optimization. Secondly, to supplement the class information that each source branch does not have, the mixing of true labels and target pseudo-labels is performed between each source pair. Correspondingly, the source and target RSIs are also mixed, and collaborative learning on multiple source branches is carried out according to the mixing results. This step builds on the problem setting in Section III-A, where each source most likely contains class information that the other does not. Thirdly, the predictions from different source branches in the teacher model are used to generate the final target pseudo-labels. In the student model, the module $H$ fuses the deep features from different branches and performs multi-domain knowledge transfer based on the final target pseudo-labels. In the following sections, the above learning process will be described in detail.

Acquiring rich and robust source knowledge is a prerequisite for realizing class asymmetry multi-source domain adaptation. Related researches have shown that simply combining different RSIs into one single source will lead to suboptimal domain adaptation effect due to the domain discrepancies [38, 39, 40]. Therefore, in each iteration, the RSIs from different sources will be fed into the corresponding expert branches, and supervised training will be conducted on the basis of learning common and source-specific features, which can be expressed as:

where $i$ and $j$ index different source domains and RSIs samples respectively in a training batch. Utilizing different expert branches for supervised learning can effectively avoid the interference of domain discrepancies on model training, so as to provide support for the multi-source collaborative learning and knowledge integration in the next steps.

Different source RSIs have different class sets. Therefore, the multi-source supervised learning can only enable each expert branch to learn the knowledge within its corresponding class space. In this case, the domain adaptation cannot be achieved due to the class asymmetry problem between each source-target pair. To this end, a novel cross-domain mixing strategy is proposed to supplement each expert branch with the class information that it does not possess.

Specifically, the information supplementation from source 2 to source 1 is used as an example for detailed explanation. As shown in Fig. 2, the class set of source 1 contains only Building, Low vegetation, and Impervious surface, thus during the initial phase of model training, the source 1 branch can only accurately recognize the three classes in target RSIs. In contrast, the class set of source 2 contains the Tree class that source 1 does not have, and thus the target segmentation results of the source 2 branch will contain high-quality pseudo-labels of the Tree class. Based on these observations, the proposed mixing strategy pastes part of the true label of source 1 onto the target pseudo-label generated by the source 2 branch, and the source 1 RSIs and the target RSIs are also mixed according to the same mask, which actually introduces the unseen class information into the source 1 branch. Obviously, the mixing techniques play a key role in the proposed strategy. In our work, the coarse region-level mixing [46] and fine class-level mixing [47] are adopted simultaneously. Both the two techniques belong to the local replacement approaches, so they can be uniformly formalized as:

where $y^{T}_{S_{2}}$ denotes the target pseudo-labels of the source 2 branch of the EMA teacher model, the symbol $\odot$ represents the element-wise multiplication, and M is a binary mask determining which region of pixels is cut and pasted. When the class-level mixing is performed, the mask $\bf M$ is generated by randomly selecting partial classes in the true source labels, which delivers the inherent properties of a certain class of objects completely. When the region-level mixing is performed, the mask $\bf M$ is obtained by randomly cutting a region patch from the true source labels, which can retain the local structure and context information. Therefore, the application of the two mixing techniques can combine different advantages at the fine class and coarse region levels, effectively improving the performance of the information supplement between different sources. In addition, the diversity of mixed samples is further enhanced, which can help to improve the robustness of the trained model.

After the mixed sample-label pairs $(x_{mix}^{S_{1}},y_{mix}^{S_{1}})$ are obtained, the source 1 branch will carry out the weighted self-supervised learning. Specifically, the confidence-based weight map is first generated as follows:

where $h$ and $w$ denote the height and width of RSIs samples, the operation $[\cdot]$ is the Iverson bracket, and $w_{t}$ represents the pixel percentage in which the maximum softmax probability of prediction class exceeds the threshold $\tau$. Then, the self-supervised loss of source 1 branch can be calculated as:

which actually achieves the domain adaptation from source 1 to the target domain, since the input sample contains partial target RSIs.

At this point, the process of class information supplementation from source 2 to source 1 has been completed. Conversely, the information supplementation from source 1 to source 2 is similar to the above process. To intuitively observe the effectiveness of the proposed cross-domain mixing strategy, the target pseudo-labels generated by different source experts are visualized, as shown in Fig. 3. Obviously, at the initial training stage, each source expert can only segment the target RSIs within its own class space. For example, in the first column of Fig. 3, the source 1 expert could not accurately identify the Car class, while the source 2 expert wrongly classifies the objects of impervious surface and building into the Cluster and Low vegetation classes. With the continuous training, however, the class information among multiple source domains is supplemented with each other, and all the expert branches can identify all source classes more accurately.

When more source RSIs are adopted, the Equation 2 will be extended to more forms including multiple mixed results of pairwise source combinations. Summarily, in the proposed mixing strategy, each source can provide additional class information for other sources, which is actually the collaborative learning process between different experts. As a result, each expert can learn all the class knowledge contained in the source union, thus effectively solving the class asymmetry problem during the domain adaptation process of each source-target pair.

As stated in challenge 2, discrepancies between source RSIs domains can give different expert branches the strengthes of focusing on different feature knowledge. After each source domain is adapted to the target domain separately, the advantages of different source experts should be combined to further improve the performance of multi-domain transfer. Therefore, a multi-source pseudo-label generation strategy is proposed, which can deal with both cases where the source class union is equal to or includes the target class set. And a multiview-enhanced knowledge integration module is developed for target inference through learning the high-level relations between different domains.

By combining the losses of multi-source supervised learning, collaborative learning and multi-domain knowledge transfer, the final optimization objective can be obtained:

where $\mathcal{L}_{sup}$ and $\mathcal{L}_{ssl}$ both contains the sum of the losses of different source branches, and $\alpha$ and $\beta$ denote the weight coefficients. In addition, Algorithm 1 summarizes the pseudo code of the proposed MS-CADA method, to clearly show the entire workflow of class asymmetry RSIs domain adaptation in the two-source scenario.

The four public RSIs datasets, including ISPRS Potsdam (PD), ISPRS Vaihingen (VH), LoveDA [53] and BLU [54], are used for experiments, and the details are listed in Table I.

The VH and PD datasets contain the same six classes: Impervious surface, Building, Low vegetation, Tree, Car and Cluster. Referring to relevant researches, the target VH dataset contains 398 samples for domain adaptation training and 344 samples for evaluation, while the source PD dataset contains 3456 samples for training [21, 55]. The space size of each sample is $512\times 512$. In the two-source union equality scenario of airborne RSIs, the whole PD training set is divided into two groups, i.e., the PD1 subset with 1728 RGB samples and the PD2 subset with 1728 IRRG samples. As listed in Table. II, the first two settings are obtained by discarding part of the classes in PD1 and PD2 respectively. In addition, the class symmetry two-source setting is also used for experiments.

The class space merge is first performed on the LoveDA dataset by referring to [54], so that LoveDA and BLU datasets have the same six classes: Background, Building, Vegetation, Water, Agricultural and Road. The Urban dataset used as the target domain contains 677 training samples and 1156 testing samples, and the first three tiles in the BLU dataset are used for different source domains, each of which contains 196 samples for domain adaptation training. Each sample is cropped to the size of $1024\times 1024$. Similar to the scenario of airborne RSIs, the two three-source union equality scenarios are established, as listed in Table. III.

In addition, the two-source union inclusion scenario is established on the airborne RSIs, as shown in Table. IV. Through discarding the RSIs containing the Clutter class of the VH dataset, the partial VH subset is obtained, where the number of training and testing samples is reduced to 350 and 319 respectively.

All algorithms are developed based on Python 3.8 and relevant machine learning libraries. A computer equipped with an Intel Xeon Gold 6152 CPU and an Nvidia A100 PCIE GPU provides hardware support for programs running.

Referring to most relevant researches, the ResNet-101 network pretrained on ImageNet dataset is used as the shared backbone $G$. Each expert branch actually contains a improved atrous spatial pyramid pooling (ASPP) module as $E$ and a $1\times 1$ convolution classification layer as $F$. The number of output channels of $G$, $E$ and $F$ is 2048, 64 and $N_{\mathcal{C}^{S}}$ respectively, where $N_{\mathcal{C}^{S}}$ is the number of all source classes. The HGCN in the knowledge integration module $H$ contains two hypergraph convolution layers. In the view of space, the number of neighbor vertices used for building hypergraph is 64, and the number of channels of two layers is $64\times k$ and 64 respectively, while in the view of feature, the above values are changed to 8, $N_{h}\times N_{w}\times k$ and $N_{h}\times N_{w}$ respectively. In the multi-source union equality scenario, the liner classifier in $H$ outputs the $N_{\mathcal{C}^{S}}$ classes, while in the multi-source union inclusion scenario, it outputs the $N_{\mathcal{C}^{T}}$ classes.

During the domain adaptation training, the batchsize is set to 4, which means that there are $4\times k$ source samples and $4$ target samples in a training batch. For each source domain, half of the samples are mixed using the coarse region-level strategy, and the other half are mixed using the fine class-level strategy. More specifically, in each source labeled sample, half of the classes or 40% of the region is pasted to the target sample. In the process of multi-domain transfer, the strong transformation of target RSIs containing flipping, rotation, cropping, color jitter and gaussian blur, and only the pixels with the prediction probability larger than 0.968 are used for loss calculation. In addition, the Adam algorithm is used for model optimization, the training iteration is set to 40,000, and the learning rates of the backbone and multi-branch segmentation head are set to $6\times 10^{-5}$ and $6\times 10^{-4}$ respectively. The weight coefficients $\alpha$ and $\beta$ are both set to 1. Other related hyperparameter optimization techniques are consistent with [56].

To verify the effectiveness of the proposed MS-CADA method, four conventional single-source UDA methods including Li’s [57], DAFormer [56], HRDA [58] and PCEL [55], and four multi-source UDA methods including UMMA [41], DCTN [36], He’s [32] and MECKA [40], are used for performance comparison.

The UDA method presented by Li et al. introduces the two strategies of gradual class weights and local dynamic quality into the process of self-supervised learning, which can achieve better performance than most adversarial learning methods. DAFormer is a simple and efficient UDA method, which can improve the target segmentation accuracy to a certain extent, by improving the training strategies of class sampling, feature utilization and learning rate. HRDA is an improvement method over DAFormer, which can effectively combine the advantages of high-resolution fine segmentation and low-resolution long-range context learning. PCEL is an advanced UDA method for RSIs segmentation, which can significantly improve the domain adaptation performance through the enhancement of prototype and context. The above methods take the simple combination of different RSIs domains as the singe source, which can be referred to as "Combined source" for short.

Considering that there is no existing method that can exactly fit the problem setting of class asymmetry domain adaptation segmentation of RSIs, several multi-source UDA methods designed for related problems are modified appropriately and used for comparison. UMMA is an advanced model adaptation method for street scene segmentation, which can achieve the adaptation to target domains using multiple models trained on different sources. DCTN is an adversarial-based UDA method for natural images classification. It can utilize different discriminators to generate the weights of different sources and combine different classifiers to achieve multi-source adaptation. The method proposed by He et al. is actually a multi-source UDA method for street scene segmentation, and MECKA is a multi-source UDA method for RSIs scene classification based on the consistency learning between different domains. Among them, only the methods UMMA and He’s are capable of dealing with the multi-source union equality scenario of RSIs segmentation. Therefore, appropriate modifications are imposed on the methods DCTN and MECKA. Specifically, the classifiers of DCTN is replaced by the ASPP module, and the complete predictions are obtained through casting the results of different classifiers over the target class space and calculating the weighted results. And the cross-domain mixing strategy proposed in Section III-D is integrated into different branches of MECKA. In addition, in the multi-source union inclusion scenario, for the above four methods, the outlier class that target RSIs do not contain will be discarded to re-form the multi-source union equality scenario, which is significantly different from the proposed MS-CADA method. Consequently, the above methods can utilize multiple separated sources for class asymmetry UDA of RSIs, which can be referred to as "Separated multiple sources" for short.

All the above methods utilize the ResNet-101 trained on the ImageNet dataset as the backbone network, and perform domain adaptation training with the random seed of 0. In addition, to fairly compare the performance of different methods, the three widely used measures including IoU per class, mIoU and mF1, are selected for quantitative comparisons.

Tables V-VII list the segmentation results of different methods in the two-source union equality scenario of airborne RSIs, which correspond to the three different class settings in Table II respectively. Observations and analysis can be obtained from the following four aspects.

Firstly, simply combining different RSIs and implementing domain adaptation based on the single-source UDA methods will weaken the segmentation performance of target RSIs. According to the results of DAFormer and PCEL in Table VII, the mIoU based on the combined source decreases to different degrees, compared with the mIoU based on the best single source. This is not difficult to understand that, different RSIs with domain discrepancies interfere with each other, resulting in the suboptimal domain adaptation performance.

Secondly, the performance of the four conventional single-source UDA methods with the combined source is obviously inferior to that of the proposed method. Specifically, in the three different class settings, the mIoU of the proposed method is 2.76%, 3.36% and 3.23% higher than that of the conventional UDA method with the best performance, respectively. Correspondingly, the mF1 value of the proposed method is increased by 2.35%, 3.07% and 2.82% respectively. These observations directly verify the effectiveness of the knowledge integration and transfer with multiple separated sources, which effectively avoids the negative transfer problem in the combined source.

Thirdly, the results of four improved multi-source UDA methods and the proposed MS-CADA method are compared. Specifically, UMMA can only utilize the models pretrained on different sources for domain adaptation without feature alignment between source and target RSIs, resulting in the relatively poor results. DCTN can simply combine the advantages of different classifiers to some extent by weighted calculation, and its performance is better than UMMA. The two methods He’s and MECKA can perform the consistency learning and knowledge supplement between different source domains, and thus their domain adaptation performance is further improved. The proposed method always performs better than the other four methods, wether in the class asymmetry case or the class symmetry case. Compared to the second place, the proposed method improved by 9.23%, 12.80% and 4.58% in mIoU and 8.94%, 11.43% and 4.23% in mF1, in the three different class settings. This demonstrates the superiority of the proposed methods in airborne RSIs domain adaptation segmentation in the two-source union equality scenario.

Fourthly, the performance of the proposed method in the class asymmetry scenario is still better than that of some advanced methods in the class symmetry scenario. For example, the proposed method can achieve the higher mIoU and mF1 in class setting 2 than the other eight methods do in class setting 3, which fully demonstrates the effectiveness of the proposed method in integrating and transferring multi-source knowledge in the case of class asymmetry.

To intuitively compare the segmentation results of different methods, Fig. 6 shows the segmentation maps of several examples in the target RSIs. Specifically, the results of the three representative methods including PCEL, MECKA and MS-CADA in class settings 2 and 3 are visualized, from which the three main points can be drawn. (1) The quality of the segmentation maps of the proposed method is superior to that of PCEL using the combined source. Specifically, the segmentation maps of the proposed method possess the more complete context structure and less noise. (2) Compared with MECKA, the proposed method can produce the more accurate segmentation maps, which is especially obvious for the minority class and small objects. For example, in line 3, the proposed method can recognize the Clutter class accurately, and in line 5, the proposed method can locate the objects of the Car class more precisely. (3) The segmentation maps in the setting 3 of class symmetry are superior to those in the setting 2 of class asymmetry, which can be clearly observed from Fig. 6 (d)-(g). Implementing RSIs domain adaptation with completely consistent class space can provide more class information and labeled samples for finely segmenting objects, so as to produce the segmentation maps with better quality.

The segmentation results of different methods in the three-source union equality scenario of spaceborne RSIs are given in Tables. VIII-IX. Compared with airborne RSIs, the domain adaptation of large-scale spaceborne RSIs is more difficult, especially knowledge integration and transfer using multiple class asymmetry sources. It can be seen that, there is a general decline in the segmentation performance of different methods, however, several observations similar to the airborne RSIs scenario can be obtained.

Firstly, according to the results of PCEL in Table IX, the performance based on the combined source is still inferior to that based on the best single source. Although the expansion of training samples from three sources effectively reduces the decline range, the discrepancies between different RSIs sources still damage the segmentation performance in the target domain to some extent. Secondly, in the two different class settings, the proposed method performs better than the best UDA method PCEL with the combined source, with the increase of 1.38% and 2.11% in mIoU, and 1.46% and 2.30% in mF1. Thirdly, the segmentation results of the proposed method are better than those of other improved multi-source UDA methods. Specifically, the proposed method improves mIoU by at least 1.81% and 3.45%, and mF1 by at least 1.83% and 3.77%. It can be seen from the above statistics that, the proposed method can achieve better results than existing advanced single-source and multi-source UDA methods, whether in the class symmetry case or the class asymmetry case.

Fig. 7 shows the segmentation maps of PCEL, MECKA and MS-CADA. Compared with other methods, the segmentation maps of the proposed method in class setting 2 have the best visual effect and are closest to the ground truth. Specifically, the recognition of the Building objects is more accurate and complete, and the edge of the segmentation results of linear objects is smoother. In addition, under the same class setting, the segmentation maps of the proposed method are better than those of MECKA, which is mainly reflected in the less misclassification and noise. This verifies the advantages of the proposed method in the three-source union equality scenario of spaceborne RSIs from the perspective of visualization.

In addition to the multi-source union equality scenario, this section verifies the effectiveness of the proposed method in the two-source union inclusion scenario, and the results are listed in Table X. It should be pointed out that, for the eight UDA methods for comparison, the pixels corresponding to the Clutter class of the PD2 dataset in Table IV are discarded. In other words, the eight UDA methods actually still perform domain adaptation in the two-source union equality scenario, because they cannot handle the source outlier classes that the target RSIs do not contain.

As listed in Table X, the proposed method achieves the highest mIoU and mF1, which increase by 1.63% and 1.75% respectively compared with PCEL, and 4.22% and 3.61% respectively compared with MECKA. It is believed that, such improvement benefits from the proposed multi-source pseudo-label generation strategy, which can effectively filter out the low-quality pseudo-labels located at the boundary between objects, and improves the performance of knowledge transfer in the self-supervised learning process. In addition to the statistics, the segmentation maps of different methods are visualized, as shown in Fig. 8. Compared with PCEL and MECKA, the proposed method can produce more fine segmentation maps. For example, the segmentation results of small targets of the Car class and the edge of the Building objects are more accurate and detailed.

To intuitively verify the effectiveness of the proposed method in the class asymmetry RSIs domain adaptation and better understand the process of class information supplement and multi-domain knowledge transfer, the visualization analysis is carried out for the deep features generated by different branches at different training iterations. Fig. 9 shows the distribution of features after the t-SNE dimension reduction [59]. Specifically, the same target RSI sample in the class setting 2 of the two-source union equality scenario is used as an example for analysis, and the features used for visualization are all derived from the stage before the classifiers in the source branches or in the module $H$. Next, the detailed comparison and analysis are given from three different domain adaptation stages.

In this section, the influence of three important hyperparameters on segmentation results is analyzed, to explore the sensitivity of the proposed method. Firstly, the optimal combination of class mixing ratio and region mixing ratio in the proposed cross-domain mixing strategy is explored, and the results are shown in Fig. 10. As we can see, when one ratio is too small, increasing the other ratio can lead to a significant improvement. However, when one ratio is large enough, endlessly increasing the other ratio results in a slight decline in performance. The purpose of constructing the mixed samples and labels is to supplement the feature information of other sources to a certain source branch and realize the domain adaptation of each source-target pair. Therefore, the appropriate mixing ratios can better play the advantages of the proposed strategy. Obviously, the best combination is "50% class + 40% region".

The channel dimension of the intermediate features $N_{f}$ determines the richness of information in the process of knowledge transfer from multiple domains to target predictions. Specifically, the value of $N_{f}$ directly affects the structure and size of the constructed hypergraphes. Fig. 11 shows the influence of $N_{f}$ on the segmentation results of target RSIs. On the whole, the curves corresponding to the three different scenarios show a trend of first rising and then stabilizing or even slightly declining. Therefore, considering the performance and cost simultaneously, setting $N_{f}$ to 64 is a better choice for the experimental settings in this paper.

Last but no least, the influence of the weight coefficients of different losses on the domain adaptation performance is analysed. By observing the performance when $\alpha$ and $\beta$ are set to different values, the contribution of different components to the obtained results can be analyzed. When one of $\alpha$ or $\beta$ is set to 0.5, the segmentation results of target RSIs are significantly worse, which indicates that the proposed collaborative learning method and the multi-domain knowledge integration module are very important for the proposed method to achieve excellent performance. When one of A or B is set to 2, the performance is suboptimal, while setting both A and B to 1 can lead to the optimal performance. This indicates that, it is better to treat $\mathcal{L}_{ssl}$ and $\mathcal{L}_{ssl}^{M}$ equally, because they respectively achieve the adaptation of each source-target pair and the knowledge aggregation of multiple domains, which together contribute to the significant improvement in the performance of class asymmetry RSIs domain adaptation with multiple sources.