Unsupervised Adaptive Semantic Segmentation with Local Lipschitz Constraint

An important challenge in adaptive semantic segmentation is domain alignment. Most adaptive semantic segmentation methods introduce adversarial training [12, 24, 25] to align the source and target domain in different levels. Several previous works [14, 15, 16, 17] align domains in pixel level. They utilize generative adversarial networks (GANs) to transfer images in the target domain into the visual style of the source domain so that the model trained in the source domain can be applied to the target domain. Some works [18, 19, 20, 21] align domains in feature level. In detail, they conduct adversarial training in the middle activation of networks to align the feature distributions. Semantic-level alignment are introduced by [12, 24, 25]. They focus on aligning semantic output space of different domains. Besides adversarial training, non-adversarial domain alignment [29] is also introduced to achieve a simple alignment without the vanishing gradient problem.

Instead of domain alignment, some previous works [13, 30, 31, 32, 33] focus on exploring domain-specific knowledge. They always conduct a two-stage self-learning. In the first stage, a model trained on the source domain or trained with a domain alignment method predicts pseudo labels for the target domain. Then, the model is trained in a supervised manner by regarding the pseudo labels as ground truth data in stage two. Because the accuracy of pseudo labels affects the ultimate performance heavily, they typically choose a state-of-the-art domain alignment method in the first stage to achieve strong performance.

The Lipschitz constraint is widely explored in GANs, because it helps avoid the gradient vanishing problem and the loss of gradient informativeness [34, 35, 36]. They propose different tricks into the discriminator and loss function to ensure the Lipschitz continuity of networks. By modifying the Lipschitz constraint, a local Lipschitz constraint [37] is introduced to cope with semi-supervised learning. In UDA, a theoretical analysis derives the error bound of UDA based on the Lipschitz constraint [38]. Inspired by the theoretical derivation, several methods [39, 40, 41] apply the Lipschitz constraint to solve the classification problem. Some of them [37, 39, 40] ensure Lipschitz continuity by adding noises in pixel level, whereas the other [41] adds noises in feature level. As far as we know, although the Lipschitz constraint demonstrates a promising direction to explore non-adversarial UDA, there is no adaptive semantic segmentation method that introduces the Lipschitz constraint. In this work, we first build up an adaptive semantic segmentation with the Lipschitz constraint.

Self-learning, also called co-training or self-training, is widely studied in semi-supervised learning [42, 43] and unsupervised learning [44, 45], with various applications, such as computer vision [46, 47, 48], natural language process [49, 50, 44] and speech understanding [51]. The intuition of self-learning is to let a model learn complementary information from other models. Recently, UDA inherits the intuition of self-learning. Some works [52, 53, 54] use several classifiers to guide each other to cope with UDA classification. Some semantic segmentation UDA works [13, 30, 31, 32, 33] first predict pseudo labels and then use pseudo labels to self-training. A crucial problem for self-learning is the correctness of pseudo labels. When pseudo labels fit the ground truth labels, self-learning enhances the model gradually. Otherwise, such learning makes the model worse and worse.

Given a labeled source domain $D_{S}=\{x_{s}^{i},y_{s}^{i}\}_{i=1}^{M}$ and an unlabeled target domain $D_{T}=\{x_{t}^{j},y_{t}^{j}\}_{j=1}^{N}$, where $x$ denotes the input image and $y$ denotes the output semantic segmentation. All $y_{t}$ in the target domain is inaccessible. We denote $P_{S}$ over $X_{S}$ and $P_{T}$ over $X_{T}$ as the marginal distributions of source and target domain, respectively. An adaptive semantic segmentation method focuses on learning a projection function $h$ that maps the input image $x$ to $y$. The error of $h$ with respect to $P_{T}$ is defined as $E_{P_{T}}(h)=P_{T}(y_{t}\neq h(x_{t}))$. For a practical semantic segmentation adaptation, there exists an optimal function $h^{*}$ and the optimal error is $E^{*}(P_{T})=min_{h}E_{P_{T}}(h)$.

A theoretical derivation [38] proves that the error of UDA is bounded by a probabilistic Lipschitzness with respect to a binary classification setting. The definition of probabilistic Lipschitzness is defined as follows:

Given a labeled sample batch $S$, for any $x\in S$, $l(x)=y$ denotes the label of $x$ and $N_{S}(x)$ denotes the nearest neighbor to $x$ in $S$, i.e., $N_{S}(x)=\operatorname{argmin}_{z\in S_{\chi}}\mu(x,z)$. Define a hypothesis $\mathbbm{H}$ determined by the Nearest Neighbor algorithm as $\mathbbm{H}(x)=l(N_{S}(x))$ for all $x\in\chi$. A error bound of UDA is derived as follows:

where $m$ denotes the size of $S$ that contains points sampled i.i.d. from the input samples according to some distribution $P^{m}$ and $d$ denotes the dimension of input samples. $R$ is a positive constant defined in weight ratio [38]. It is determined by finite samples in the source and target domain.

According to (2), the error bound of UDA includes three crucial factors. The first one is the optimal error $E^{*}(\mathbbm{P_{T}})$. It is impractical to optimize because it is an ideal value. The second term is the probabilistic Lipschitzness $\Phi(\gamma)$. To reach a tight error bound, it is reasonable to satisfy the Lipschitz constraint to minimize $\Phi(\gamma)$. Thus, in this work, we focus on training an adaptive semantic segmentation model that satisfies the Lipschitz constraint. The third term comprises several constants. It is practical to minimize $m$ and $d$ to obtain a tight error bound, i.e., enlarging the number of samples and decreasing the dimension of samples.

An intuitive description of the Lipschitz constraint is shown in Fig. 1. For samples near the decision boundary, if a domain shift happens, they could easily cross the boundary, thus, are classified into a wrong category. These sensitive samples cause the performance drop when a model that is well trained on the source domain applies to the target domain. The Lipschitz constraint alleviates such performance drop as it pushes sensitive samples away from the decision boundary. Specifically, if a sample is near the decision boundary, samples around it could potentially cross the boundary and their outputs would be different. Because the Lipschitz constraint tries to make these samples’ outputs to be the same, these sensitive samples would be pushed away from the boundary to form a robust decision boundary. The robust decision boundary avoids misclassification even a domain shift is introduced.

Although the error bound is derived in a binary classification setting, it indeed inspires us to propose an adaptive semantic segmentation method, because semantic segmentation is also a classification problem that performs in pixel level. In this work, considering the feature that segmentation is a pixel-level problem, we propose a local Lipschitz regularization to satisfy the pixel-level Lipschitz constraint in a neural network. A problem caused by the pixel-level Lipschitz constraint is that a large dimension of image samples leads to a loose error bound as the third term in (2) is large. To resolve the conflict, we design an optimization trick to optimize the network in feature level such that the dimension of input samples is greatly reduced, and satisfy the pixel-level Lipschitz constraint at the same time. Thus, both tight error bound and pixel-level Lipschitz constraint are simultaneously satisfied.

In detail, our method includes two stages, where the first stage takes the charge of aligning different domains and the second stage performs self-learning to further explore domain-specific knowledge. A local Lipschitzness regularization is proposed to satisfy the pixel-level probabilistic Lipschitz constraint. Briefly, a Gaussian noise is added to the middle feature of a network, and we measure the difference between the original semantic output and the one added noises to represent the Lipschitzness of each pixel. To ensure a small dimension of input samples, we add noises to the feature of last convolution layer so that the dimension of input to $f$ in (1) is small. This trick regards the downstream layers after the last convolution layer as $f$ in (1). The goal of optimization is to make the downstream layers satisfy the Lipschitz constraint. To ensure the pixel-level Lipschitz constraint, the difference of output is calculated in the pixel-level semantic output. In stage one, we regard the local Lipschitzness regularization as a loss function to directly enhance the Lipschitzness. In stage two, the local Lipschitzness regularization is utilized to estimate the probability of satisfying Lipschitzness for each pixel. The probability is regarded as a dynamic pixel-wise threshold to guide the self-learning.

In (1), $\Phi(\lambda)$ is the supremum of $P_{T}[\exists y:|f(x_{t})-f(z)|>\lambda\mu(x_{t},z)]$. To estimate $P_{T}[\exists y:|f(x_{t})-f(z)|>\lambda\mu(x_{t},z)]$, for each $x_{t}$, we need to sample a bunch of $z$ and calculate $|f(x_{t})-f(z)|$. In this work, we propose a local Lipschitzness regularization to avoid such burdensome computation:

where $r$ is a noise vector sampled from a Gaussian distribution and $\epsilon$ denotes the maximum norm of $r$. $D$ denotes a divergence metric, such as $L_{2}$ distance and kullback-leibler divergence. $G$ is the feature extractor parameterized by $\theta_{G}$ of a semantic segmentation network, e.g., the four basic blocks of ResNet [55]. $H$ is the semantic head parameterized by $\theta_{H}$ of a semantic segmentation network, e.g., the interpolation layer, the atrous spatial pyramid pooling module and the prediction layer. $H(G(x))$ performs just like $\mathbbm{H}(x)$ in (2).

(3) assumes that every $x$ has a neighborhood $U$ whose norm is constrained by $\epsilon$, and $H$ is optimized to be Lipschitz continuous with respect to $x$ and samples in $U$. Namely, by regarding samples in $U$ as $y$ in (1), $\mathcal{L}_{lip}(x,\theta_{G},\theta_{C})$ is an approximation of the probabilistic Lipschitzness. $\mathcal{L}_{lip}(x,\theta_{G},\theta_{C})$ can be calculated in a point-wise way and the computation complexity is greatly reduced, e.g., for each $x$, we can generate many $x+r$ easily to estimate the Lipschitzness instead of sample $y$ from the target domain. Another point needs to pay attention is that both (1) and (3) are defined in the target domain. According to (1) and (2), even though we further ensure the Lipschitzness in the source domain like [39, 40], it is meaningless for a tight error bound.

In stage one, to align different domains, we combine the local Lipschitzness regularization with a segmentation loss. The overview of the framework in stage one is shown in Fig. 2. The segmentation loss is a common cross-entropy loss for semantic segmentation [1]. It applies for images in the source domain:

where $h$ denotes the height of output segmentation, $w$ denotes the weight of output segmentation and $C$ denotes the number of categories. Thus, the ultimate objective function in stage one is as follows:

where $\lambda_{1}$ is a coefficient to balance the segmentation loss and local Lipschitzness regularization.

The local Lipschitzness regularization focuses on the intra-domain knowledge in the target domain. Intuitively, it makes the classification of each pixel robust enough to avoid an ambiguous output. It is optimized in a non-adversarial training to avoid the vanishing gradient problem. Meanwhile, because the intuition and detailed optimization of it is totally different from adversarial-training based methods [12, 22], it can also collaborate with them to further enhance the UDA performance.

To explore domain-specific knowledge in the target domain, we introduce a dynamic self-learning, which first utilizes a model to predict pseudo labels of the target domain, and then conduct a supervised training with pseudo labels. In stage two, we can use the model trained in stage one to predict pseudo labels. The conventional self-learning loss of each pixel is as follows:

where $\hat{y}_{t}^{h,w,c}$ is the pseudo label of $x_{t}$. $h$ denotes the height of output segmentation, $w$ denotes the weight of output segmentation and $C$ denotes the number of categories.

The most challenge issue in self-learning is the noisy labels in pseudo labels that are different from the ground truth labels. If noisy labels are regarded as the ground truth to perform a supervised learning, the model would become worse and worse. To tackle this problem, we utilize the local Lipschitzness regularization to measure the probability of satisfying Lipschitzness for each pixel and use the probability to dynamically set a threshold for each pixel to determine the pseudo label is whether positive or not.

In detail, we first recall (3) that if a pixel in $x_{t}$ has a small probability to satisfy Lipschitzness, $\mathcal{L}_{lip}$ would be large for it. Considering that Lipschitzness constrains the error bound in (1), we regard pseudo labels of pixels with large $\mathcal{L}_{lip}$ as noisy labels. To avoid the effects of these noisy labels, we can modify the supervised training with pseudo labels by introducing the dynamic thresholds based on $\mathcal{L}_{lip}$:

where $h$ denotes the height of output segmentation, $w$ denotes the weight of output segmentation and $C$ denotes the number of categories. To stabilize the training, we adopt the policy in [56] that replace $\frac{1}{\mathcal{L}_{lip}(x_{t})}$ to $exp(-\mathcal{L}_{lip}(x_{t}))$. Thus, (7) is modified into:

The proposed self-learning dynamically adjusts the threshold for each pixel. When the pseudo label of a pixel has a large probability to be wrong, the threshold would be large, i.e., its loss would be refined to a small value to decrease its impact on $\theta_{G}$ and $\theta_{C}$. Such dynamical threshold is more reasonable than previous manual threshold [13], because in different settings, such as different datasets, different categories and different locations, the threshold should be adaptively chosen. It is burdensome for people to determine the threshold for different settings accurately, whereas our method uses the Lipschitz constraint to dynamically determine it. It is more convenient and has a convincing theoretical support. Another advantage of the proposed self-training is that no extra parameters or modules are added. Different from [57] that introduces the extra branches, and [30] that requires two semantic heads, our method can be easily adapt to any CNN-based semantic segmentation networks without adding extra parameters. This makes it more practical. We verify the model-agnostic property of the proposed dynamic self-learning with an ablation study in Section IV-E.

Another problem in previous self-learning methods [13, 31] is that they ignore the objective function in state one. It is easy to leads the catastrophic forgetting [28] that forgets knowledge learned in stage one if no regularization is introduced. To consolidate knowledge learned in stage one, we add $\mathcal{L}_{lip}$ as regularization to (8) to avoid the catastrophic forgetting:

where $\lambda_{2}$ is a coefficient to balance the self-learning loss and the local Lipschitzness regularization. The overview of the framework in stage two is shown in Fig. 3.

By combining the first and second stages, the whole training procedure is formed. First, we train the model to optimize the objective function in (5) to align different domains with the local Lipschitzness regularization. Because the optimization is non-adversarial, the training procedure gets rid of the vanishing gradient problem [26]. Meanwhile, the local Lipschitz regularization focuses on the intra-domain knowledge and adversarial training tries to learn the inter-domain knowledge, thus, our method can collaborate with adversarial training to further boost the UDA performance. Second, after predicting pseudo labels of the target domain with the model trained in stage one. We further train the model to optimize the objective function in (9) to conduct the dynamic self-learning with pseudo labels.

Another advantage of the proposed method is that it can easily adapt to any CNN-based semantic segmentation backbones and any other adaptive semantic segmentation methods. When calculating $\mathcal{L}_{lip}$, noises are added to the middle feature of a network. This operation has no requirement about the network architecture. Thus, our method can adapt to any CNN-based semantic segmentation backbones. Further, because the optimization in stage two only needs pseudo labels and our method has no requirement about the network architecture, it can adapt to any other adaptive semantic segmentation methods only if they could predict pseudo labels. The whole training procedure is shown in Algorithm 1.

Datasets. We conduct experiments on three standard benchmarks: GTA5 [58], SYNTHIA [59] and Cityscapes [60]. GTA5 is collected from a video game. It contains 24,966 images. SYNTHIA is rendered from a virtual city and is annotated with pixel-level segmentation. It contains 9,400 images. Cityscapes is a realistic dataset, which collects street-view scenes from 50 different cities. It contains 2,975 images for training and 500 images for validation. Considering we want to adapt models trained in a synthetic environment to a realistic one in practice, we form two domain adaptation tasks: GTA5 $\rightarrow$ Cityscapes and SYNTHIA $\rightarrow$ Cityscapes. In detail, the notation A $\rightarrow$ B means that A is the source domain and B is the target domain. Our goal is to adapt the model well-trained on the source domain to the target one without annotations of the target domain.

Metric. Following previous adaptive semantic segmentation methods [13, 30, 31], we report per-category IoU and mean IoU (mIoU) over all categories. For SYNTHIA $\rightarrow$ Cityscapes, because the categories in the source domain are less than these in the target domain, we report the results of 13 categories and 16 categories that are common parts of the source and target domain.

Backbone. Although our method can adapt to any CNN-based semantic segmentation network architectures, to ensure a fair comparison, we deploy the widely-used Deeplab-v2 [2] as the backbone, which is based on ResNet101 [55]. Meanwhile, following most previous adaptive semantic segmentation methods [13, 31, 12], we add an auxiliary semantic head to boost the performance. The auxiliary semantic head has a similar design to the primary one where an atrous spatial pyramid pooling module and a dense layer are included. The auxiliary semantic head is added after res4b22 layer. The primary semantic head is added after res5c layer.

Training Setting. The input image is resized to 1280$\times$640 with scale jittering from $[0.8,1.2]$, and then is randomly cropped to 512$\times$256 for training. Batchsize is set to 2 for the first stage and is changed to 9 for the second stage. Learning rate is set to 0.0002 for stage one and is set to 0.0001 for stage two. In the first stage, $\lambda_{1}$ is set to 0.2 for the auxiliary head and is set to 1 for the primary head. In the second stage, $\lambda_{2}$ is set to 1 for both semantic heads. Similar to [30], we deploy the learning rate policy by multiplying the factor $(1-\frac{iter}{total-iter})$. The total iteration is to 100$k$ steps and the early-stop strategy is used. The training procedure is stopped after 40$k$ steps. As for the divergence metric in (3), we can choose many metrics, such as $L_{2}$ distance and $L_{1}$ distance. Generally, we use kullback-leibler divergence. Thus, (3) is detailed as:

When multiple semantic heads are adopted in the network architecture, we follow the ensemble trick in [30] that combines the output of all semantic output as the final inference output. Specifically, if two semantic heads are used, the final output equals the output of the primary head plus the output of the auxiliary head multiplying 0.5.

Models. To show the adaptability of the proposed method, besides implementing LCDA and LCRF, we adapt LCDA and LCRF with several adaptive semantic segmentation methods. In detail, to show that the local Lipschitzness regularization can cooperated with adversarial learning, we implement a model named LCDA+Adv that adds the objective function in AdaptSeg [12] to $\mathcal{L}_{1}$. Further, to show the superiority of LCRF compared with the manual threshold for pseudo-label learning, we implement a model named LCDA+Adv+Pseudo that utilizes a manual threshold [13] to conduct pseudo-label learning over LCDA+Adv. Instead, LCDA+Adv+LCRF uses the proposed LCRF to refine the results of LCDA+Adv. To verify that LCRF generalizes well with other adaptive semantic segmentation models, we adapt it with MRNET [61], denoted as MRNET+LCRF.

GTA5 $\rightarrow$ Cityscapes. Results of GTA5 $\rightarrow$ Cityscapes experiment are shown in Table I. We compare the proposed method with other recent adaptive semantic segmentation methods. Results of all models are based on the same backbone, i.e., DeepLabv2 [2]. These methods can be divided into two categories, i.e., domain-alignment-based and pseudo-label-based methods. In detail, the former includes AdaptSeg [12], SIBAN [21], CLAN [62], APODA [63], PatchAlign [22], AdvEnt [22], and MRNET [61]. The latter includes CBST [13], LRENT [31], MRL2 [31], MRENT [31], MRKLD [31], MRNET+Pseudo [61] and RPLUE [30].

Overall, MRNET+LCRF and RPLUE are the top two methods on the task. MRNET+LCRF achieves the result 49.9% mIoU that ranks second among all methods. LCDA obtains 44.0% mIoU that improves the method without adaptation by 6.7% mIoU. The obvious improvement verifies the effectiveness of the proposed local Lipschitzness regularization. After adding adversarial training into LCDA, the performance of LCDA+Adv increases to 45.2%. The improvement shows that the Local Lipschitzness regularization, which is a non-adversarial domain alignment, further enhances adversarial training. Self-learning-based methods also refine the results of domain-alignment-based methods. Specifically, LCDA+Adv+Pseudo, which adopts pseudo-label learning with a manual threshold, improves the result of LCDA+Adv by 2.9% mIoU. LCRF performs better than Pseudo. It achieves 3.9% mIoU improvement over LCDA+Adv. The results on GTA5 $\rightarrow$ Cityscapes show that LCDA and LCRF can effectively transfer the knowledge of the source domain to the target one.

SYNTHIA $\rightarrow$ Cityscapes. Results are shown in Table II. We compare the proposed method with other recent adaptive semantic segmentation methods. For a fair comparison, we report all results based on the same backbone, i.e., DeepLabv2 [2]. These methods can be divided into two categories, i.e., domain-alignment-based and pseudo-label-based methods. In detail, the former includes AdaptSeg [12], SIBAN [21], CLAN [62], APODA [63], PatchAlign [22], AdvEnt [22] CCM [64] and MRNET [61]. The latter includes CBST [13], LRENT [31], MRL2 [31], MRENT [31], MRKLD [31], MRNET+Pseudo [61] and RPLUE [30]. The mean IoU of 16 categories is denoted as mIoU while the mean IoU of 13 categories is denoted as mIoU^∗.

Overall, MRNET+LCRF performs the best on the task. MRNET+LCRF arrives at the state-of-the-art result 48.2% mIoU^∗ among 16 categories that surpasses other methods. It also achieves 54.9% mIoU among 13 categories that ranks the first. LCDA achieves 40.9% mIoU and 47.4% mIoU^∗ that gains 5.7% mIoU and 7.0% mIoU^∗ improvement over the model without adaptation. It verifies the effectiveness of the proposed local Lipschitzness regularization. By adding adversarial training to LCDA, LCDA+Adv gains 1.6% mIoU and 1.8% mIoU^∗ improvement over LCDA. Such improvement indicates that the local Lipschitzness regularization can cooperate with adversarial training and further boost the performance of LCDA. Comparing with LCDA+Adv, LCDA+Adv+Pseudo that adopts pseudo-label learning with a manual threshold brings 3.8% mIoU and 3.8% mIoU^∗ improvement. The dynamic self-learning, i.e., LCRF, gains 4.4% mIoU and 4.8% mIoU^∗ improvement over LCDA+Adv, which performs better than the pseudo-label learning with a manual threshold. The comparison between LCDA+Adv+Pseudo and LCDA+Adv+LCRF shows the superiority of our proposed dynamic self-training. Besides, MRNET+LCRF yields the competitive performance in terms of the per-category IoU. It ranks the first in 6 categories. In several categories that are hard to adapt, such as wall and traffic Light, it performs much better than other methods.

To give more qualitative results of the proposed method, we provide several visualizations: (1) The semantic output of different models. (2) The confidence weight of different models. The confidence weight indicates the probability of the predictive category for each pixel. (3) The variance weight of different models. The variance weight is the probability that the predictive category after adding noises is different from the original predictive category.

Semantic Output. Visualization of semantic output is shown in Fig. 4. It is clear that by cooperating with adversarial training, the local Lipschitzness regularization further enhances its performance. If a self-learning is introduced in the second stage, the results of an adaptive semantic segmentation model would be refined a lot. In detail, LCRF performs slightly better than Pseudo. MRNET+LCRF performs best, which avoids the error on the categories that cover a large area, such as road, sidewalk and sky. It also contains more details with a small area than other methods, such as pole and traffic sign. It examines that a better model in stage one promotes the performance of LCRF.

Confidence Weight. Visualization of confidence weights is shown in Fig. 5. The confidence weight of all methods focuses on the boundary of different regions, which shows that they distinguish different categories with high confidence. An interesting discovery is that the highlighted lines of the best model, i.e., MRNET+LCRF, are thin. It means that MRNET+LCRF only gives high confidence on the edge of different categories. This phenomenon indicates that except for the edges, a good adaptive semantic segmentation method should give low confidence in other areas to avoid ambiguous predictions. However, comparing LCDA+Adv+Pseudo with LCDA+Adv+LCRF, it is clear that pseudo-label learning with manual threshold causes more ambiguous predictions. The difference verifies the robustness of the proposed dynamic self-learning.

Variance Weight. Visualization of variance weights is shown in Fig. 6. We randomly sample several variance weights of images in the Cityscapes’s validation set. Large variance weights usually appear in small-scale regions. When noises are added to these regions, they are highly potential to be classified into wrong categories. These regions belong to rare categories such as traffic light and pole. Because annotations of these categories are scarce and large-scale regions around them tend to cover them, an adaptive semantic segmentation method is hard to generalize well on them. To solve the problem, in the first stage, the proposed local Lipschitz regularization detects such variance and tries to reduce it. In the second stage, the dynamic self-learning assigns more weights to these rare categories to ensure they would not be ignored.

Dynamic Self-learning v.s. Manual Threshold. To give a quantitative comparison between the proposed dynamic self-learning and pseudo-label learning with a manual threshold, we implement both of them based on LCDA+Adv and MRNET. The backbone, training setting and metric follow the description in IV-B. Results of mIoU on GTA5 $\rightarrow$ Cityscapes and mIoU* on SYNTHIA $\rightarrow$ Cityscapes are shown in Table III.

In both GTA5 $\rightarrow$ Cityscapes and SYNTHIA $\rightarrow$ Cityscapes, LCRF performs better than pseudo-label learning with a manual threshold. LCRF gains approximate 1% mIoU improvement in all settings. The results verify the advantage of the proposed dynamic self-learning.

Architecture Agnostic. As shown in Table I and II, RPLUE [30] achieves comparable performance with MRNET+LCRF. RPLUE also deploys a dynamic self-training based on uncertainty estimation to avoid noisy pseudo labels. However, a limitation of it is that RPLUE requires two semantic heads to estimate uncertainty, which heavily decreases its application scope. For example, if a source-only model only includes a single semantic head, RPLUE cannot apply to it. On the contrary, our method has no requirement of network architecture such that it can apply to any semantic segmentation method. To verify the architecture agnostic, we implement LCDA, LCDA+Adv and LCDA+Adv+LCRF based on a single-head Deeplab-v2. It also adopts ResNet101 as the backbone, however, the auxiliary semantic head is not included. Both $\lambda_{1}$ and $\lambda_{2}$ are set to 1. Other settings follow the training settings in IV-B. All models are tested on GTA5 $\rightarrow$ Cityscapes.

Results of single-head models are shown in Table IV. Compared with the model without adaptation, LCDA achieves 43.2% mIoU, which gains 6.7% mIoU improvement. By adding adversarial training, LCDA+Adv gains 0.3% mIoU improvement over LCDA. LCDA+Adv+LCRF arrives at the best performance, i.e., 44.6% mIoU. The results demonstrate that LCDA and LCRF perform well on single-head models. Such architecture-agnostic property makes our method practical for more application scopes.

Adaptability. Another advantage of our method is that LCDA and LCRF can cooperate with other adaptive semantic segmentation methods to further boost the performance. To verify the adaptability of LCDA, we compare AdaptSeg [12] with AdaptSeg+LCDA that adds (3) to its objective function. Similarly, to show the adaptability of LCRF, we compare MRNET [61] with MRNET+LCRF that adopts the proposed dynamic self-learning to MRNET. The results of mIoU on GTA5 $\rightarrow$ Cityscapes and mIoU* on SYNTHIA $\rightarrow$ Cityscapes are recorded.

As the results shown in Table V, LCDA helps AdaptSeg gain 2.8% mIoU improvement on GTA5 $\rightarrow$ Cityscapes and 2.5% mIoU improvement on SYNTHIA $\rightarrow$ Cityscapes. LCDA boosts the performance of AdaptSeg stably on both tasks. The results shown in Table VI verifies the adaptability of LCRF. Specifically, compared with MRNET, MRNET+LCRF gains 4.4% mIoU improvement on GTA5 $\rightarrow$ Cityscapes and 4.7% mIoU improvement on SYNTHIA $\rightarrow$ Cityscapes. The adaptability of LCDA and LCRF enables them to be a flexible basic component for adaptive semantic segmentation. Other adaptive semantic segmentation methods can adopt them to arrive at better performance.

Effectiveness of Each Module. In this section, we verify the effectiveness of each module in our method. Specifically, we test three modules, i.e., adversarial training [12], LCDA and LCRF. Results are shown in Table VII. On both tasks, adversarial training and LCDA obtain obvious improvement over the model without adaptation. LCDA performs better than adversarial training on both tasks. If LCDA cooperates with adversarial training, the performance could be further improved. Such improvement verifies that the intra-domain knowledge explored by LCDA could provide additional information for the inter-domain alignment, e.g. adversarial training. LCRF obtains 3.9% and 4.8% mIoU improvement over LCDA+Adv on two tasks, respectively. The results verify the effectiveness of our proposed dynamic self-learning.