Pred&Guide: Labeled Target Class Prediction for Guiding Semi-Supervised Domain Adaptation

In real world, the training data distribution is often different from the testing data distribution, necessitating that deep learning models learnt using data from one domain are adapted to data from a different distribution. Unsupervised Domain Adaptation (UDA) [1, 2, 3, 4] addresses this problem by utilizing a related label rich source domain and unlabeled samples from the target domain. Recent research has shown that only unlabeled target samples may not be sufficient to counter the domain shift, and thus the area of Semi-supervised Domain Adaptation (SSDA) [5, 6, 7, 8, 9, 10] is gaining traction, where additionally, a few labeled targets are utilized to aid the adaptation process.

The advantage of SSDA over UDA approaches lies in the manner in which the few labeled targets are utilized in the adaptation process. A straightforward approach is to include them along with the labeled source examples in the Cross-Entropy loss  [5, 6]. Though this approach helps in improving the final performance, it does not fully leverage the usefulness of the labeled target samples, since they form a very small part of the final loss as compared to the large number of source examples.

In this work, we propose a novel framework Pred&Guide, which utilizes the class label prediction inconsistencies on the few target labeled examples to effectively guide the domain adaptation process. Pred&Guide has three main stages: (1) UDA with Self-Training: Access to a few labeled target data can create bias in the domain adaptation process from the very beginning towards those samples as also noted in [3, 11]. Thus, we propose to utilize an UDA framework to start the domain adaptation process in an unbiased manner, which utilizes all the target data, but ignores the available labels of the few target examples. Simultaneously, we also compute the pseudo-labels of all the unlabeled data using the current model, and perform strong augmentations of those target examples, for which we are confident about their pseudo-labels  [12, 13, 13]. (2) Labeled Target Prediction: The model trained in the first stage is used to predict the labels of the targets (which have ground truth labels available) to analyze the adaptation process, and further guide it. (3) Source Example Weighting: We surmise that different source examples either help or hinder the process of domain adaptation. Based on the inconsistency of the predicted and true labels of the few labeled target examples, Pred&Guide steers the adaptation process by appropriately weighing the source samples according to their cosine similarity to the labeled target examples of the corresponding class. Specifically, the source samples in the neighborhood of the incorrectly classified targets are given more weight and the ones which are far away are down-weighed, following a linear weighing scheme as described in Section 4.3, so that the classifier can eventually classify the labeled targets correctly.

We extensively evaluate the proposed Pred&Guide framework on two large-scale benchmark datasets, namely Office-Home [14] and DomainNet [15]. Pred&Guide outperforms all the state-of-the-art approaches for both the datasets. The main contributions of our work are as follows:

1. 1.

   We propose a novel approach for semi-supervised domain adaptation, termed Pred&Guide, by leveraging the prediction inconsistency of the labeled targets.

2. 2.

   We propose to effectively weigh the source examples based on the label prediction inconsistency to guide the DA process.

3. 3.

   Extensive experiments on two benchmark datasets show the effectiveness of the proposed framework.

We now describe the related work in literature followed by the proposed approach and results of extensive evaluation.

Here, we provide pointers to the related work in literature on domain adaptation, data augmentation and semi-supervised learning.
Domain Adaptation: Most of the current literature in domain adaptation can be broadly categorized into unsupervised (UDA) and semi-supervised domain adaptation (SSDA). UDA approaches, which use only unlabeled target domain data have been explored rigorously in [1, 2, 11, 16, 17]. Initial approaches used statistical methods such as Maximum Mean Discrepancy [18], Correlation Alignment (CORAL) [19] and Geodesic Flow Kernel [4] to handle the mismatch in feature distribution. More recently, DL based UDA approaches such as [2, 1] aim to learn a domain invariant features by a domain confusion module for feature alignment. It has been observed that UDA performance can be boosted significantly with just a few labeled examples from the target domain, as noted in works such as [5, 8, 9, 20]. Most of the recent SSDA approaches use $1-3$ labeled target examples per class. The seminal work in [5] uses a mini-max entropy to compute domain-invariant class-representatives. In [21] authors propose an adversarial loss instead of min-max entropy. In [9], adversarial examples are used to bridge the source and target domain gap.  [20, 22] use a style-agnostic network and contrastive learning for reducing the source and target domain gaps.

However, majority of approaches treat examples of labeled target no differently than the labeled source, thus not fully leveraging information contained in them. Some works have attempted to use labeled target in triplet loss for bringing source closed to target [23] and [24] propose an effective pseudo labeling strategy using labeled target. Presence of few labeled target examples in SSDA setting results in intra-domain discrepancy, which is recently noted and addressed in [8]. We also address this critical issue Pred&Guide.
 
Data Augmentation: Data augmentation [25, 26, 27] is widely used as a regularizer to prevent overfitting. Different data augmentation policies have been proposed in works such as  [26, 25, 12]. From a pool of augmentations,  [25] randomly selects augmentation to be used.  [26] selects an augmentation during model training using reinforcement learning. Other approaches such as CT-Augment have been proposed in [12, 28] which uses a similar strategy as AutoAugment [26], but uses a fixed algorithm for assigning augmentation magnitudes. In our work, we use an augmentation based consistency-regularization, based on RandAugment [25] to assign pseudo-labels to the unlabeled targets to boost the performance.
 
Semi-Supervised Learning (SSL): SSL approaches utilize a small amount of labeled data and a huge amount of unlabeled data for the required task [29, 30, 31, 12, 32, 33, 13, 28], which reduces the labeling cost. Several SSL algorithms use consistency constraint, i.e. they aim to minimize the entropy between prediction of different versions of a data-point [12, 13, 13, 33].

Mean-Techer [33] trains a student and a teacher model and forces their predictions to match. The student model is updated at every iteration, while the teacher model is updated with a suitable momentum. Virtual Adversarial Training [32] uses adversarial perturbations to train the SSL model. Pseudo-labeling [31] predicts labels of the unlabeled data and uses these as true labels for supervised learning. Recent approaches such as Mixmatch [13], ReMixmatch [28] and Fixmatch [12] provide an elegant framework for SSL, leveraging consistency regularization and MixUp  [34] to obtain state-of-the-art results for SSL. The proposed SSDA framework Pred&Guide is inspired by these seminal SSL approaches, but utilizes the few labeled target domain examples in a novel manner to guide the adaptation process.

We first discuss the problem statement and the notations used. In this work, we address the SSDA task, where there are very few labeled target examples available (i.e. one and three shot settings). The training data consists of (1) labeled source: $\mathcal{D}_{ls}=\{x^{s}_{i},y^{s}_{i}\}_{i=1}^{i=n_{s}}$, where $n_{s}$ is the number of labeled source examples, (2) unlabeled target: $\mathcal{D}_{ut}=\{x^{ut}_{i}\}_{i=1}^{i=n_{ut}}$, where $n_{ut}$ is the number of unlabeled target examples and (3) few labeled target: $\mathcal{D}_{lt}=\{x^{t}_{i},y^{t}_{i}\}_{i=1}^{i=n_{t}}$, where $n_{t}$ is the number of labeled target examples. Here $x_{i}$ denotes the $i^{th}$ data-point and $y_{i}$ denotes the corresponding class label, which belongs to one of the $K$ classes (same for both source and target domains). We test the model’s performance on $\mathcal{D}_{ut}$. For all the experiments, we have a feature extractor network $F$ (with parameters $\Theta_{F}$), sequentially followed by a classifier $C$ (with parameters $\Theta_{C}$).

The proposed SSDA framework has three sequential modules, namely (1) unsupervised domain adaptation (UDA) with self-training (2) labeled target prediction, where we predict the labels of the target domain data using the current model, for which we have the ground truth labels and finally (3) source sample weighting, where the source examples are effectively weighted using the prediction inconsistencies of the labeled target examples in the previous step. Figure 1 illustrates the proposed Pred&Guide framework and its different modules. Each of the modules with its motivation are explained below in detail.

Here, we provide details of the extensive experiments conducted to evaluate the efficacy of Pred&Guide for semi-supervised domain adaptation. First, we discuss the benchmark datasets used, the implementation details before reporting the results. We also conduct detailed analysis of Pred&Guide and perform ablation studies to evaluate the usefulness of the individual modules.

Datasets Used and Evaluation Protocol: To evaluate Pred&Guide, we use two large-scale benchmark domain adaptation datasets, DomainNet [15] and Office-Home [14]. DomainNet consists of $6$ domains with $345$ classes, comprising of about $0.6M$ images. Since the entire DomainNet dataset is noisy, we select $4$ domains (Real, Painting, Clipart, Sketch) and $126$ classes with $7$ standard domain adaptation scenarios, which are generally used to benchmark DA methods  [5, 9, 8, 6]. In addition to DomainNet, we also evaluate Pred&Guide on the Office-Home dataset, which is a smaller, but challenging dataset. It consists of $65$ classes and $4$ domains (Real, Clipart, Product, Art). We evaluate Pred&Guide on all the $12$ possible adaptation scenarios for Office-Home.

Implementation Details: The code is implemented using PyTorch  [36] on a single Nvidia RTX-2080 GPU. The underlying domain adaptation technique that we used in Pred&Guide is the very successful mini-max entropy approach [5]. We use Resnet34 [37] and AlexNet [38] for the feature extractor network $F$ to compare with state-of-the-art approaches. For the classifier $C$, we use a $4096\times K$ fully-connected layer for AlexNet and a $512\times K$ fully-connected layer for ResNet34. We use SGD optimizer, with the starting learning rate of $0.01$, momentum of $0.9$, weight decay of $0.0005$ and $\lambda=0.1$ for the weight of $\mathcal{L}_{ULT}$. We use batch size of $24$ and $32$ for Resnet34 and Alexnet backbones respectively on labeled samples, and twice the batch size for unlabeled data. We set $\phi=0.5$ in Equation (9) for DomainNet and $\phi=0.1$ for Office-Home, and $m_{s}=0.1$ in Equation 8 for all our experiments. We run the first module, i.e. UDA with self-training for $T_{1}$ iterations. $T_{1}$ is the iteration-step when UDA with self-training has converged. Then, the weights of the source examples are computed every $T_{n}$ iterations. $T_{n}=1000$ for DomainNet and $T_{n}=140$ for Office-Home. Finally, we bring in the labeled target examples in the CE loss at the $T_{2}$ iteration. $T_{2}$ is started when the source example weighing performance has converged. To calculate $T_{1}$ and $T_{2}$, we consider the validation accuracy to be converged when it has not increased for $500$ iterations.

Evaluation for SSDA: The results of Pred&Guide for Office-Home dataset with AlexNet backbone for both $1$-shot and $3$-shot scenarios are reported in Table I. Comparison with the recent approaches show that Pred&Guide achieves the state-of-art results on an average for both the settings and also outperforms the others for most of the individual domain pair settings. The results on DomainNet dataset for both $1$ and $3$ shots (i.e. one and three target samples of each class in the target domain are labeled) using AlexNet and ResNet34 backbones are reported in Table II. The results of all the other approaches has been directly taken from  [5]. We observe that the proposed approach significantly outperforms all the recent SSDA approaches, thus justifying its usefulness.

Here, we perform extensive analysis and ablation studies of the proposed Pred&Guide framework.
 
Importance of Source Example Weighing: To analyze the importance of weighting the source examples in the proposed Pred&Guide framework, we run the algorithm with and without the source example weighing on the DomainNet dataset. The results in Table IV show that when the algorithm is trained using the standard cross entropy loss $\mathcal{L}_{s}$ without the weighting (i.e. No weights), the performance is considerably lower as compared to when the source samples are properly weighted. Figure 3 gives a visual illustration of the near and far source examples of the labeled targets for one class of DomainNet. We would like to highlight that the proposed source example weighting (SEW) can be seamlessly integrated with other SSDA methods to improve their performance. We experiment with a recent SSDA approach CDAC [41], and observe that SEW improves the mean performance ($\%$) of CDAC for both 1-shot and 3-shot settings from 73.6 to 74.4 and from 76.0 to 76.2 respectively.
 
Weighting Scheme: Now, we analyze the effectiveness of the proposed weighting strategy based on class-wise adaptability. Specifically, we compare with fixed up-weighing and down-weighting. For this experiment, we used fixed weights of $1.5$ and $0.5$ for the near and far samples respectively. We observe from Table III that fixed weighting can give unpredictable results, i.e. for some cases (1-shot, R to C), it improves the results, and for other cases (3-shot R to S and R to C), the performance degrades. We observe that for almost all scenarios, the proposed source example weighting helps to boost the performance. We also compare with another popular weighting scheme, namely Focal Loss [42], which emphasizes on correct classification of hard training examples by weighing them more. We observe that for all scenarios, the proposed technique outperforms focal loss. As an example, for 3-shot setting, the proposed technique gives 72.5$\%$ and 79.2$\%$ for R to S and R to C as compared to 69.6$\%$ and 74.8$\%$ obtained using Focal loss.
 

Near & Far Weighing: Next, we analyze the effect of weighing only the near or far samples, and the results are reported in Table IV. Near source samples are defined as the samples having weight $w_{i}>1$ and far samples are defined as the ones having $w_{i}<1$. The results obtained using only UDA with self-training is shown in the first row. If we introduce the labeled target samples right after this stage, as expected, the performance improves as shown in the second row. The third and fourth rows show the performance when only near and far away source examples are weighted. We observe that both these steps help to boost the performance. The final row is the complete Pred&Guide, which weighs both the near and far away source examples and outperforms all the other variants.
 
Sensitivity Analysis of Hyperparameters: There are two primary hyperparamters in Pred&Guide, $\phi$ which control the amount of weighing for the source examples and $T_{n}$. We plot the SSDA results (%) for a wide range of $\phi$ values for the 3-shot R to S setting of DomainNet in Fig. 4 (using Resnet34). We observe that the performance is quite stable for a wide range of parameter values, and even the lowest accuracy is better than the second best approach. We also vary the values of $T_{n}$ between 1000 to 2000 for DomainNet and observe that the performance is stable for all the $T_{n}$ values. Note that we have used the same $\phi$ and $T_{n}$ values for all the domain settings of a dataset.

In this work, we addressed the importance of using labeled target samples effectively for the domain adaptation task in the semi-supervised setting. The proposed Pred&Guide framework initially performs the domain adaptation in an unsupervised manner to avoid any bias that may be introduced due to the few labeled target examples. After this stage, we introduce two effective modules, labeled target prediction and source example weighting to effectively weigh the source examples to better guide the adaptation. With Pred&Guide, we set a new state-of-the-art for semi-supervised domain adaptation as illustrated by extensive experiments on two large-scale benchmark datasets.