Weak Adaptation Learning:
Addressing Cross-domain Data Insufficiency with Weak Annotator

Weakly Supervised Learning is a large concept that may have multiple problem settings [65]. The problem we consider in this paper is related to the incomplete supervision setting that is often addressed by Semi-Supervised Learning (SSL) approaches. Standard SSL solves the problem of training a model with a few labeled data and a large amount of unlabeled data. Some of the widely-applied methods [46, 9, 43, 2] assign pseudo labels to unlabeled samples and then perform supervised learning. And there are works that address the noises in the labels of those samples [37, 11, 29]. Our target problem is related to SSL with inaccurate supervision, but is different since we consider the feature discrepancy between the (unlabeled) source data and the (labeled) target data – a case that occurs often in practice but has not been sufficiently addressed.

Positive-unlabeled Learning (PuL) is usually regarded as a sub-problem of SSL. Its goal is to learn a binary classifier to distinguish positive and negative samples from a large amount of unlabeled data and a few positive samples. Several works [23, 5] can achieve great performance on selecting samples that are similar to the positive data, and there are also works using samples selected by PuL to perform other tasks [62, 30].

The training of machine learning models, especially deep neural networks, often requires a large amount of data samples. However, in many practical scenarios, there is not sufficient training data to feed the learning process, degrading the model performance sharply [52, 19, 61]. Many approaches have been proposed to make up for the lack of training samples, e.g., data re-sampling [56], data augmentation [44], metric learning and meta learning [3, 4, 54, 57]. And there are works [39, 1, 3, 58] conducting theoretical analysis on the relation between training data quantity and model performance. These analyses are usually in the form of bounding the prediction error of the models and provide valuable information on how the sample quantity of training data affects the model performance. In our work, we also perform a theoretical analysis on the error bound of the trained model, with respect to not only the data quantity but also the performance of the weak annotator.

We consider the task of classification, where the goal is to predict labels for samples in the target domain. Two types of supporting data can be accessed for training the model – source domain data and target domain data. The source domain data samples are initially unlabeled and come from a joint probability distribution $\mathbb{Q}^{s}$. They can be labeled by a weak annotator $\mathbf{h}^{w}$ (which may be inaccurate) and denoted as $D_{s}=\{(\mathbf{x}_{s},y_{s})_{i}\}_{i=1}^{N_{s}}$, where $N_{s}$ is the number of source data samples. The target domain data $D_{t}=\{(\mathbf{x}_{t},y_{t})_{i}\}_{i=1}^{N_{t}}$ consists of $N_{t}$ samples collected from the target distribution $\mathbb{Q}^{t}$. Note that $\mathbb{Q}^{t}$ may be different from $\mathbb{Q}^{S}$. And we use $\mathbb{Q}^{s}_{X}$, $\mathbb{Q}^{s}_{Y}$ and $\mathbb{Q}^{t}_{X}$, $\mathbb{Q}^{t}_{Y}$ to represent the marginal distributions of the source and target domains, respectively. Moreover, as stated before, we consider the case where there is only a small amount of target domain data, i.e., $N_{t}\ll N_{s}$.

Our goal is to learn an accurate classifier for the target domain. The classifier is initialized from a parameter distribution $\mathcal{H}$, which denotes the hypothesis parameter space of all possible classifiers.

In the following analysis, we will define the classification risk of a classifier and then derive its bound. According to the PAC-Bayesian framework [36, 10], the expected classification risk of a classifier drawn from a distribution $\mathcal{Q}$ that depends on the training data can be strictly bounded. Let $\mathbf{h}_{\Theta}$ denote a learned classifier from the training data, and its parameter $\Theta$ is drawn from $\mathcal{Q}$. We consider that the prior parameter distribution $\mathcal{H}$ over the hypothesis is independent of the training data. And given a $\delta$ with the probability $\geq 1-\delta$ over the training data set of size $m$, the expected error of $\mathbf{h}_{\Theta}$ can be bounded as follows [35]:

Here $L(\mathbf{h}_{\Theta})$ is the expected error of $\mathbf{h}$ over parameter $\Theta$, and $\widehat{L}(\mathbf{h}_{\Theta})$ is the empirical error computed from the training set ($\widehat{L}(\mathbf{h}_{\Theta})=\frac{1}{m}\sum_{i=1}^{m}\mathcal{L}(\mathbf{x}_{i},y_{i})$, where $\mathcal{L}$ denotes the loss of a single training sample). In Eq. (1), $K\!L(\mathcal{Q}\|\mathcal{H})$ represents the Kullback-Leibler (KL) divergence between parameter distribution $\mathcal{Q}$ and $\mathcal{H}$. For any two distributions $p,\,q$, the specific form of their KL divergence is $K\!L(p\|q)=-\mathbb{E}[p\cdot\ln{\frac{q}{p}}]$. In most cases of mini-batch training, the training loss $\widehat{L}(\mathbf{h}_{\Theta})$ is much smaller than $\Omega$, and thus we can get a further bound as follows [39]:

Then if we denote the model parameters of $\mathbf{h}$ before the training that are drawn from $\mathcal{H}$ as $\Theta^{\mathbf{p}}$, the KL divergence can be written as $K\!L(\mathcal{Q}\|\mathcal{H})=-\mathbb{E}[\Theta\cdot(\ln{\Theta^{\mathbf{p}}}-\ln{\Theta})]$. As aforementioned, $\mathbf{h}_{\Theta}$ is trained with the training data set from $\mathbf{h}_{\Theta^{\mathbf{p}}}$, and we consider that the training is optimized by gradient-based method. Thus, we can formulate that $\Theta=\Theta^{\mathbf{p}}+\nabla(\widehat{L}(\mathbf{h}_{\Theta^{\mathbf{p}}}))$. Here we omit the learning rate to simplify the formula.

The PAC-Bayesian error bound is valid for any parameter distribution $\mathcal{H}$ that is independent of the training data, and any method of optimizing $\Theta^{\mathbf{p}}$ dependent on the training set [39]. Therefore, in order to simplify the problem, we instantiate the bound as setting $\mathcal{H}$ to conform to a Gaussian distribution with zero mean ($\mu_{\mathcal{H}}=0$) and $\text{Var}_{\mathcal{H}}=\sigma^{2}_{\mathcal{H}}$ variance. This simplification is the same as previous PAC-Bayesian works [39, 40]. We further assume that the parameter change of the overall model during training can also be regarded as conforming to an empirical Gaussian distribution. This Gaussian distribution is independent of model parameters if we regard the parameter updates induced by gradient back-propagation as accumulated random perturbations, i.e., each training sample corresponds to a small perturbation [40]. And we denote the mean and the variance of a single training sample as follows:

Then, the specific formula of KL divergence to any two Gaussian distributions $p\sim\mathcal{N}(\mu_{1},\sigma_{1}^{2})$, $q\sim\mathcal{N}(\mu_{2},\sigma_{2}^{2})$ is written as follows:

For further analysis, we also define two operators in a parameter distribution $\mathcal{H}$:

• $\bullet$

  $\oplus$: $\forall\mathbf{h_{1}},\mathbf{h_{2}}\in\mathcal{H}$, and $\forall\mathbf{x}\in\mathbb{P}$, a new classifier $\mathbf{h_{3}}=\mathbf{h_{1}}\oplus\mathbf{h_{2}}$ can be acquired by conducting operator $\oplus$ on $\mathbf{h_{1}}$ and $\mathbf{h_{2}}$, and $\mathbf{h_{3}}(\mathbf{x})=\mathbf{h_{1}}(\mathbf{x})+\mathbf{h_{2}}(\mathbf{x})$.

• $\bullet$

  $\ominus$: $\forall\mathbf{h_{1}},\mathbf{h_{2}}\in\mathcal{H}$, and $\forall\mathbf{x}\in\mathbb{P}$, a new classifier $\mathbf{h_{3}}=\mathbf{h_{1}}\ominus\mathbf{h_{2}}$ can be acquired by conducting operator $\ominus$ on $\mathbf{h_{1}}$ and $\mathbf{h_{2}}$, and $\mathbf{h_{3}}(\mathbf{x})=\mathbf{h_{1}}(\mathbf{x})-\mathbf{h_{2}}(\mathbf{x})$.

Let $\mathbf{h}^{o_{s}}$ and $\mathbf{h}^{o_{t}}$ denote the ideal classifiers that perform optimally on the source data and target data, respectively:

In our approach, we design a classifier that learns the discrepancy between the weak annotator and the ground truth (details will be introduced in Section 4), and we denote it as $\mathbf{d}$ drawn from $\mathcal{Q}$. Thus, we can get a model that is the product of conducting the aforementioned $\oplus$ operator on $\mathbf{h}$ and $\mathbf{d}$, i.e., $\mathbf{h}\oplus\mathbf{d}$. Here $\mathbf{h}$ is designed for approximating the weak labels. And for the risk of $\mathbf{h}\oplus\mathbf{d}$, we can obtain the following relation:

Based on the error bound derived in Eq. (11), we can put efforts into the following ideas in our approach to improve the classifier performance in the target domain:

• $\bullet$

  Performance of annotator ($2\widehat{L}_{t}(\mathbf{h}^{w})+\widehat{L}_{s}(\mathbf{h}^{w})$): The supervision provided by the weak annotator can guide the model to better target the given task. Ideally, we want $\mathbf{h}^{w}$ to produce more accurate labels for both source and target data, reducing $2\widehat{L}_{t}(\mathbf{h}^{w})$ and $\widehat{L}_{s}(\mathbf{h}^{w})$ simultaneously. Practically though, we may just be able to make the annotator perform better on the source domain and cannot do much with the target domain.

• $\bullet$

  Discrepancy between domains ($\mathcal{DD}(\mathbb{Q}^{t}_{X},\mathbb{Q}^{s}_{X})$): Designing loss to quantify the discrepancy between the source and target domains is well studied in Domain Adaptation. In our approach, we propose a novel inter-domain loss (called Classified-MMD) to minimize $\mathcal{DD}(\mathbb{Q}^{t}_{X},\mathbb{Q}^{s}_{X})$, as introduced later.

• $\bullet$

  Quantity of source and target samples ($N_{s},N_{t}$): First, the learning of $\mathbf{d}$ needs the supervision of the ground truth, and thus we can only use the labeled target data to train $\mathbf{d}$. Then, in our method, $\mathbf{h}$ is designed to approximate the weak annotator, and therefore it may see enough that we just use the source data to train $\mathbf{h}$. However, to further reduce $K\!L_{\mathbf{h}}$ according to Theorem 1, we also use target samples to train $\mathbf{h}$, which increases the sample size of training data. Moreover, since the sample quantity of source data is much larger than that of target data (i.e., $N_{t}\ll N_{s}$), $\sqrt{K\!L_{\mathbf{d}}/N_{t}}$ in Eq. (11) dominates over $\sqrt{K\!L_{\mathbf{h}}/N_{s}}$, and in the case of $\delta\leq 2/e$, $12\sqrt{\ln{\frac{2N_{t}}{\delta}}/N_{t}}$ also strictly dominates over $8\sqrt{\ln{\frac{2N_{s}}{\delta}}/N_{s}}$. As the result, the terms influenced by a few target samples dominates the overall error risk. Therefore, directly applying $\mathbf{h}\oplus\mathbf{d}$ to the target domain will still be impacted by the insufficient samples. However, note that $\mathbf{h}\oplus\mathbf{d}$ can produce more accurate labels for the source data than the weak annotator. Therefore, we add a final step in our learning process that utilizes re-labeled source data and conducts supervised learning with such augmentation.

In this section, we present the detailed process of our weak adaptation learning (WAL) method, which is designed based on the observations from the above error bound analysis. The overview of our WAL process is shown in Figure 1. The designed network consists of three parts – ($\Phi_{0},\Phi_{1},\Phi_{2}$). $\Phi_{0}$ can be seen as a shared feature network for both source and target data, using typical classification networks such as VGG, ResNet, etc. $\Phi_{1}$ consists of three fully-connected layers that follow the output of $\Phi_{0}$. And we denote the combination of $\Phi_{0}$ and $\Phi_{1}$ as $F_{1}$. $\Phi_{2}$ consists of two fully-connected layers that follow the output of $\Phi_{0}$. The combination of $\Phi_{0}$ and $\Phi_{2}$ is denoted as $F_{2}$. The detailed network architecture is shown in the Supplementary Materials. The workflow of our method is shown in Algorithm 1.

Stage 1: The first goal we step on is to obtain a common representation for both the source and target data, which helps us encode the inputs while mitigating the domain discrepancy in the feature representation. We gather all the unlabeled source data and the target data without their labels and use weak annotator $\mathbf{h}^{w}$ to assign a label for each data sample $\mathbf{x}_{i}$ and $y^{w}_{i}=\mathbf{h}^{w}(\mathbf{x}_{i})$. We denote the dataset obtained in this way as $D=\{(\mathbf{x},y^{w})_{i}\}_{i=1}^{N_{s}+N_{t}}$. Then we fix $\Phi_{2}$ and only consider the left part of the network, which is $F_{1}=\Phi_{1}\circ\Phi_{0}$. It is normally trained by supervised learning using the dataset $D$ for ${ep}_{1}$ training epochs, and uses the following loss function:

In this loss function, there are two loss terms and the hyper-parameter $\alpha$ is a scaling factor to balance the scale of two loss functions (we set it as $0.0001$ in our experiments). The first term $\mathcal{L}_{K\!L}$ is the Kullback-Leibler (KL) divergence loss, stated as follows:

where $y_{pre}^{1}$ is the output prediction value of $F_{1}$ and $y^{w}$ is the corresponding weak label produced by the weak annotator $\mathbf{h}^{w}$. The second term $\mathcal{L}_{cmmd}$ aims to mitigate the domain discrepancy of the source and target domain at the feature representation level in the neural networks. Based on the basic MMD loss introduced by [55], we further change it into the version with data labels. We call this loss function as Classified-MMD loss (corresponding to the subscript $cmmd$), which is defined as:

where $M$ is the number of classes, $D_{X}$ is the data from the produced dataset $D$ without labels, and $D_{X}^{(S,i)}$ is the source data selected from $D_{X}$ with $\arg\max(y^{w})=i$. Then, we utilize target data with its accurate labels to continue to train the network component $F_{1}$ under the loss function $\mathcal{L}_{K\!L}$ for ${ep}_{2}$ training epochs, which helps further fine-tune the feature we learned through accurate labels of the target data.

Stage 2: After finishing training in Stage 1, the next step is to estimate the distance of the optimal classifier for target data $\mathbf{h}^{o_{t}}$ and the weak annotator $\mathbf{h}^{w}$. We estimate this distance through available target data with accurate labels. We adopt the parameters trained from Stage 1 and train network component $F_{2}=\Phi_{2}\circ\Phi_{0}$ using the target data $D_{t}$. For an input data sample $\mathbf{x}$, it is brought into both $\Phi_{0}$ and the weak annotator as their input. And then $\Phi_{2}$ takes the output feature of $\Phi_{0}(\mathbf{x})$ and $\mathbf{h}^{w}(\mathbf{x})$ as input feature (these two features are concatenated as the input feature of $\Phi_{2}$). For data sample $(\mathbf{x}_{t},y_{t})\in$ target dataset $D_{t}$, the learning of $F_{2}$ uses the following classifier discrepancy loss function:

The network is trained for ${ep}_{3}$ training epochs.

Stage 3: The third step is to generate a new dataset $D_{new}$ through the obtained network $F_{2}$ above. Specifically, we collect data $\mathbf{x}$ from both source data and target data, and we re-label these data based on the weak annotator and $F_{2}$ obtained from the previous steps:

Stage 4: In the last step, we focus on $F_{1}=\Phi_{1}\circ\Phi_{0}$ again. We fix the parameters of network component $\Phi_{2}$ and train $F_{1}$ using the new dataset $D_{new}$ obtained in Stage 3. To avoid introducing feature bias from the previous steps, we clean all previous network weights and re-initialize the whole network before training. The training lasts for ${ep}_{4}$ epochs, and the loss function for this step is $\mathcal{L}=\mathcal{L}_{K\!L}+\alpha\mathcal{L}_{cmmd}$, which is the same as the function in Stage 1. Finally, we get the final model $F_{1}$ as the desired classifier.

To sum up, in Stage 1, we learn the model $\mathbf{h}$ with the help of the weak annotator to decrease the empirical loss $\widehat{L}_{s}(\mathbf{h})$, and the CMMD loss will reduce the term $\mathcal{DD}(\mathbb{Q}^{t}_{X},\mathbb{Q}^{s}_{X})$. Stage 2 uses a new classifier $\mathbf{d}$ to learn the classification distance corresponding to the term $\widehat{L}_{t}(\mathbf{d})$. The Stage 3 uses the annotator and the learned $\mathbf{d}$ to give more accurate labels than those given solely by the annotator. Then in Stage 4, the model is trained by the relabeled data, making both $\widehat{L}_{s}(\mathbf{h})$ and $\mathcal{DD}(\mathbb{Q}^{t}_{X},\mathbb{Q}^{s}_{X})$ be further decreased. The setting of the hyper-parameters used in this section can found in the Supplementary Materials.

The experiments are conducted on three application scenarios, the digits recognition with domain discrepancy (SVHN[38], MNIST[6] and USPS[18] digit datasets), object detection with domain discrepancy (VisDA-C[42]), and object detection without domain discrepancy (CIFAR-10[25]). For space, we introduce details of these datasets in the Supplementary Materials.

All experiments are conducted on a server with Ubuntu 18.04 LTS with NVIDIA TITAN RTX GPU cards. The implementation is based on the Pytorch framework. The hyper-parameter $\alpha$ mentioned above is set to $1e-4$. We use the standard Adam optimizer  [22] for optimizing the learning. The network architectures, the learning rate for each part of the network components, the training epoch setting, as well as other hyper-parameters are specified in the Supplementary Materials. And we get weak annotators in different performance by applying early stop for the training. The implementation details of weak annotators can also be found in the Supplementary Materials.

We conduct comparison experiments with the following baselines. Baseline $B_{wa}$ is the performance of the weak annotator chosen in the experiments in the target domain. Baseline $B_{t}$ is training $F_{1}$ only with target data. Baseline $B_{f_{1}}$ is a fine-tuning result. It takes the same model as $F_{1}$ and first uses source domain data and weak labels generated by the weak annotator to train it. Then it uses target domain data to fine-tune the last three layers. Baseline $B_{f_{2}}$ is also a fine-tuning result. The difference is that instead of fine-tuning the last three layers, it trains all network parameters.

As introduced before, our problem is related to the Semi-Supervised Learning (SSL) and the Semi-Supervised Domain Adaptation (SSDA). For SSL, although we can replace the unlabeled data with samples drawn from another domain instead of the target domain, we cannot find a good way to incorporate the weak annotator into SSL methods for fair comparison with our approach. For SSDA, we were able to extend it to our setting for comparison. Specifically, we add 1,000 unlabeled target samples (plus 1,000 labeled target samples, and this setting will be changed accordingly in digits recognition to keep consistent settings) to meet the semi-supervised requirement, and we apply weak annotator to produce weak labels instead of accurate ones for source data. We compare our approach with the following SSDA baselines: FAN [21], MME [48], ENT [13], S+T [4, 45]. Note that to the best of our knowledge, there is no previous work with exactly the same problem setting as ours. The above changes aim at making the comparison as fair as possible. Another thing that is worth to mention is that most SSDA methods conduct adaptation on the ImageNet pre-trained models, which introduces a lot of irrelevant data information from the ImageNet dataset. Thus, we disable the pre-training and only allow training with the available data.

We evaluate our methods on the digit recognition datasets: SVHN (S), MNIST (M),and USPS (U). According to the results shown in Table 1, when the weak annotator performs much worse than the model learned only from the provided target data $B_{t}$ ($B_{wa}$ = $73.28\%$ on M $\to$ U, $73.28\%$ on S $\to$ U and $76.41\%$ on S $\to$ M), its corresponding baseline $B_{f_{1}}$ is also lower than $B_{t}$, and only the second fine-tuning method $B_{f_{2}}$ is better than or competitive with $B_{t}$. This indicates that the feature learned from the source domain data and with weak labels introduce data bias, and this bias can be mitigated when the parameters from the front layers are fine-tuned by the target data.

Overall, we can clearly see that with 15,000 source domain data, limited number of labeled target domain data (second line), and a weak annotator, our method can outperform all the baselines in Table 1 with $80.00\%$ on M $\to$ S, $95.99\%$ on M $\to$ U, $96.36\%$ on S $\to$ U and $97.24\%$ on S $\to$ M.

The results of various methods on the VisDA-C dataset are presented in Figure 2. In this task, we utilize the synthetic images as the source domain dataset, and the real-world images as the target domain dataset. And we can see from the table that the performance of the network trained only with the target data is merely $32.86\%$. Then, when the weak annotator is provided, it can help two fine-tune baselines $B_{f_{1}}$ and $B_{f_{2}}$ reach $27.67\%$ and $35.03\%$ respectively. As for the SSDA baselines, all of them perform very badly, and they are provided with more target samples with no labels. The best SSDA methods FAN can only achieve $32.99\%$. Our method can provide a result of $40.83\%$, which again exceeds all baselines above. Besides, we also provide additional experiment results using a different weak annotator in the supplementary materials.

Moreover, we also test on the scenario without domain discrepancy using the CIFAR-10 dataset. We randomly select 10,000 data samples from the dataset as the source data and another 1,000 samples as the target data. The result is included in Table 2. As we can see, when the weak annotator is given at $48.96\%$ accuracy, the model trained only with the target data can reach $30.46\%$, while our method nearly doubles the performance and hits $61.71\%$, which exceeds all other baselines.

We also study how the quantity of target domain samples and the performance of the weak annotator affect the overall performance of our method. To reduce the impact of domain discrepancy when we study these two factors, we conduct the ablation study on CIFAR-10.