Domain Adaptation via Maximizing Surrogate Mutual Information

Let $\mathcal{X}$ be the data space and $\mathcal{Y}$ be the label space. In UDA, there is a source distribution $P_{S}(X,Y)$ and a target distribution $P_{T}(X,Y)$ on $\mathcal{X}\times\mathcal{Y}$. Note that distributions are also referred to as domains in UDA. Our work is based on the data shift hypothesis, which assumes $P_{S}(X,Y)$ and $P_{T}(X,Y)$ satisfy the following properties: $P_{T}(Y)=P_{S}(Y)$ and $P_{T}(X|Y)\not=P_{S}(X|Y)$ .

In our work, we focus on classification tasks. Under this setting, an algorithm has access to $n_{S}$ labeled samples $\{(x_{S}^{i},y_{S}^{i})\}_{i=1}^{n_{S}}\sim P_{S}(X,Y)$ and $n_{T}$ unlabeled samples $\{(x_{T}^{i})\}_{i=1}^{n_{T}}\sim P_{T}(X)$, and outputs a hypothesis composed of an encoder $G$ and a classifier $F$. Let $\mathcal{Z}$ be the feature space. The encoder maps data to feature space, denoted by $G:\mathcal{X}\rightarrow\mathcal{Z}$. Then the classifier maps the feature to a corresponding class, $F:\mathcal{Z}\rightarrow\mathcal{Y}$.

For brevity, given encoder $G$ and data-label distribution $P(X,Y)$, denote the distribution of $G$-encoded feature and label by $P^{G}$, i.e. $P^{G}(z,y)=P(x=G^{-1}(z),y)$.

Let $F$ be a hypothesis, and $P$ be the distribution of feature and label. The expected risk of a F w.r.t. P is denoted as

where $\delta_{F(z)}(y)$ equals to 1 if $y=F(z)$ and equals 0 in else cases. Our objective is to minimize the expected risk of $F$ on target feature distribution encoded by $G$,

In UDA task, the model needs to generalize across different domains with varying distributions; thus the encoder needs to extract appropriate features that are transferable across domains. The challenges of class-level adaptation are two folds: learning transferable features, and modeling $P^{G}_{T}(Z|Y)$ without label information.

To solve the first problem, we use MI based methods. Following the InfoMax principle, we maximize the mutual information between features from the same class on the target and source mixture distribution. This encourages the features of the source domain to carry more information about the features of the same class in target domain, and thus provides opportunities for transferring classifier across domains.

As for the second challenge, we first revisit the data shift hypothesis. The distribution of labels $P(Y)$ remains independent of domains; therefore the key is to model the conditional distribution $P(Z|Y)$ on the target domain. However, modeling $P(Z|Y)$ is intractable, since labels on the target domain are inaccessible. To tackle this problem, we model a surrogate distribution $Q(Z|Y)$ instead.

We introduce the goal of maximizing MI in section 4.2, and theoretically explain how MI affects domain adaptation risk. In Section 4.3, we will introduce the model in detail, including the variational estimation of MI, the modeling of the surrogate distribution, and the optimization of the loss function of the model.

MI measures the degree to which two variables can predict each other. Inspired by InfoMax principle Hjelm et al. (2018), we maximize the MI between the features within the same class. It encourages features from different classes to be discriminable from each other.

We maximize MI between features on both source and target domain, regardless of which domain they come from. So we introduce mixture distribution $S+T$ of both domain, which is

Note that because $P_{S}(y)=P_{T}(y)=P_{S+T}(y)$, $P_{S+T}(x|y)=\frac{1}{2}P_{S}(x\mid y)+\frac{1}{2}P_{T}(x\mid y)$. Define the distribution of features from the same class as

which means the feature $z_{1}$ and $z_{2}$ are sampled independently from the conditional distribution of the same class, with equal probability from source domain and target domain.

MI between features is maximized within the mixture distribution, as formalized bellow:

However, due to the lack of target domain labels, $P^{G}_{S+T}$ is hard to model and thereby it is infeasible to estimate $I^{G}_{S+T}$ directly. To address this problem, we propose a surrogate joint distribution $Q(Z,Y)$ as the substitute for target domain $P^{G}_{T}$. Then the mixture distribution becomes $P_{S+Q}^{G}=\frac{1}{2}(P^{G}_{S}+Q)$, and the objective becomes maximizing $I_{S+Q}^{G}(Z_{1};Z_{2})$. The construction and optimization of the surrogate joint distribution is explained in Section 4.3.2.

We employ MI maximization and surrogate distribution in our SIDA framework, as shown in Figure 2. During training, a surrogate distribution is first built from target and source data via optimizing w.r.t. Laplacian and MI. Then a mixture data distribution is created by encoding source data to features and sampling target features from the surrogate distribution. The encoder is optimized by maximizing MI, and minimizing classification error. The overall loss is:

We elaborate each module in the following sections, and introduce the optimization of surrogate distribution in the last sections.

VisDA-2017 Peng et al. (2017) is a challenging benchmark for UDA with the domain shift from synthetic data to real imagery. It contains 152,397 training images and 55,388 validation images across 12 classes. Following the training and testing protocol in Long et al. (2017a), the model is trained on labeled training and unlabeled validation set and tested on the validation set.

Office-31 Saenko et al. (2010) is a commonly used dataset for UDA, where images are collected from three distinct domains: Amazon (A), Webcam (W) and DSLR (D). The dataset consists of 4,110 images belonging to 31 classes, and is imbalanced across domains, with 2,817 images in A domain, 795 images in W domain, and 498 images in D domain. Our method is evaluated on all six transfer tasks. We follow the standard protocol for UDA Long et al. (2017b) to use all labeled source samples and all unlabeled target samples as the training data.

Office-Home Venkateswara et al. (2017) is another classical dataset with 15,500 images of 65 categories in office and home settings, consisting of 4 domains including Artistic images (A), Clip Art images (C), Product images (P) and Real-World images (R). Following the common protocol, all 65 categories from the four domains are used for evaluation of UDA, forming 12 transfer tasks.

For each transfer task, mean ($\pm$std) over 5 runs of the test accuracy are reported. We use the ImageNet pre-trained ResNet-50 He et al. (2016) without final classifier layer as the encoder network $G$ for Office-31 and Office-Home, and ResNet-101 for VisDA-2017. The details of experiments are in appendix. The code is available at https://github.com/zhao-ht/SIDA.

We compare our approach with the state of the arts. Domain alignment methods include DAN Long et al. (2015), DANN Ganin et al. (2016), JAN Long et al. (2017b). Class-level methods include conditional alignment methods (CDAN Long et al. (2018), DCAN Li et al. (2020c), ALDA Chen et al. (2020a)), and contrastive methods (DRMEA Luo et al. (2020), ETD Li et al. (2020a), DADA Tang and Jia (2020), SAFN Xu et al. (2019)). We only report available results in each baseline. We use NA, DA, CA, CT to note no adaptation method, domain alignment methods, conditional alignment methods and contrastive methods respectively.

In this section we will present our results and compare with other methods for evaluation on three standard benchmarks mentioned earlier. We report average classification accuracies with standard deviations. Results of other methods are collected from original papers or the follow-up work. We provide visualizations of the features learned by the model in the appendix.

VisDA-2017    Table 1 summarizes our experimental results on the challenging VisDA-2017 dataset. For fair comparison, all methods listed here use ResNet-101 as the backbone network. Note that SIDA outperforms baseline models with an average accuracy of 84.0, surpassing the previous best result reported by +4%.

Office-31   The unsupervised adaptation results on six Office-31 transfer tasks based on ResNet-50 are reported in Table 2. As the data reveals, the average accuracy of SIDA is 90.4, the best among all compared methods. It is noteworthy that our proposed method substantially improves the classification accuracy on hard transfer tasks, e.g. W$\rightarrow$A, A$\rightarrow$D, and D$\rightarrow$A, where source and target data are not similar. Our model also achieves comparable classification performance on easy transfer tasks, e.g. D$\rightarrow$W, W$\rightarrow$D, and A$\rightarrow$W. Our improvements are mainly on hard settings.

Office-Home   Results on Office-Home using ResNet-50 backbone are reported in Table 3. It can be observed that SIDA exceeds all compared methods on most transfer tasks with an average accuracy of 71.2. The performance reveals the importance of maximizing MI between feature in difficult domain-adaptation tasks which contain more categories.

In summary, our surrogate MI maximization approach achieves competitive performance compared to traditional alignment based methods and recent pseudo-label based methods for UDA. It underlines the validity of using information theory methods for UDA via MI maximization.

In this section, to evaluate how different components of our work contribute to the final performance, we conduct ablation study for SIDA on Office-31. We mainly focus on harder transfer tasks, e.g. A$\rightarrow$W , A$\rightarrow$D, D$\rightarrow$A and W$\rightarrow$A. We investigate different combinations of two components:MI maximization and surrogate distribution (SD). Note that without surrogate distribution, we use pseudo label computed by the same method as surrogate distribution initialization to estimate MI. The average classification accuracy on four tasks are in Table 4.

From the results, we can observe that the model with MI maximization outperforms the base model without the two components by about 2.5% on average, which demonstrates the effectiveness of the maximization strategy. The surrogate distribution also improves the average performance by 1.1% compared to base model, confirming that the surrogate distribution improves the estimation quality of target domain compared to pseudo label method. The combination of two components yields the highest improvement.