Multi-level Consistency Learning for Semi-supervised Domain Adaptation

In the semi-supervised domain adaptation (SSDA) problem, the source domain $\mathcal{D}^{s}=\{\mathcal{X}_{s},\mathcal{Y}_{s}\}$ is fully labeled, and the target domain $\mathcal{D}^{t}=\{\mathcal{X}_{t},\mathcal{Y}_{tl}\}$ contains a few labels $\mathcal{Y}_{tl}$ for partial samples $\mathcal{X}_{tl}$, so $\mathcal{D}^{t}=\{\mathcal{X}_{tl},\mathcal{Y}_{tl}\}\cup\{\mathcal{X}_{tu}\}$, where $\{\mathcal{X}_{tu}\}$ is the unlabeled target sample set. SSDA methods aim to learn a classifier with high accuracy on the target domain using the rich data-label pairs $\{\mathcal{X}_{s},\mathcal{Y}_{s}\}$ and few ones $\{\mathcal{X}_{tl},\mathcal{Y}_{tl}\}$. We assume that (i) the source and target domains share the same label space of $C$ classes; (ii) there are only a few labels for each class in the target domain, e.g. one label (1-shot) or three labels (3-shot).

SSDA model usually consists of two components, a feature extractor $\mathcal{G}$ and a classifier $\mathcal{F}$, parameterized by $\theta_{\mathcal{G}}$ and $\theta_{\mathcal{F}}$, respectively. Following Sohn et al. (2020), we first generate two views for each target sample $x_{t}\in\mathcal{X}_{t}$, denoted as $x_{t}^{A}$ and $x_{t}^{B}$, through the weak (view $A$) and strong (view $B$) augmentation, respectively. The two views are then fed into $\mathcal{G}$ to obtain their feature representations $\mathbf{f}_{t}^{A},\mathbf{f}_{t}^{B}\in\mathbb{R}^{d}$, which finally go through $\mathcal{F}$ to get the probabilistic prediction $\mathbf{p}_{t}^{A},\mathbf{p}_{t}^{B}\in\mathbb{R}^{C}$. This process can be formulated as:

where $\sigma(\cdot)$ is the softmax function. Similarly, in the source branch, each source sample $x_{s}$ is fed into the same feature extractor $\mathcal{G}$ to generate $\mathbf{f}_{s}\in\mathbb{R}^{d}$, and then into the classifier $\mathcal{F}$ to obtain the prediction $\mathbf{p}_{s}\in\mathbb{R}^{C}$.

To train the model, each sample-label pair $(x,y)$ (from either the source or target domain) can be used to minimize a standard cross-entropy (CE) classification loss as follows:

where $p(x)$ is the prediction of sample $x$ through $\mathcal{G}$ and $\mathcal{F}$. As detailed below, to fully utilize the unlabeled target data $\mathcal{X}_{tu}$, our MCL framework regularizes the consistencies in SSDA at three levels.

At the inter-domain level, we propose a novel consistency regularization method to robustly and accurately align samples from the target domain to those of the source domain, so that the knowledge in the source domain can be better transferred into the target domain. To obtain such a high-quality alignment, we compute the mapping plan from two views of the target sample to the source sample, and leverage the consistency between the two mappings to guide the learning process. Specifically, in each iteration, we first compute the mapping from the view $A$ of a target sample $x_{t}^{A}$ to a source sample $x_{s}$, according to which the representations of its view $B$, $x_{t}^{B}$ and $x_{s}$ are clustered closer. Accordingly, our method alternates the following two steps throughout the training:

To facilitate representation learning in the target domain, we incorporate consistency regularization into the intra-domain level. Specifically, we aim to learn both compact and discriminative class-wise clusters in the target domain, i.e., the clusters of the same class from two views should be grouped, while those of different classes should be pushed away from each other. Existing unsupervised contrastive learning methods focus on sample-level representation learning, which still hardly learn highly discriminative clusters for each class Khosla et al. (2020).

Instead of the sample-wise contrastive learning, we propose a novel class-wise contrastive clustering loss function, by modeling the cluster of each class as all samples’ classification confidence of this class in a batch, named as batch-wise class assignment (illustrated as the column bounded by a red rectangle in Fig .1). The class-wise contrastive clustering has two purposes, (i) keeping the consistency of two views for the same batch-wise class assignment (positive pairs), and (ii) enlarging the distance between the batch-wise assignment of different classes (negative pairs).

Formally, given a batch of target samples’ predictions $\mathbf{P}\in\mathbb{R}^{n\times C}$, where $n$ is the mini-batch size and $\mathbf{P}_{i,j}$ represents the confidence of classifying the $i$th sample into the $j$th class. With the cluster assumption that each feature of samples in the same class should form a high-density cluster, we regard each column of $\mathbf{P}$ as the batch-wise assignment of one class. Although the columns of $\mathbf{P}$ will dynamically change with the iteration of mini-batches during training, the class assignment in different views of the same image should be similar, while dissimilar to other images. Therefore, we compute the cross-correlation matrix $\mathbf{\Lambda}\in\mathbb{R}^{C\times C}$ between two views $A$ and $B$ to measure the class-wise similarity, formulated as:

where $\mathbf{P}^{A}$ and $\mathbf{P}^{B}$ are the batch prediction from view $A$ and $B$, respectively. $\mathbf{\Lambda}$ is an asymmetry matrix with each element $\mathbf{\Lambda}_{i,j}$ measuring the similarity between $\mathbf{P}^{A}$’s $i$th column and $\mathbf{P}^{B}$’s $j$th column. The objective is to increase the correlation values between two views of the same class assignment (diagonal values) while reducing ones for different classes’ views (off-diagonal values), which yields the loss function of intra-domain prediction consistency learning:

where $\phi(\cdot)$ is a normalization function to keep the row summation as 1, e.g., similar to MCC  Jin et al. (2020), divide the elements with its row summation, $\mathbf{I}\in\mathbb{R}^{C\times C}$ is an identity matrix, and $\|\cdot\|_{1}$ denotes the summation of the absolute values of the matrix. Note that the loss function is the summation of two terms for respectively normalizing the rows and columns of the correlation matrix $\mathbf{\Lambda}$. Such two normalization operations can alleviate the influence of class imbalance in a batch, i.e., the values in the class assignment may be too large or small according to the number of samples that belong to this class. Intuitively, the normalized row of correlation values can be regarded as the scores of matching one class assignment with each column in the other view to be a positive pair. Thus the dual design helps bi-directional matching score computation, i.e., the $i$th row in $\phi(\mathbf{\Lambda})$ represents the matching scores from $\mathbf{P}^{A}$’s $i$th column to $\mathbf{P}^{B}$’s each column, while the one in $\phi(\mathbf{\Lambda}^{\top})$ represents matching in the inverse direction.

Note that minimizing Eq. 11 has the following properties: i) The rows of $\mathbf{P}^{A}$ and $\mathbf{P}^{B}$ are encouraged to be sharper, which implements entropy minimization Grandvalet et al. (2005) in label-scarce scenarios. ii) For each sample, the prediction of view $A$ and $B$ are encouraged to become similar, which coincides with the consistency regularization Chen et al. (2020) that helps the network to learn better representations. iii) It avoids a trivial solution that predicts all samples as the same class: the diagonal values of the normalized cross-correlation matrix are enforced to be close to 1, which punishes the trivial solution by a large loss.

Similar to Li et al. (2021a), we regularize the consistency at the sample level via pseudo labeling: the prediction of view $A$ of a target sample is set as the pseudo label of view $B$ of the target sample. Specifically, for each sample $x_{tu}$, we compute the cross-entropy loss as:

where $p_{i}^{A}$ and $p_{i}^{B}$ are the $i$th elements in the prediction of $x_{tu}$ from view $A$ and view $B$, respectively, $\mathds{1}(\cdot)$ is the indication function, and $\tau$ is the threshold of confidence score.

Therefore, the overall objective can be formulated as:

where $\lambda_{1}$ and $\lambda_{2}$ are the hyper-paramters to balance $\mathcal{L}_{inter}$ and $\mathcal{L}_{intra}$, respectively.

We compare our MCL framework with i) two baseline methods S+T and ENT Grandvalet et al. (2005); and ii) several existing methods, including DANN Ganin et al. (2016), MME Saito et al. (2019), UODA Qin et al. (2021), BiAT Jiang et al. (2020), MetaMME Li and Hospedales (2020), APE Kim and Kim (2020), ATDOC Liang et al. (2021), STar Singh et al. (2021), CDAC Li et al. (2021a), and DECOTA Yang et al. (2021). Between the two baselines, “S+T” uses only source samples and labeled target samples in the training; “ENT” uses entropy minimization for unlabeled samples Grandvalet et al. (2005). Note that all methods are implemented on a ResNet34 backbone for a fair comparison.