Contrastive Conditional Alignment based on Label Shift Calibration for Imbalanced Domain Adaptation

In domain adversarial learning, an auxiliary domain classifier $\mathbf{D}$ is employed to determine whether the features extracted by $\mathbf{G}$ are derived from the source or target domain. Simultaneously, $\mathbf{G}$ is trained to deceive $\mathbf{D}$. When this adversarial game reaches a state of equilibrium, the features produced by G demonstrate domain invariance. Formally,

However, domain-invariance does not mean cross domain class-invariance. In  [41], they propose to use moving average centroid alignment strategy. This strategy explicitly constrains the distance between centroids with identical class but different domains, ensuring close mapping of same-class features. The transfer objective is:

where $C^{k}_{S}$ and $C^{k}_{T}$ represent the centroid of class $K$ of the source and target domains respectively, and $\Phi(\cdot)$ represents the Euclidean distance between the two. This strategy is designed to mitigate the adverse effects of incorrect pseudo-labels. However, in situations with severe label shift, an excess of unreliable pseudo-labels can misalign centroids. We suggest that each sample’s contribution to the centroid calculation varies based on its reliability. For instance, in a binary classification problem, if samples $x_{1}$ and $x_{2}$ have probability outputs $[0.9,0.1]$ and $[0.6,0.4]$ respectively, $x_{1}$ is more reliable. Hence, we use confidence as a sample weight. For a sample $x$, the final probability output through a deep model parameterized by $\theta$ is represented as $p_{\theta}(y|x)$, with a weight of $w=\max p_{\theta}(y|x)$.

Inspired by contrastive learning  [43], the centroids with same class label but different domains should be closer, while the centroids with different class labels and domains should be futher away. We rewrite Eq.(2) as follows:

where $C^{k}_{wS}$ and $C^{k}_{wT}$ represent the centroids weighted by $w$.

In scenarios with two domains exhibiting significant distribution disparities, our goal is to ensure domain-invariant and class-discriminative features. Features with identical class labels across domains align closely, while those with different labels are distinctly separated. We propose discriminative feature alignment, a contrastive learning-based method, to facilitate this. It computes the difference between each feature pair from the source and target domains, using actual labels for the source and classifier-produced pseudo-labels for the target. Identical class labels draw features closer, while differing labels push them apart, effectively enabling cluster learning for robust classification boundaries. To avoid over-attracting unreliable samples, we persist in using w as a sample weight. Formally,

The above strategies address covariate shift by aligning conditional distributions. However, when class imbalance is present, label shift becomes more pronounced, leading to a biased classifier and impacting the reliability of pseudo-labels. Given the unknown target domain, for generality and simplicity, we employ class-balanced sampling on the source domain. Specifically, when selecting samples for a mini-batch, each class has an equal probability of being selected.

Label shift, quantifies the disparity in label distributions between source and target domains. It varies per class due to differing quantity distributions across domains. $M_{ls}$ is defined with respect to the probability distributions $P_{S}$ and $P_{T}$ of the source and target domains respectively.

$M_{ls}$, a $1\times C$ tensor, measures label shift, where $C$ is the number of classes and $M_{ls}^{i}$ represents the label shift degree for class $i$. If ${M_{ls}^{i}}=1$, there’s no label shift for class $i$. If ${M_{ls}^{i}}>1$, class $i$ is more prevalent in the target domain, and if $0<{M_{ls}^{i}}<1$, it’s more prevalent in the source. Both cases indicate label shift, affecting pseudo-label reliability and potentially leading to error accumulation and performance degradation. We derive $P_{S}$ from source labels. The unlabeled target domain’s $P_{T}$ is approximated using pseudo-labels, denoted as $\widehat{P}_{T}$.

In deep learning classification models, we decompose them into a feature extractor $G$ and a classifier $F$. The goal of domain adaptation is to align the features extracted by $G$ from two domains. When dealing with long-tailed source data, $F$ tends to favor head classes due to their larger quantity, which can lead to suboptimal learning for tail classes with fewer instances. However, even if we adopt class-balanced sampling for the source domain, ensuring an unbiased $F$, the reliability of target sample labeling remains uncertain when the target domain follows a class-imbalanced long-tail distribution. To address this, we propose LSC based on the degree of label shift $M_{ls}$. LSC calibrates the classification predictions for target samples during training, making the pseudo labels more consistent with the real target data’s probability distribution, thus improving the reliability of target pseudo-labels.

For a target sample through a model with parameters $\theta$, we use $p_{\theta}(y|x_{T})$ to represent its final probability output. The idea of LSC is to reweight $p_{\theta}(y|x_{T})$ based on the degree of label shift $M_{ls}$, in order to re-estimate the target pseudo-labels. The class weighting matrix $W_{m}$ is designed as:

Then we obtain target pseudo labels after calibration and its confidence weight:

A larger $M_{ls}[i]$ suggests that class $i$ is less frequent in the source but more so in the target domain, and vice versa for a smaller $M_{ls}[i]$. As per Eq.6 and Eq.7, when a sample’s feature is on the boundary of two classes and $M_{ls}[i]>M_{ls}[j]$, we prefer to label the sample as $i$, as shown in Figure 1. $W_{m}$ bounds the class weighting values, with $h_{m}$ set to 1.5, indicating that only unreliable samples at the classification boundary are calibrated to prevent over-calibration. The sample’s confidence weight $w^{m}$ is determined by the classifier’s output, mitigating the negative effects of incorrect classification calibration.

In summary, our training process comprises two stages. The first stage involves pre-training for three epochs, utilizing high-confidence target samples from the training results to estimate the target domain’s label distribution. The optimization objective of the first stage is:

In the second stage, we employ LSC to rectify target pseudo-labels $\widehat{y}_{t}^{m}$, and utilize $\widehat{y}_{t}^{m}$ for the training of CCA. Then our optimization objective is:

where $\lambda$ and $\mu$ and $\gamma$ are hyperparameters no less than zero.

Next, we demonstrate how our approach reduces the expected error on the target samples from domain adaptation theory.

It is defined as $C_{0}=\min_{h\in\mathcal{H}}\varepsilon_{\mathcal{S}}(h,f_{\mathcal{S}})+\varepsilon_{\mathcal{T}}(h,f_{\mathcal{T}})$ where $f_{\mathcal{S}}$ and $f_{\mathcal{T}}$ are labeling functions for source and target domain respectively. Previous methods often assume that $C_{0}$ is negligible. However, when $C_{0}$ is large, ignoring $C_{0}$ can prevent the learning of an effective target classifier.

In the given formula, the first two terms quantify the discrepancy between the hypothesis $h$ and the source labeling function $f_{\mathcal{S}}$. Given the availability of source labels, these terms are typically minimal, facilitating the learning of a hypothesis space $h$ that closely approximates $f_{\mathcal{S}}$. The third term measures the inconsistency between the source and pseudo-target labeling functions on target samples, while the final term indicates the divergence between the pseudo-target labeling function and the true target label, serving as a reliability measure for the pseudo-labels. Our method seeks to minimize the last two terms to optimize the upper bound of $C_{0}$. The moving average centroid alignment strategy, discussed in  [41], optimizes the third term by aligning the centroids of target and source features in class $C_{0}$, ensuring prediction consistency. Our approach employs both sample-weighted moving average centroid alignment and discriminative feature alignment to foster feature alignment across different domains but within the same class, thereby minimizing the third term.

However,  [41] presumes the fourth term will minimize over time and disregards it. This assumption falls short in the presence of data imbalance and label shift, where optimizing the third term could induce class bias in the pseudo-target labeling function, amplifying the fourth term. Our proposed LSC rectifies this by adjusting the classification prediction of the pseudo-target labeling function based on the label shift index $M_{ls}$, reducing the false pseudo-rate, and aligning the prediction with the true target data’s label distribution, thereby also minimizing the fourth term. Our experiments demonstrate that LSC consistently curtails the false pseudo-rate on target samples (refer to section 4.4).

In essence, the efficacy of domain adaptation methods hinges on managing each term that could escalate the target classification error, thus broadening the applicability of domain adaptation methods.

We utilized three datasets . First, we employed Office-Home (RS-UT), an imbalanced version of Office-Home created by  [32], where the source and target domains follow two reverse Paredo distributions. This benchmark includes three domains: Clipart (Cl), Product images (Pr), and Real-world images (Rw). The Art images (Ar) domain in Office-Home, being too small for sampling an imbalanced subset, is not considered here. Second, we used a subset of DomainNet created by  [32], which includes 40 classes from four domains (Real (R), Clipart (C), Painting (P), Sketch (S)). As a noticeable label shift already exists, we made no additional modifications. The label distributions can be seen in Figure 1. Office-31  [44] contains 4,110 images of 31 categories. The domains are Amazon (A), Webcam (W), and DSLR (D).

We benchmarked our method against eight state-of-the-art techniques that tackle both covariate shift and label shift. (i) COAL  [32] aligns feature and label distributions using prototype-based conditional alignment and self-training on confident pseudo-labels. (ii) MDD+Implicit Alignment (I.A)  [36] removes explicit model parameter optimization from pseudo-labels via sampled implicit alignment. (iii) InstaPBM  [45] employs instance-based prediction behavior matching. (iv) F-DANN  [37] introduces a DANN based on asymmetric relaxed distribution matching. (v) SENTRY  [39] minimizes the entropy of reliable instances and maximizes that of unreliable ones. (vi)TIToK [46] and (vii)BIWAA-I [47] and (viii)RHWD [48] also solve both label and feature shifting problems. All methods, except F-DANN, use target pseudo-labels. We also compared with conventional UDA methods like BBSE [23], which only addresses label shift, and MCD [17], DAN [4], DANN [5], JAN [7], BSP [49], which solely focus on covariate shift.

All experiments are conducted using the Pytorch framework with resnet50. The model’s hyper-parameters are $\lambda=3$, $\mu=0.6$, and $\gamma=1$. The bottleneck layer dimension is 256, and the batch size is 50. We use the SGD optimizer with a momentum of 0.9. The initial learning rate for the classifier is 0.005 for OfficeHome and 0.01 for DomainNet, adjusted as  [5]. The model trains for 20 epochs, with the first 3 forming the initial stage. After this, the model evaluates target samples and uses pseudo-labels with a confidence level of $w>0.5$ to estimate the target domain’s label distribution. The model then enters the second stage. The random seed is set to 100 for reproducibility. For imbalanced data, we use per-class mean accuracy, as suggested by  [32], for a fair performance assessment.