IDAL: Improved Domain Adaptive Learning for Natural Images Dataset

Unsupervised Domain Adaptation (UDA) addresses the performance degradation caused by domain shift in supervised learning, where there’s a significant distribution difference between training (source) and testing (target) data domains. Adversarial-based UDA, utilizing frameworks like Generative Adversarial Networks (GANs) [25] and Domain Adversarial Neural Networks (DANN) [8], aims to mitigate this by learning domain-invariant features from unlabeled target data. By promoting feature harmonization while retaining class information, these models enhance target domain generalization. Despite promising results in image classification and object detection, adversarial UDA faces challenges such as hyper-parameter sensitivity, high-dimensional space navigation, and domain shift detection.

To address the aforementioned challenge, we developed an unsupervised domain adaptation approach that surpasses the state-of-the-art UDA performance of convolution neural networks (CNNs) for benchmark natural image datasets – Office-Home [36], Office-31 [29], VisDA-2017 [26], and shows comparable results for DomainNet [27].

Inspired by the concept of a conditional domain adversarial network (CDAN)  [22], our core approach – Improved Domain Adaptive Learning (IDAL) – involves concurrent training of a feature extractor (typically a deep neural network) and a domain classifier (discriminator) tasked with distinguishing between source and target domains. We have explored various CNN-based feature extractors such as ResNet-101, ResNet-50 [12], ViT [6], and ConvMixer [33] to extract meaningful features. The feature extractor’s aim is to learn representations that are invariant to domain shifts, and thus deceive the domain classifier that endeavors to correctly classify the domain of the extracted features. The integration of ResNet-50 and FPN combines [20] deep feature representation and multi-scale extraction, essential for tasks like object detection and segmentation. Given that object scale and style vary by domain, this synergy makes ResNet + FPN a strong candidate for Unsupervised Domain Adaptation (UDA), focusing on higher-level domain-specific feature suppression. This application to UDA is novel.

In the adversarial training process, the feature extractor (ResNet-50 + FPN) and domain classifier compete: the extractor aims to produce domain-agnostic features, while the classifier attempts to distinguish between domains. This method fosters the development of domain-invariant features, enhancing transferability across source and target domains.

To improve the training process, we propose a novel loss function called pseudo label maximum mean discrepancy (PLMMD). We use this loss in addition to certain existing losses – maximum information loss (entropy loss) [15], maximum mean discrepancy (MMD) loss [21], minimum class confusion (MCC) loss [13]. Our model integrates several loss functions to enhance domain adaptation and classification accuracy: Maximum information loss clusters target features by class, preserving key information. MMD loss bridges domain gaps by comparing mean embeddings. MCC loss boosts accuracy by minimizing class confusion, vital for uneven datasets. Our innovative PLMMD loss selectively extracts domain-invariant features, speeding up training. This tailored mix of loss functions enables our method to outperform existing CNN models and achieve quicker convergence on natural image datasets– Office-Home [36], Office-31 [29], and VisDA [26].

In unsupervised domain adaptation (UDA), we have data from a source domain $D_{s}=\{(x_{s_{i}},y_{s_{i}})\}_{i=1}^{n_{s}}$ as ${n_{s}}$ labeled examples and that from a target domain $D_{s}=\{(x_{t_{i}},y_{t_{i}})\}_{i=1}^{n_{t}}$ as ${n_{t}}$ unlabeled examples where $y_{t_{i}}$’s are unknown. The source domain and target domain are sampled from the distributions $P({x_{s}},{y_{s}})$ and $Q({x_{t}},{y_{t}})$ respectively. Notably, the two distributions are initially not aligned; that is, $P\neq Q$.

Domain adversarial neural network (DANN) [8] is a framework of choice for UDA. This is a dual-player game involving two key components: the domain discriminator, denoted as $D$, and the feature representation, denoted as $F$. In this setup, $D$ is trained to differentiate between the source domain and the target domain, while $F$ is simultaneously trained to both confound the domain discriminator $D$ and accurately classify samples from the source domain. The discrepancy between the feature distributions $P_{F}$ and $Q_{F}$ [7] has well corresponding with the error function of the domain discriminator. This is a key to bound the risk associated with the target in the domain adaptation theory [2].

An alternative approach in the field of Unsupervised Domain Adaptation (UDA) focuses on reducing the domain discrepancy as quantified by various metrics, e.g., maximum mean discrepancy(MMD). To establish class-level alignment across domains, the methodology outlined in the study conducted by Pei and colleagues [26] incorporates a multiplicative interaction between feature representations and class predictions. In their studies [3], efforts are made to ensure alignment between the centroids of labeled source data and the centroids derived from pseudo-labeled target data, particularly for shared classes within the feature space.

Another approach to UDA involves employing separate task classifiers for each of the two domains. These classifiers are used to identify non-discriminative features. In turn, they facilitate the learning of a feature extractor that focuses on generating discriminative features [17]. Several other studies emphasize the importance of directing attention towards transferable regions as a means to establish a domain-invariant classification model, as exemplified by [16]. In addition, for the purpose of extracting target-discriminative features,[14] employ techniques such as generating synthetic data from the raw input of the two domains, as described in[25].

Since our work modifies the network and losses of the CDAN framework [22], we explain it here for completeness. To reduce the shift in data distributions across the domains, CDAN trains a deep network $N:x\rightarrow y$, so that source risk $r_{s}$= $E_{(x_{s},y_{s})\sim P}[N(x_{s})\neq y_{s}]$ can bound the target risk $r_{t}$= $E_{(x_{t},y_{t})\sim Q}[N(x_{t})\neq y_{t}]$ plus the distribution discrepancy $disc(P,Q)$ quantified by a novel conditional domain discriminator. In the context of adversarial learning, Generative Adversarial Networks (GANs) [8] play a pivotal role in mitigating differences between domains. A deep network $N$ generates features represented by $f=F(x)$ and classifier prediction denoted by $g=N(F(x))$.

We enhance existing methods for adversarial domain adaptation in two specific ways. Firstly, when dealing with non-identical joint distributions of features and classes across domains, as characterized by $P({x_{s}},{y_{s}})$ and $Q({x_{t}},{y_{t}})$, relying solely on the adaptation of the feature representation $f$ may prove insufficient, as highlighted in [22]. A quantitative analysis indicates that deep representations tend to transition from a more general to a domain-specific nature as they traverse deeper layers within neural networks. This transition leads to a notable decrease in transferability, particularly observed in the layers responsible for domain-specific feature extraction ($f$) and classification ($g$), as detailed in [37]. Secondly, due to the nature of multi-class classification, the feature distribution is multi-modal, and hence adapting feature distribution may be challenging for adversarial networks.

Simultaneous modeling the domain variances in feature representation $(f)$ and classifier prediction $(g)$ facilitates effective domain gap reduction [22]. This joint conditioning helps capture and align data distributions between source and target domains. Thus, incorporating classifier prediction as a conditioning factor in domain adaptation shows promising potential for enhancing transferability and producing domain-invariant representations in challenging cross-domain scenarios. CDAN originally introduced a minimax optimization framework featuring two adversarial loss terms: (a) the source classifier loss, aimed at minimizing it to ensure a lower source risk, and (b) the discriminator loss applied to both the source classifier $N$ and the domain discriminator $D$, spanning both the source and target domains. This loss is minimized with respect to $D$ while simultaneously maximized with respect to $f=F(x)$ and $g=N(F(x))$:

In this context, $L$ corresponds to the cross-entropy loss, $L_{clc}$ is the classifier loss, $L_{dis}$ is the discriminator loss and $h=(f,g)$ signifies the combined variable encompassing the feature representation $f$ and classifier prediction $g$. The minimax game of CDAN is

Here, $\lambda$ denotes a hyper-parameter that balances between the two objectives, allowing for a trade-off between source risk and domain adversary concerns.

As depicted in Figure 1, the domain discriminator $D$ is conditioned on the classifier prediction $g$ via the joint variable $h=(f,g)$, aiming to address the two challenges inherent in adversarial domain adaptation, as discussed in [22]. To incorporate a basic form of conditioning for $D$, we employ $(D(f\oplus g))$—where we concatenate the feature representation and classifier prediction into a vector $(f\oplus g)$, which serves as the input for the conditional domain discriminator $D$. This conditioning approach aligns with the common practice observed in existing conditional GANs [8]. However, when utilizing the concatenation approach, $f$ and $g$ operate independently, missing the opportunity to fully capture the crucial multiplicative interactions between the feature representation and classifier prediction that play a pivotal role in domain adaptation. A multilinear map is formed by computing the outer product of multiple random vectors. This technique, which involves multilinear maps applied to infinite-dimensional nonlinear feature maps, has proven successful in embedding joint or conditional distributions into reproducing kernel Hilbert spaces [31]. In addition to the theoretical advantages offered by the multilinear map $(x\otimes y)$ in comparison to concatenation $(x\oplus y)$, as discussed in [31]. In this research, we harness the capabilities of the multilinear map to condition $D$ on $g$. In contrast to concatenation, the multilinear map, denoted as $(x\otimes y)$, excels in capturing the intricate multi-modal structures that are inherent in complex data distributions. However, it’s important to note that a drawback of the multilinear map is its potential for dimension explosion.

Our approach involves the joint minimization of the source classifier $N$ and feature extractor $F$ with respect to Equation (1). Additionally, we minimize Equation (2) to optimize the domain discriminator $D$ and simultaneously maximize Equation (2) to enhance the feature extractor $F$ and source classifier $N$. This yields the mini-max problem of Domain Adversarial Networks:

In this context, $\lambda$ serves as a hyper-parameter responsible for adjusting the weightage between the source classifier and the conditional domain discriminator and $G$ acts as the generator. Meanwhile, $h=(f,g)$ represents the composite variable encompassing both the domain-specific representation $f$ and the classifier prediction $g$ which play pivotal roles in adversarial adaptation. [10] shows the improvement in performance with use of tailored loss function for medical datasets.

In our Unsupervised Domain Adaptation (UDA) strategy for cross-domain classification, we leverage insights from a labeled source domain to enhance target domain performance, despite unlabeled conditions. Our method utilizes a multi-scale optimized neural architecture, ensuring well-separated, multi-modal class distributions. Data augmentation techniques such as flipping, resizing, and normalization are applied for domain consistency. A novel aspect of our approach is a unique loss function that, combined with selected existing ones, minimizes domain discrepancies and aligns feature distributions across multi-class datasets with varying image sizes. For augmented image feature extraction, we employ ResNet-50 coupled with the Feature Pyramid Network (FPN), blending deep feature capture with multi-scale extraction to effectively represent detailed and broad image features, a pioneering application in UDA and image classification.

To validate the efficacy of our model, we undertake extensive investigations on well-established benchmarks and juxtapose our results with those achieved by state-of-the-art UDA methods. We also studied the impact of using a feature pyramid network (FPN) [20] for domain adaptation for classification. Additionally, we studied how the feature (representation) space of the target domain evolves during training. We also studied the contribution of various components of the loss function. We also studied the convergence speed of our method compared to FixBi [24].

We proposed a novel method for unsupervised domain adaptation for image classification. We proposed a novel neural network architecture and a loss function. Architecturally, we have demonstrated that synergy between two deep learning architectures – ResNet [11] and feature pyramidal network (FPN) [20] – complement each other to extract multi-scale features and effectively separate style (domain) and content (class) information components. Our ablation studies confirm the importance of using FPN with ResNet. The proposed loss component PLMMD and judiciously chosen existing loss components leads to significant improvements in unsupervised domain adaptation (UDA) performance that can surpass the performance of CNNs using other UDA methods. Our ablation study confirmed the importance of each of the loss components. Additionally, using the proposed loss led to faster convergence and a rapid evolution of a class-wise multi-modal distribution of the target domain features.

In the future, computationally heavier architectures, such as, vision transformers [6] and its derivatives may be used for further improvements in domain adaptation. Additionally, the proposed loss function may be adapted for other tasks, such as semantic segmentation and object detection.