Towards Inheritable Models for Open-Set Domain Adaptation

Notation. Given an input space $\mathcal{X}$ and output space $\mathcal{Y}$, the source and target domains are characterized by the distributions $p$ and $q$ on $\mathcal{X}\times\mathcal{Y}$ respectively. Let $p_{x}$, $q_{x}$ denote the marginal input distributions and $p_{y|x},q_{y|x}$ denote the conditional output distribution of the two domains. Let $\mathcal{C}_{s},\mathcal{C}_{t}\subset\mathcal{Y}$ denote the respective label sets for the classification tasks ($\mathcal{C}_{s}\subset\mathcal{C}_{t}$). In the UODA problem, a labeled source dataset $\mathcal{D}_{s}=\{(x_{s},y_{s}):x_{s}\sim{p_{x}},y_{s}\sim{p_{y|x}}\}$ and an unlabeled target dataset $\mathcal{D}_{t}=\{x_{t}:x_{t}\sim{q_{x}}\}$ are considered. The goal is to assign a label for each target instance $x_{t}$, by predicting the class for those in shared classes ($\mathcal{C}_{t}^{sh}=\mathcal{C}_{s}$), and an ‘unknown’ label for those in unshared classes ($\mathcal{C}_{t}^{uk}=\mathcal{C}_{t}\setminus\mathcal{C}_{s}$). For simplicity, we denote the distributions of target-shared and target-unknown instances as $q^{sh}$ and $q^{uk}$ respectively. We denote the model trained on the source domain as $h_{s}$ (source predictor) and the model adapted to the target domain as $h_{t}$ (target predictor).

Performance Measure. The primary goal of UODA is to improve the performance on the target domain. Hence, the performance of any UODA algorithm is measured by the error rate of target predictor $h_{t}$, i.e. $\xi_{q}(h_{t})$ which is empirically estimated as $\hat{\xi}_{q}(h_{t})=\mathbb{P}_{\{(x_{t},y_{t})\sim q\}}[h_{t}(x_{t})\neq y_{t}]$, where $\mathbb{P}$ is the probability estimated over the instances $\mathcal{D}_{t}$.

Definition 1 (vendor-client paradigm). Consider a vendor with access to a labeled source dataset $\mathcal{D}_{s}$ and a client having unlabeled instances $\mathcal{D}_{t}$ sampled from the target domain. In the vendor-client paradigm, the vendor learns a source predictor $h_{s}$ using $\mathcal{D}_{s}$ to model the conditional $p_{y|x}$, and shares $h_{s}$ to the client. Using $h_{s}$ and $\mathcal{D}_{t}$, the client learns a target predictor $h_{t}$ to model the conditional $q_{y|x}$.

Definition 2 (Inheritability criterion). Let $\mathcal{H}\subseteq\{h~{}|~{}h:\mathcal{X}\rightarrow\mathcal{Y}\}$ be a hypothesis class, $\epsilon>0$, and $\delta\in(0,1)$. A source predictor $h_{s}:\mathcal{X}\rightarrow\mathcal{Y}$ is termed inheritable relative to the hypothesis class $\mathcal{H}$, if a target predictor $h_{t}:\mathcal{X}\rightarrow\mathcal{Y}$ can be learned using an unlabeled target sample $\mathcal{D}_{t}=\{x_{t}~{}:~{}x_{t}\sim{q_{x}}\}$ when given access to the parameters of $h_{s}$, such that, with probability at least $(1-\delta)$ the target error of $h_{t}$ does not exceed that of the best predictor in $\mathcal{H}$ by more than $\epsilon$. Formally,

where, $\xi_{q}(\mathcal{H})=\min_{h\in\mathcal{H}}\xi_{q}(h)$ and $\mathbb{P}$ is computed over the choice of sample $\mathcal{D}_{t}$. This definition suggests that an inheritable model is capable of reliably transferring the task-specific knowledge to the target domain in the absence of the source data, which is necessary for the vendor-client paradigm. Given this definition, a natural question is, how to quantify inheritability of a vendor model for the target task. In the next Section, we address this question by demonstrating the design of inheritable models for UODA.

a) Architecture. As shown in Fig. 2A, the feature extractor $F_{s}$ comprises of a backbone CNN model $M_{s}$ and fully connected layers $E_{s}$. The classifier $G$ contains two sub-modules, a source classifier $G_{s}$ with $|\mathcal{C}_{s}|$ classes, and an auxiliary out-of-distribution (OOD) classifier $G_{n}$ with $K$ classes accounting for the ‘negative’ region not covered by the source distribution (Fig. 3C). The output $\hat{y}_{s}$ for each input $x_{s}$ is obtained by concatenating the outputs of $G_{s}$ and $G_{n}$ (i.e. concatenating $G_{s}(F_{s}(x_{s}))$ and $G_{n}(F_{s}(x_{s}))$) followed by softmax activation. This equips the model with the ability to capture the class-separability knowledge (in $G_{s}$) and to detect OOD instances (via $G_{n}$). This setup is motivated by the fact that the overconfidence issue can be addressed by minimizing the classifier’s confidence for OOD instances [19]. Accordingly, the confidence of $G_{s}$ is maximized for in-distribution (source) instances, and minimized for OOD instances (by maximizing the confidence of $G_{n}$).

b) Dataset preparation. To effectively learn OOD detection, we augment the source dataset with synthetically generated negative instances, i.e. $\mathcal{D}_{n}=\{(u_{n},y_{n}):u_{n}\sim{r_{u}},y_{n}\sim{r_{y|u}}\}$, where $r_{u}$ and $r_{y|u}$ are the marginal latent space distribution and the conditional output distribution of the negative instances respectively. We use $\mathcal{D}_{n}$, to model the low source-density region as out-of-distribution (see Fig. 3C). To obtain $\mathcal{D}_{n}$, a possible approach explored by [19] could be to use a GAN framework to generate ‘boundary’ samples. However, this is computationally intensive and introduces additional parameters for training. Further, we require these negative samples to cover a large portion of the OOD region. This eliminates a direct use of linear interpolation techniques such as mixup [55, 50] which result in features generated within a restricted region (see Fig. 3A). Indeed, we propose an efficient way to generate OOD samples, which we call as the feature-splicing technique.

c) Training procedure. We train the model in two steps. First, we pre-train $\{F_{s},G_{s}\}$ using source data $\mathcal{D}_{s}$ by employing the standard cross-entropy loss,

a) Quantifying inheritability. In presence of a small domain-shift, most of the target-shared instances (pertaining to classes in $\mathcal{C}_{t}^{sh}$) will lie close to the high source-density regions in the latent space (e.g. Fig. 3E). Thus, one can rely on the class-separability knowledge of $h_{s}$ to predict target labels. However, this knowledge becomes less reliable with increasing domain-shift as the concentration of target-shared instances near the high density regions decreases (e.g. Fig. 3D). Thus, the inheritability of $h_{s}$ for the target task would decrease with increasing domain-shift. Moreover, target-unknown instances (pertaining to classes in $\mathcal{C}_{t}^{uk}$) are more likely to lie in the low source-density region than target-shared instances. With this intuition, we define an inheritability metric $w$ which satisfies,

b) Adaptation procedure. For performing adaptation to the target domain, we learn a target-specific feature extractor $F_{t}=\{M_{t},E_{t}\}$ as shown in Fig. 2B (similar in architecture to $F_{s}$). $F_{t}$ is initialized from the source feature extractor $F_{s}=\{M_{s},E_{s}\}$, and is gradually trained to selectively align the shared classes in the pre-classifier space (input to $G$) to avoid negative-transfer. The adaptation involves two processes - inherit (to acquire the class-separability knowledge) and tune (to avoid negative-transfer).

where, $\operatorname{H}$ is the Shannon’s entropy. The total loss $\mathcal{L}_{tune}=\mathcal{L}_{t1}+\mathcal{L}_{t2}$ selectively aligns the shared classes, while avoiding negative-transfer. Thus, the final adaptation loss is,

Result 1. Let $\mathcal{H}$ be a hypothesis class of VC dimension $d$. Let $\mathcal{S}$ be a labeled sample set of $m$ points drawn from $q^{sh}$. If $\widehat{h}_{t}\in\mathcal{H}$ be the empirical minimizer of $\xi_{q^{sh}}$ on $\mathcal{S}$, and $h_{t}^{*}=\operatorname{argmin}_{h\in\mathcal{H}}\xi_{q^{sh}}(h)$ be the optimal hypothesis for $q^{sh}$, then for any $\delta\in(0,1)$, we have with probability of at least $1-\delta$ (over the choice of samples),

a) Datasets. Office-31 [38] consists of 31 categories of images in three different domains: Amazon (A), Webcam (W) and DSLR (D). Office-Home [49] is a more challenging dataset containing 65 classes from four domains: Real World (Re), Art (Ar), Clipart (Cl) and Product (Pr). VisDA [36] comprises of 12 categories of images from two domains: Real (R), Synthetic (S). The label sets $\mathcal{C}_{s}$, $\mathcal{C}_{t}$ are in line with [24] and [39] for all our comparisons. See Suppl. for sample images and further details.

b) Implementation. We implement the framework in PyTorch and use ResNet-50 [12] (till the last pooling layer) as the backbone models $M_{s}$ and $M_{t}$ for Office-31 and Office-Home, and VGG-16 [43] for VisDA. For inheritable model training, we use a batch size of $64$ ($32$ source and negative instances each), and use the hyperparameters $d=15$ and $K=4|\mathcal{C}_{s}|$. During adaptation, we use a batch size of 32 and set the hyperparameter $k=15$. We normalize the instance weights $w(x_{t})$ with the $\operatorname{max}$ weight of each batch $B$, i.e. $w(x_{t})/\max_{x_{t}\in B}w(x_{t})$. During inference, an unknown label is assigned if $\hat{y}_{t}=\operatorname{argmax}_{c_{i}}[\sigma(G(F_{t}(x_{t})))]_{c_{i}}$ is one of the $K$ negative classes, otherwise, a shared class label is predicted. See Supplementary for more details.

c) Metrics. In line with [39], we compute the open-set accuracy (OS) by averaging the class-wise target accuracy for $|\mathcal{C}_{s}|+1$ classes (considering target-unknown as a single class). Likewise, the shared accuracy (OS*) is computed as the class-wise average of target-shared classes ($\mathcal{C}_{t}^{sh}=\mathcal{C}_{s}$).

a) State-of-the-art comparison. In Tables 1-3, we compare against the state-of-the-art UODA method STA [24]. The results for other methods are taken from  [24]. Particularly, in Table 1, we report the mean and std. deviation of OS and OS* over 3 separate runs. Due to space constraints, we report only OS in Table 2. It is evident that adaptation using an inheritable model outperforms prior arts that assume access to both vendor’s data (source domain) and client’s data (target domain) simultaneously. The superior performance of our method over STA is described as follows. STA learns a domain-agnostic feature extractor by aligning the two domains using an adversarial discriminator. This restricts the model’s flexibility to capture the diversity in the target domain, owing to the need to generalize across two domains, on top of the added training difficulties of the adversarial process. In contrast, we employ a target-specific feature extractor ($F_{t}$) which allows the target predictor to effectively tune to the target domain, while inheriting the class-separability knowledge. Thus, inheritable models offer an effective solution for UODA in practice.

b) Hyperparameter sensitivity. In Fig. 4, we plot the adaptation performance (OS) on a range of hyperparameter values used to train the vendor’s model ($K$, $d$). A low sensitivity to these hyperparameters highlights the reliability of the inheritable model. In Fig. 5C, we plot the adaptation performance (OS) on a range of values for $k$ on Office-31. Specifically, $k=0$ denotes the ablation where $\mathcal{L}_{inh}$ is not enforced. Clearly, the performance improves on increasing $k$ which corroborates the benefit of inheriting class-separability knowledge during adaptation.

c) Openness ($\mathbb{O}$). In Fig. 5A, we report the OS accuracy on varying levels of Openness [41] $\mathbb{O}=1-{|\mathcal{C}_{s}|}/{|\mathcal{C}_{t}}|$. Our method performs well for a wide range of Openness, owing to the ability to effectively mitigate negative-transfer.

d) Domain discrepancy. As discussed in [2], the empirical domain discrepancy can be approximated using the Proxy $\mathcal{A}$-distance $\hat{d}_{\mathcal{A}}=2(1-2\epsilon)$ where $\epsilon$ is the generalization error of a domain discriminator. We compute the PAD value at the pre-classifier space for both target-shared and target-unknown instances in Fig. 6B following the procedure laid out in [10]. The PAD value evaluated for target-shared instances using our model is much lower than a source-trained ResNet-50 model, while that for target-unknown is higher than a source-trained ResNet-50 model. This suggests that adaptation aligns the source and the target-shared distributions, while separating out the target-unknown instances.

a) Model inheritability ($\mathcal{I}$). Following the intuition in Sec. 4.2a, we evaluate the model inheritability ($\mathcal{I}$) for the tasks D$\rightarrow$W and A$\rightarrow$W on Office-31. In Fig. 6C we observe that for the target W, an inheritable model trained on the source D exhibits a higher $\mathcal{I}$ value than that trained on the source A. Consequently, the adaptation task D$\rightarrow$W achieves a better performance than A$\rightarrow$W, suggesting that a vendor model with a higher model inheritability is a better candidate to perform adaptation to a given target domain. Thus, given an array of inheritable vendor models, a client can reliably choose the most suitable model for the target domain by measuring $\mathcal{I}$. The ability to choose a vendor model without requiring the vendor’s source data enables the application of the vendor-client paradigm in practice.

b) Instance-level inheritability ($w$). In Fig. 5D, we show the histogram of $w(x_{t})$ values plotted separately for target-shared and target-unknown instances, for the task A$\rightarrow$D in Office-31 dataset. This empirically validates our intuition that the classifier confidence of an inheritable model follows the inequality in Eq. 5, at least for the extent of domain-shift in the available standard datasets.

c) Reliability of $w$. Due to the mitigation of overconfidence issue, we find the classifier confidence to be a good candidate for selecting target sample for pseudo-labeling. In Fig. 5B, we plot the prediction accuracy of the top-$k$ percentile target instances based on target predictor confidence ($\max_{c_{i}\in\mathcal{C}_{s}}[\sigma(G(F_{t}(x_{t})))]_{c_{i}}$). Particularly, the plot for epoch-$0$ shows the pseudo-labeling precision, since the target predictor is initialized with the parameters of the source predictor. It can be seen that the top-15 percentile samples are predicted with a precision close to 1. As adaptation proceeds, $\mathcal{L}_{tune}$ improves the prediction performance of the target model, which can be seen as a rise in the plot in Fig. 5B. Therefore, the bound in Eq. 12 is tightened during adaptation. This verifies our intuition in Sec. 4.3

d) Qualitative results. In Fig. 6A we plot the t-SNE [30] embeddings of the last hidden layer (pre-classifier) features of a target predictor trained using STA [24] and our method, on the task A$\rightarrow$D. Clearly, our method performs equally well in spite of the unavailability of source data during adaptation, suggesting that inheritable models can indeed facilitate adaptation in the absence of a source dataset.

e) Training time analysis. We show the benefit of using inheritable models, over a source dataset. Consider a vendor with a labeled source domain A, and two clients with the target domains D and W respectively. Using the state-of-the-art method STA [24] (which requires labeled source dataset), the time spent by each client for adaptation using source data is 575s on an average (1150s in total). In contrast, our method (a single vendor model is shared with both the clients) results in 250s of vendor’s source training time (feature-splicing: 77s, $K$-means: 66s, training: 154s), and an average of 69s for adaptation by each client (138s in total). Thus, inheritable models provide a much more efficient pipeline by reducing the cost on source training in the case of multiple clients (STA: 1150s, ours: 435s). See Supplementary for experiment details.