Domain-shift adaptation via linear transformations

Without loss of generality, we assume that $E[Z]=0$ and $Cov[Z]=I$. Then, given a dataset with instances drawn from the source domain $X_{A}$ and a dataset with instances drawn from the target domain $X_{B}$:

Note that we can compute the empirical estimates $\hat{\mu}_{A},\hat{\mu}_{B},\hat{\Sigma}_{A},\hat{\Sigma}_{B}$, given the datasets $X_{A}$ and $X_{B}$. The empirical estimators of the mean directly give us half of the transformation parameters. For the case of the covariance matrix, we can compute the singular value decomposition:

Since $\Sigma$ is a positive semi-definite matrix, its eigenvalues are non-negative, which allows us to decompose the diagonal matrix as $S=S^{\frac{1}{2}}S^{\frac{1}{2}}$. By comparing Equations 3 and 4, we can estimate the parameters $\theta$ as:

A consequence of Equation 6 is that matching the empirical mean and covariance matrices of the source and target domain is not enough to correct for domain-shift adaptation: The orthogonal matrix $Q$, which represents rotations or reflections, might cause misalignment in the data; see Figure 4(a).

The matrices $(S_{A},U_{A})$ and $(S_{B},U_{B})$, for the source and target domains, respectively, can be computed from the SVD of their empirical covariance matrices, $\hat{\Sigma}_{A}$ and $\hat{\Sigma}_{B}$. Since the objective of domain adaptation is to align the distributions, regardless of the “direction” of the alignment, we arbitrarily set $Q_{A}=I$. Then, we find an orthogonal matrix that minimizes the divergence between both probability distributions:

A common measure of the divergence between two probability distributions is the Maximum Mean Discrepancy (MMD) Gretton et al. (2012).

Informally, the purpose of the MMD is to determine if two probability distributions, $p$ and $q$, are different. The associated algorithm involves taking samples from $p$ and $q$, then finding a function that take large values on samples from $p$ and small (or negative) values on samples of $q$. The MMD is then the difference between the mean values of the function of the samples.

By defining the class of functions $\mathcal{F}$ as the unit ball in a reproducing kernel Hilbert space, Gretton et al. (2012) proposed (biased) empirical estimator of the $\mbox{MMD}^{2}$ as follows:

Equation 8 has two nice properties. (1) It computes an estimation of the MMD with a finite number of instances from each domain. (2) It is a differentiable function, so it can be optimized with iterative methods, such as gradient descent.

For solving the optimization problem with orthogonality constraints presented in Equation 7, we used the Wen and Yin (2013) algorithm (Algorithm 1), which is an iterative method based on the Cayley transform. Their algorithm is similar to gradient descent, but instead of looking for solutions in the Euclidean space, they look for solutions in the Stiefel manifold, which is the set that contains all the orthogonal matrices.

Formally, their proposed algorithm solves:

Algorithm 1 is guaranteed to converge when the learning rate ($\tau$) meets the Armijo-Wolfe conditions Nocedal and Wright (2006). However, it is not guaranteed to find the global minimum of $\mathcal{F}(X)$. Similarly to gradient descent approaches, the algorithm might converge to a local minimum. One heuristic to alleviate this problem is to perform multiple restart with different seed points; however, this is still not guaranteed to convergence to the global minimum.

For finding the parameters $\theta_{A}$ and $\theta_{B}$, we can arbitrarily set $Q_{A}=I$ in Equation 5, and then use Algorithm 1, with the Maximum Mean Discrepancy, for computing $Q_{B}$:

Note that Algorithm 1 requires the gradient of the $\mbox{MMD}^{2}$ with respect to the matrix $Q_{B}^{T}$. By applying standard matrix calculus we compute (see Appendix A.3 for details):

Algorithm 2 shows the procedure to map the source and target domain into a common space in an unsupervised way (The code is publicly available; see Appendix B ). For notation, the source domain (resp., target, shared space) is $m$-dimensional space, (resp. $n$-dimensional, $p$-dimensional; here we assume that $p\leq\min(m,n)$. For mapping into this lower dimensional space, we project the data into the first $p$ eigenvectors of the empirical covariance matrice $\Sigma_{A}$ (resp. $\Sigma_{B}$). The eigenvalues corresponding these eigenvectors are positive, while the other $m-p$ and $n-p$ eigenvalues will be equal to zero.

After mapping both domains to a common space, we can use the labels of the source domain to learn a predictor, and then use it to make predictions in the target domain. Section 4 will show that the Maximum Mean Discrepancy is not convex with respect to the orthogonal matrix $Q_{B}^{T}$, meaning Algorithm 1 might converge to a local minimum. Additionally, in the unsupervised case there is an inherent identifiability problem caused by the missing labels David et al. (2010); Koller and Friedman (2009) – i.e., there are $\theta_{B}$ and $\theta_{B}^{\prime}$ such that $P(\ f(X,\theta_{B})\ )=P(\ f(X,\theta_{B}^{\prime})\ )$. This can create an “anti-alignment” problem; see Figure 3.

Note than when the “anti-alignment” occurs, the source and target domains have the same marginal probability $P_{S}(Z)=P_{T}(Z)$, but $P_{S}(Y\ |\ Z)\neq P_{T}(Y\ |\ Z)$. In other words, a classifier learned on the source domain will have good performance on more data from the same domain; however, it will have very poor performance on the target domain. Zhao et al. (2019) shows that aligning the marginal probability of the covariates, then learning a good predictor on the source domain, is not sufficient for successfully performing domain adaptation.

If we have access to a few labeled instances in the target domain, we might reduce the chance of converging to an “anti-alignment”. Since the MMD is not convex with respect to the rotation matrix $Q_{B}^{T}$, a common strategy is to attempt multiple re-start (with different seed points) of an iterative optimization algorithm. We then choose the one with the lowest cost. In the unsupervised case, the MMD itself is the cost function. For the semi-supervised case, we can first run Algorithm 2 for each seed point, then learn a predictor, using only the labeled data from the source domain. Then, for every alignment generated by each of the seed points, evaluate those predictors on the labeled data of the target domain, and choose the one with the lowest error.

Alternatively, we could incorporate the cross entropy loss of the source and target domains into Equation 10. This approach requires optimizing a weighted linear combination of three terms in the loss function: the MMD, the cross-entropy in the source domain, and the cross-entropy in the target domain. Since this path requires setting these three extra weights, we limited our experiments to the first approach.

For the simulated data we sampled 600 instances to creaet a dataset, $D_{z}\in\mathbb{R}^{600\times 2}$, from a mixture of bi-variate Gaussians with parameters $\mu_{1}=\begin{bmatrix}1\\ 1\end{bmatrix}$, $\mu_{2}=\begin{bmatrix}5\\ -5\end{bmatrix}$, $\Sigma_{1}=\begin{bmatrix}2&0.7\\ 0.7&1\end{bmatrix}$, $\Sigma_{2}=\begin{bmatrix}2&1\\ 1&4\end{bmatrix}$. These instances correspond to a common shared space $Z$. We then created two random transformation matrices $\theta_{A},\theta_{B}\in\mathbb{R}^{5\times 2}$, and two random translation vectors $\mu_{A},\mu_{B}\in\mathbb{R}^{5}$ to create the observations. Of course, neither the real parameters, nor the instances in the shared space are visible to our algorithm.

We randomly divided $D_{z}$ into two disjoint datasets $D_{A}$ (source domain) and $D_{B}$ (target domain) with 300 instances each. Then, we created the datasets $X_{A}=D_{A}\theta_{A}^{T}+\mu_{A}$ and $X_{B}=D_{B}\theta_{B}^{T}+\mu_{B}$. Our algorithm only sees $X_{A}$ and $X_{B}$, which each contain 5-dimensional vectors.

Figure 4(b) shows the result of applying Algorithm 2 to the simulated datasets $X_{A}$ and $X_{B}$. Figure 4(a), on the other hand, shows the effect of ignoring the effect of the orthogonal matrix $Q_{B}^{T}$. In this last case we successfully mapped both datasets into the same lower-dimensional space, and that both datasets have zero mean and an identity covariance matrix; however, they are not aligned. By finding the orthogonal matrix that minimizes the MMD between the source and target datasets we can obtain the correct alignment.

As mentioned in Section 3.4, the maximum mean discrepancy is not convex with respect to the orthogonal matrix $Q_{B}^{T}$. For 2-dimensional spaces, an orthogonal matrix is either a rotation $R=\begin{bmatrix}\cos\alpha&-\sin\alpha\\ \sin\alpha&\cos\alpha\end{bmatrix}$ or a reflection $S=\begin{bmatrix}-\cos\alpha&-\sin\alpha\\ -\sin\alpha&\cos\alpha\end{bmatrix}$ Winter (1992). Figure 5 shows the MMD between the projection of $X_{A}$ into $Z_{A}$ and the rotated (or reflected) projection of $X_{B}$ into $Z_{B}$ at different angles. Note that we have a total of 4 local minima for the simulated data. The global minimum corresponds to the proper alignment, shown in Figure 4(b). Similar to gradient descent, Algorithm 2 might converge to a local minimum depending on the seed point.

The second experiment is a variation of the digit classification task with the dataset MNIST (source domain) LeCun et al. (1998) and USPS (target domain) Hull (1994). We simplified the task from 10-class digit classification to 45 binary digit classification (0 vs 1, 0 vs 2, … , 8 vs 9).

We first trained a 10-class convolutional neural network on the training data of the source domain (60,000 images) to create image embeddings in a 20-dimensional space. The convolutional neural network contained 4 convolutional layers (32, 128, 256 and 512 filters, respectively), each followed by a MaxPooling layer ($2\times 2$ kernel). Then we added a fully connected layer with 20 hidden neurons, and finally the output layer with 10 output neurons. While the output layer used a softmax activation function, the other layers used a rectified linear unit (ReLU) as the activation function; see Appendix B.

We used the output of the fully connected layer with 20 neurons as the image embeddings for both, the training data of the MNIST (60,000 images) and the test data USPS datasets (2,007 images). Then, for each of the 45 binary classification tasks, we compared three scenarios: (1) Baseline: learn the parameters of a logistic regression model using the MNIST dataset, and then test it on the USPS dataset. (2) Use Algorithm 2 to project the MNIST and USPS dataset into a common space of dimension 5. Then learn the parameters of a logistic regression model using only the projected data of the MNIST, and test it on the projected data of the USPS dataset. We chose a “small” dimensionality of the shared space because computing distances in high-dimensional spaces is harder because the instances sparsely populate the input space Friedman et al. (2001). (3) Similar to the second scenario, but now assume that 10% of the data in the USPS dataset is labelled (semi-supervised case).

Figure 6(a) shows the classification accuracy of the unsupervised domain adaptation approach. The number in parenthesis indicates the improvement (or decrease) in performance relative to the Scenario 1. Similarly, Figure 6(b) shows the classification accuracy of the semi-supervised experiment.