Improving the Generalization of Meta-learning on Unseen Domains
via Adversarial Shift

Meta-learning aims to assist the learning process in the new task by studying how learning models perform on each learning task. Recent meta-learning methods can be broadly divided into three categories, metric-based, gradient-based, and model-based methods.

Metric-based methods. Metric-based meta-learning framework can be considered as learning to compare, and a nonparametric similarity function is designed to evaluate the similarity between examples. For example, Matching networks [41] firstly use attention recurrent network as a feature encoder to mapping images from different classes to a common meta-feature space, and applies cosine similarity to obtain the predicted result, Prototypical networks [35] adopt euclidean distance, and RelationNet [37] directly utilizes a deep distance metric to measure the similarity. In general, metric-based methods are simply and effective, however thus far are restricted to classification.

Model-based methods. In this category, meta-learning models are usually designed as a parameterized predictor to generate parameters for the new tasks. For example, Ravi et al. [29] and Santoro et al. [31] both used the recurrent neural network as the predictor.

Gradient-based methods. Gradient-based methods focus on extracting meta-knowledge required to improve the optimization performance. Model-agnostic meta-learning (MAML) [10] regards the initialization of deep network as meta-knowledge and aims to learn a good initialization for all tasks, so that the learner of a new task just needs a few gradient steps from this initialization. R2D2 [4] and MetaOpt [18] adopt ridge regression or support vector machine [7] as the task-specific learner for each learning task, respectively. With these linear classification models, they can learn a feature embedding model which can generalize well on the new task. Compared with the metric-based method, gradient-based method can be applied to many learning problems, e.g. regression and reinforcement learning. However it usually suffers from second-order derivatives, Raghu et al. [27] proposed ANIL to relieve this problem.

There is a substantial body of work on domain adaptation [44], which aims to learn from one or multiple source domains a well-performing model on a target domain. Early methods of domain adaptation generally rely on instance re-weighting [8] or model parameter adaptation [48]. Since the emergence of domain adversarial neural networks (DANN) [11], recent frameworks [19, 47] are mainly based on applying adversarial training to alleviate the domain shift existing in source and target domains. There are also some methods using the discrepancy-based framework to align the marginal distribution between domains [21, 15].

However, most frameworks on domain adaptation are followed two strict priori, i.e., the label spaces of source and target domains are same and numerous unlabeled images in the target domain are available. According to the argumentations in [40, 12], these assumptions may not be realistic and restrict the domain adaptation framework to handle novel concepts. Our works consider the scenario of how to improve the generalization of the learning model on the new concept from unseen domains.

In contrast to domain adaptation, domain generalization methods [3, 33, 23] devote to learn features that perform well when transferred to unseen domains. These models needn’t data from the unseen domains during the training stage. Most recently, meta-learning based approaches [3, 20] are proposed in domain generalization and achieve impressive results. The ideas of these methods are using episodic training to simulate the domain shift during the training and evaluation stages. By this way, better generalization on the unseen domains is achieved. Yet, existing domain generalization approaches still aim at tackling the problem under the assumption that the instances in the training stage share the same label space with the data in the unseen domain. Besides label space, these algorithms also require to access the training instances drawn from different source domains, not a single one. Our method does not have this limitation.

Compared with domain adaptation and generalization, this work studies a more challenging setting, which just requires one single source domain and needs to generalize well on the new concepts from new domains.

Cross-domain few-shot classification is a scenario which our work can be applied to. In cross-domain few-shot classification, training and novel classes are drawn from different domains, and the class label sets are disjoint. This scenario is very difficult, therefore, limited works aim at cross-domain few-shot classification. Typically, Tsing et al. [40] used feature-wise transform to improve the generalization ability of the learned representations, and Guo et al. [12] implemented a broader study of cross-domain few-shot classification and proposed a challenging benchmark. Phoo et al. [25] introduced a self-training approach that allows few-shot learners to adapt feature representations to the target domain by some unlabeled data from the target domain. LRP [36] uses explanation-guided training to improve the performance.

However, all these approaches are tailored for classification. Specifically, some of them are sensitive to the degree of the domain shift or rely on the transductive setting. Our method is model-agnostic and inductive which can be applied to many learning problems.

In the typical few-shot learning, meta-learning model accesses to a set of training tasks $\mathcal{S}^{\text{meta}}=\{\mathcal{T}_{i}\}^{T}_{i}$ drawn from a task distribution $P(\mathcal{T})$. Each task $\mathcal{T}_{i}$ contains a dataset $\mathcal{D}_{i}$, which is usually divided into two disjoint sets: $\mathcal{D}^{\text{tr}}_{i}$ and $\mathcal{D}^{\text{ts}}_{i}$. These datasets each is associated with $K$ data-label pairs, i.e., let $\mathbf{x}\in\mathcal{X}$ and $\mathbf{y}\in\mathcal{Y}$ denote data and their label, respectively, $\mathcal{D}^{\text{tr}}_{i}=\{(\mathbf{x}^{k}_{i},\mathbf{y}^{k}_{i})\}^{K}_{i=1}$, and similarity for $\mathcal{D}^{\text{ts}}_{i}$. In general, $\mathcal{D}^{\text{tr}}_{i}$ and $\mathcal{D}^{\text{ts}}_{i}$ in the same task share the same label space. Different tasks have different label spaces.

We suppose that all the tasks share a same learning algorithm $\mathcal{A}lg$ and a loss function $\mathcal{L}$. Each task $\mathcal{T}_{i}$ has itself learning model (or base-learner) $\mathcal{A}lg_{i}$ parameterized by $\mathbf{w}_{i}\in\mathbb{R}^{d}$. Meta-learning model is interested in learning a meta-learner, e.g., a neural network, from the meta-training set $\mathcal{S}^{\text{meta}}$, which can help the base-learner $\mathcal{A}lg_{j}$ of a new task $\mathcal{T}_{j}$ efficiently learn with a few labeled data. This motivation can be formulated as below:

where $\boldsymbol{\theta}$ denotes the meta-parameter. Particularly, Eq. 1 and Eq. 2 can be solved as a bi-level optimization problem [6].

Note: Metric-based meta-learning method doesn’t have the step of Eq. 2, because the base-learners of these methods are a nonparametric distance function.

In the meta-test stage, when faced a new task $\mathcal{T}_{j}\in P(\mathcal{T})$, the base-learner $\mathcal{A}lg_{j}$ can achieve good performance by using the adaptation procedure of $\mathbf{w}_{j}$ with the learned meta-parameter.

Different from the typical few-shot learning, we address the few-shot problem under the domain generalization setting. In other words, the meta-training set $\mathcal{S}^{\text{meta}}=\{\mathcal{T}_{i}\}^{T}_{i=i}$ is sampled from a seen (source) domain, and the meta-learning model trained on $\mathcal{S}^{\text{meta}}$ is supposed to help new tasks from the other unseen domains learn fast. More specifically, there exists a distribution discrepancy between dataset $\mathcal{D}_{i}$ in the training task $\mathcal{T}_{i}\in\mathcal{S}^{\text{meta}}$ and $\mathcal{D}_{j}$ in the new task $\mathcal{T}_{j}$. Moreover, we can’t access the images in the unseen domain at the training stage.

Firstly, we introduce a Feature-wise Shift Layer (FiSL) which is used to simulate the domain shift in our method. The architecture of FiSL is based on feature-wise transformation [24] which is proven to be capable to represent domain-specific information in many works [9, 24]. In few-shot learning, feature-wise transformation is already adopted to dynamically represent domain-specific [40] and task-specific information [22].

Suppose that the meta-learner contains a feature encoder $f_{\boldsymbol{\theta}}$ with the parameter $\boldsymbol{\theta}\in\Theta$, we are given a feature activation map $\mathbf{z}_{0}\in\mathcal{Z}$ of an image $\mathbf{x}_{0}$ from the last layer of the feature encoder with the dimension of $C\times H\times W$. The output $\mathbf{z}$ of our shift layer is

$\boldsymbol{\gamma}$ and $\boldsymbol{\beta}$ are learnable scaling and shift vectors applied to affine transformation. For easy notation, Eq. 3 can be denoted by $\mathbf{z}=\text{FiSL}(\mathbf{x}_{0})$.

After training, the shift layer with $\boldsymbol{\phi}=\{\boldsymbol{\gamma},\boldsymbol{\beta}\}$ is supposed to be enable to transfer the images from the source domain to other unseen domains.

However, how to train a meta-learning model with FiSL is an intractable problem because of 1. How to make FiSL learn the way of generating pseudo tasks. 2. How to make meta-learning model learn useful domain-invariant meta-knowledge from these tasks.

Experimental Setup. We start with a regression problem of fitting sine curves following [10]. Each task involves regressing from the input to the output of a sine wave. The amplitude and phase of the training tasks are uniformly sampled from $[0.1,3.0]$ and $[0,\frac{3}{4}\pi]$, respectively. However, the amplitude and phase of test tasks are uniformly drawn from $[3.0,5.0]$ and $[\frac{3}{4}\pi,\pi]$, respectively. For training, five labeled datapoints are given for each task as $\mathcal{D}^{\text{tr}}$, and twenty labeled datapoints are sampled as $\mathcal{D}^{\text{ts}}$. Both of them are uniformly sampled from $[-5,5]$. We use a neural network with two hidden layers as the feature encoder and 40 nodes each and a mean-squared error (MSE) as loss $\mathcal{L}$.

Our method is applied to two gradient-based methods: MAML [10] and ANIL [27]. Because, our method is trained by multi-task mechanism, for fairness, the batchsize of tasks to train MAML and ANIL is set to 2. All models are trained $20,000$ iterations by Adam with a learning rate of 0.001. Both MAML and ANIL use one inner gradient step with a learning rate 0.01 and 0.1, respectively. During the test, we present the model with $2,000$ newly sampled tasks from the disjoint domain and measure mean squared error over 100 test points on each task. $\gamma$ and $\eta$ in our method are $0.5$ and $0.01$, respectively.

Results. Table 1 shows the results of different models on the cross-domain few-shot regression. We can see that our method effectively improves the generalization of different gradient-based models on the unseen domain. The results also verify that our method can work well in regression problem. Figure. 2 represents some results of these models on some new tasks from the unseen domain.

In cross-domain few-shot classification, we validate the efficacy of the proposed methods with two categories of meta-learning frameworks, i.e., metric-based and gradient-based frameworks. In metric-based method, we choose ProNet [35] and RelationNet [37]. As for gradient-based method, ANIL, R2D2 [4], and MetaOptNet [18] are chosen. In order to evaluate the performance on unseen domains, we train the few-shot classification model on the mini-ImageNet [41] domain and evaluate the trained model on four different domains: CUB [43], Cars [17], ISIC [39], and ChestX [45]. CUB and Car are two benchmarks which are well established for cross-domain few-shot classification. Evaluating on these two benchmarks can provide a fair comparison to the previous methods. However, the images of these two datasets are natural images that still retain a high degree of visual similarity. Moreover, according to [12], some previous state-of-the-art methods, e.g. [40] are not robust to the large domain shift. Therefore, following [12], we adopt ISIC and Chexst as the other two benchmarks. Some images from these datasets are shown in Figure. 3.

DataSets. We conduct experiments using five datasets: mini-ImageNet, CUB, Cars, ISIC, and ChestX. We follow the same dataset processing in [12, 40]. Compared with natural images in CUB and Car, ISIC and ChestX cover dermoscopic images of skin lesions, and X-ray images respectively, which are largely different to mini-ImageNet. Similar to [40], we select the training iterations with the best accuracy on the validation set of mini-ImageNet for evaluation.

Implementation details. All experiments are performed with ResNet-10 [13] for a fair comparison. Same as [12, 40], we firstly pre-train ResNet-10 by minimizing the standard cross-entropy classification loss on the $64$ training categories in the mini-ImageNet dataset. Then all the models are trained $30,000$ iterations by Adam [16] with a learning rate of $0.001$. Because our method uses multi-task training, the batchsize is set to $2$ for training ProNet, RelationNet, ANIL, R2D2, and MetaOpt. The inner learning rate and updating steps are $0.1$ and $5$ for ANIL. $\gamma$ and $\eta$ in our method are $0.5$ and $0.1$, respectively.

We present the average results over $1,000$ trials for all the experiments, and report the average accuracy and $95\%$ confidence interval. In each trial, the query set $\mathcal{D}^{\text{ts}}$ contains $15$ images.

Generalization with FiSL. Table 2 shows the results of five meta-learning models with FiSL or not on $5$way-$1$ shot and $5$way-$5$shot cross-domain few-shot classification. Both the metric-based and gradient-based models trained with our method perform favorably against the individual baselines. This observation demonstrates that our method is model agnostic and robust to different unseen domains of varying dissimilarity from the source domain. We attribute the improvement of the generalization to the use of FiSL for making the meta-learner really learn the domain-invariant meta-knowledge.

We also obverse that gradient-based methods achieve better generalization than metric-based methods. This might due to the gradient-based methods learn a base learner for the new task with the provided labeled data. The defined metric space learned in the source domain by metric-based methods is not flexible compared with adapting learner on unseen domains.

Comparison to previous state of the arts. Table 4 and Table 5 show the results. All the models are trained on mini-ImageNet and evaluate on the other four benchmarks. First, we obverse GNN+FWT achieves the best performance on CUB, however, degrades on Car. Meanwhile, our method achieves competitive performance on CUB and outperforms GNN-FWT $3.06\%$ and $6.72\%$ on 1-shot and 5-shot on Car, respectively. We think that CUB is a fine-grained bird dataset, which has the highest similarity to the mini-ImageNet in semantic content and color style among these four datasets, as shown in Figure. 3. Some recent few-shot learning methods [2] can even handle this situation with a small domain shift. From this point, learning well enough on mini-ImageNet could provide satisfying performance on CUB.

On ISIC and ChestX, our method achieves state-of-the-art performance and largely outperforms the previous methods. We also obverse FWT can’t improve the performance of ProNet and RelationNet. Some similar results of our method can be found in Table 2. Note that the decline of our methods is less than FWT. In this regard, it’s challenging to improve the generalization of the metric-based meta-learning on some domains existing a large shift.

Moreover, some recent works [38, 5] point out that meta-learning based few-shot learning algorithms underperform compared to the traditional pre-training model when there exists a large shift between base and novel class domains. Comparing the performance of MAML, ProNet with Fixed, we can find the same results in Table 5. However, based on our method, some meta-learning methods can largely outperform Fixed.

According to several recent methods [49, 30], pre-training can significantly improve the performance of meta-learning frameworks on the typical few-shot learning scenario. In this section, we investigate the influence of pre-training on the generalization on unseen domains.

As shown in Table 3, pre-training the feature encoder substantially improves the performance of ProNet and R2D2 on four unseen benchmarks. However, the influence of pre-training on ProNet is not very obvious, when there exists a large domain shift in target domains.