Uniform Sampling over Episode Difficulty

We define episodic sampling as subsampling few-shot tasks (or episodes) from a larger base dataset [7]. Assuming the base dataset admits a generative distribution, we sample an episode in two steps¹¹1We use the notation for a probability distribution and its probability density function interchangeably.. First, we sample the episode classes $\mathcal{C}_{\tau}$ from class distribution $p(\mathcal{C}_{\tau})$; second, we sample the episode’s data from data distribution $p(x,y\mid\mathcal{C}_{\tau})$ conditioned on $\mathcal{C}_{\tau}$. This gives rise to the following log-likelihood for a model $l_{\theta}$ parameterized by $\theta$:

where $q(\tau)$ is the episode distribution induced by first sampling classes, then data. In practice, this expectation is approximated by sampling a batch of episodes $\mathcal{B}$ each with their set $\tau_{Q}$ of query samples. To enable transfer to unseen classes, it is also common to include a small set $\tau_{S}$ of support samples to provide statistics about $\tau$. This results in the following Monte-Carlo estimator:

where the data in $\tau_{Q}$ and $\tau_{S}$ are both distributed according to $p(x,y\mid\mathcal{C}_{\tau})$. In few-shot classification, the $n$-way $k$-shot setting corresponds to sampling $n=|\mathcal{C_{\tau}}|$ classes, each with $k=\frac{|\tau_{S}|}{n}$ support data points. The implicit assumption in the vast majority of few-shot methods is that both classes and data are sampled with uniform probability – but are there better alternatives? We carefully examine these underlying assumptions in the forthcoming sections.

We briefly present a few representative episodic learning algorithms. A more comprehensive treatment of few-shot algorithms is presented in Wang et al. [59] and Hospedales et al. [24]. A core question in few-shot learning lies in evaluating (and maximizing) the model likelihood $l_{\theta}$. These algorithms can be divided in two major families: gradient-based methods, which adapt the model’s parameters to the episode; and metric-based methods, which compute similarities between support and query samples in a learned embedding space.

Gradient-based few-shot methods are best illustrated through Model-Agnostic Meta-Learning [15] (MAML). The intuition behind MAML is to learn a set of initial parameters which can quickly specialize to the task at hand. To that end, MAML computes $l_{\theta}$ by adapting the model parameters $\theta$ via one (or more) steps of gradient ascent and then computes the likelihood using the adapted parameters $\theta^{\prime}$. Concretely, we first compute the likelihood $p_{\theta}(y\mid x)$ using the support set $\tau_{S}$, adapt the model parameters, and then evaluate the likelihood:

where $\alpha>0$ is known as the adaptation learning rate. A major drawback of training with MAML lies in back-propagating through the adaptation phase, which requires higher-order gradients. To alleviate this computational burden, Almost No Inner Loop [41] (ANIL) proposes to only adapt the last classification layer of the model architecture while tying the rest of the layers across episodes. They empirically demonstrate little classification accuracy drop while accelerating training times four-fold.

Akin to ANIL, metric-based methods also share most of their parameters across tasks; however, their aim is to learn a metric space where classes naturally cluster. To that end, metric-based algorithms learn a feature extractor $\phi_{\theta}$ parameterized by $\theta$ and classify according to a non-parametric rule. A representative of this family is Prototypical Network [53] (ProtoNet), which classifies query points according to their distance to class prototypes – the average embedding of a class in the support set:

where $d(\cdot,\cdot)$ is a distance function such as the Euclidean distance or the negative cosine similarity, and $\phi_{\theta}^{c}$ is the class prototype for class $c$. Other classification rules include support vector clustering [32], neighborhood component analysis [30], and the earth-mover distance [67].

Given an episode $\tau$ and likelihood function $l_{\theta}$, we define episode difficulty to be the negative log-likelihood incurred on that episode:

which is a surrogate for how hard it is to classify the samples in $\tau_{Q}$ correctly, given $l_{\theta}$ and $\tau_{S}$. By definition, this choice of episode difficulty is tied to the choice of the likelihood function $l_{\theta}$.

Dhillon et al. [11] use a similar surrogate as a means to systematically report few-shot performances. We use this definition because it is equivalent to the loss associated with the likelihood function $l_{\theta}$ on episode $\tau$, which is readily available at training time.

Let us assume that the sampling scheme described in Section˜2.1 induces a distribution $q(\tau)$ over episodes. We call it the proposal distribution, and assume knowledge of its density function. We wish to estimate the expectation in Eq.˜1 when sampling episodes according to a target distribution $p(\tau)$ of our choice, rather than $q(\tau)$. To that end, we can use an importance sampling estimator which simply re-weights the observed values for a given episode $\tau$ by $w(\tau)=\frac{p(\tau)}{q(\tau)}$, the ratio of the target and proposal distributions:

The importance sampling identity holds whenever $q(\tau)$ has non-zero density over the support of $p(\tau)$, and effectively allows us to sample from any target distribution $p(\tau)$.

A practical issue of the IS estimator arises when some values of $w(\tau)$ become much larger than others; in that case, the likelihoods $l_{\theta}(\tau)$ associated with mini-batches containing heavier weights dominate the others, leading to disparities. To account for this effect, we can replace the mini-batch average in the Monte-Carlo estimate of Eq.˜2 by the effective sample size $\text{ESS}(\mathcal{B})$ [29, 34]:

where $\mathcal{B}$ denotes a mini-batch of episodes sampled according to $q(\tau)$. Note that when $w(\tau)$ is constant, we recover the standard mini-batch average setting as $\text{ESS}(\mathcal{B})=|\mathcal{B}|$. Empirically, we observed that normalizing with the effective sample size avoided instabilities. This method is summarized in Algorithm˜1.

A priori, we do not have access to the proposal distribution $q(\tau)$ (nor its density) and thus need to estimate it empirically. Our main assumption is that sampling episodes from $q(\tau)$ induces a normal distribution over episode difficulty. With this assumption, we model the proposal distribution by this induced distribution, therefore replacing $q(\tau)$ with $\mathcal{N}(\Omega_{l_{\theta}}(\tau)\mid\mu,\sigma^{2})$ where $\mu,\sigma^{2}$ are the mean and variance parameters. As we will see in Section˜5.2, this normality assumption is experimentally supported on various datasets, algorithms, and architectures.

We consider two settings for the estimation of $\mu$ and $\sigma^{2}$: offline and online. The offline setting consists of sampling $1,000$ training episodes before training, and computing $\mu,\sigma^{2}$ using a model pre-trained on the same base dataset. Though this setting seems unrealistic, i.e. having access to a pre-trained model, several meta-learning few-shot methods start with a pre-trained model which they further build upon. Hence, for such methods there is no overhead. For the online setting, we estimate the parameters on-the-fly using the model currently being trained. This is justified by the analysis in Section˜5.2 which shows that episode difficulty transfers across model parameters during training. We update our estimates of $\mu,\sigma^{2}$ with an exponential moving average:

where $\lambda\in[0,1]$ controls the adjustment rate of the estimates, and the initial values of $\mu,\sigma^{2}$ are computed in a warm-up phase lasting $100$ iterations. Keeping $\lambda=0.9$ worked well for all our experiments (Section˜5). We opted for this simple implementation as more sophisticated approaches like West [61] yielded little to no benefit.

Similar to the proposal distribution, we model the target distribution by its induced distribution over episode difficulty. Our experiments compare four different approaches, all of which share parameters $\mu,\sigma^{2}$ with the normal model of the proposal distribution. For numerical stability, we truncate the support of all distributions to $[\mu-2.58\sigma,\mu+2.58\sigma]$, which gives approximately $99\%$ coverage for the normal distribution centered around $\mu$.

The first approach (Hard) takes inspiration from hard negative mining [51], where we wish to sample only more challenging episodes. The second approach (Easy) takes a similar view but instead only samples easier episodes. We can model both distributions as follows:

where $\mathcal{U}$ denotes the uniform distribution. The third (Curriculum) is motivated by curriculum learning [2], which slowly increases the likelihood of sampling more difficult episodes:

where $\mu_{t}$ is linearly interpolated from $\mu-2.58\sigma$ to $\mu+2.58\sigma$ as training progresses. Finally, our fourth approach, Uniform, resembles distance weighted sampling [62] and consists of sampling uniformly over episode difficulty:

Intuitively, Uniform can be understood as a uniform prior over unseen test episodes, with the intention of performing well across the entire difficulty spectrum. This acts as a regularizer, forcing the model to be equally discriminative for both easy and hard episodes.

We review the standardized few-shot benchmarks and provide a detailed description in Appendix˜A.

All the models in this subsection are trained using baseline sampling as described in Section˜2.1, i.e., episodic training without importance sampling.

We compare different methods for episode sampling. To ensure fair comparisons, we use the offline formulation (Section˜3.2) so that all sampling methods share the same pre-trained network (the network trained using baseline sampling) when computing proposal likelihoods. We compute results over $2$ datasets, $2$ network architectures, $4$ algorithms and $2$ few-shot protocols, totaling in $24$ scenarios. Table˜1 presents results on ProtoNet (cosine), while the rest are in Appendix˜D.

We observe that, although not strictly dominant, Uniform tends to outperform other methods as it is within the statistical confidence of the best method in $19/24$ scenarios. For the $5/24$ scenarios where Uniform underperforms, it closely trails behind the best methods. Compared to baseline sampling, the average degradation of Uniform is $-0.83\%$ and at most $-1.44\%$ (ignoring the standard deviations) in $4/24$ scenarios. Conversely, Uniform boosts accuracy by as much as $8.74\%$ and on average by $3.86\%$ (ignoring the standard deviations) in $13/24$ scenarios. We attribute this overall good performance to the fact that uniform sampling puts a uniform distribution prior over the (unseen) test episodes, with the intention of performing well across the entire difficulty spectrum. This acts as a regularizer, forcing the model to be equally discriminative for easy and hard episodes. If we knew the test episode distribution, upweighting episodes that are most likely under that distribution will improve transfer accuracy [13]. However, this uninformative prior is the safest choice without additional information about the test episodes.

Second best is baseline sampling as it is statistically competitive on $10/24$ scenarios, while Easy, Hard, and Curriculum only appear among the better methods in $4$, $4$, and $9$ scenarios, respectively.

Although the offline formulation is better suited for analysis experiments, it is expensive as it requires a pre-training phase for the proposal network and $2$ forward passes during episodic training (one for the episode loss, another for the proposal density). In this subsection, we show that the online formulation faithfully approximates offline sampling and can retain most of the performance improvements from Uniform. We take the same $24$ scenarios as in the previous subsection, and compare baseline sampling against offline and online Uniform. Table˜2 reports the full suite of results.

We observe that baseline is statistically competitive on $9/24$ scenarios; on the other hand, offline and online Uniform perform similarly on aggregate, as they are within the best results in $16/24$ and $15/24$ scenarios respectively. Similar to its offline counterpart, online Uniform does better than or comparable to baseline sampling in $21$ out of $24$ scenarios. On the $3/24$ scenarios where online Uniform underperforms compared to baseline, the average degradation is $-0.92\%$, and at most $-1.26\%$ (ignoring the standard deviations). Conversely, in the remaining scenarios, it boosts accuracy by as much as $6.67\%$ and on average by $2.24\%$ (ignoring the standard deviations). Therefore, using online Uniform, while computationally comparable to baseline sampling, results in a boost in few-shot performance; when it underperforms, it trails closely. We also compute the mean accuracy difference between the offline and online formulation, which is $0.07\%\pm 0.35$ accuracy points. This confirms that both the offline and online methods produce quantitatively similar outcomes.

To further validate the role of episode sampling as a way to improve generalization, we evaluate the models trained in the previous subsection on episodes from completely different domains. Specifically, we use the models trained on Mini-ImageNet with baseline and online Uniform sampling to evaluate on the test episodes of CUB-200 [60], Describable Textures [8], FGVC Aircrafts [35], and VGG Flowers [38], following the splits of Triantafillou et al. [57]. Table˜3 displays results for ProtoNet (cosine) and MAML on CUB-200 and Describable Textures, with the complete set of experiments available in Appendix˜F. Out of the 64 total cross-domain scenarios, online Uniform does statistically better in $49/64$ scenarios, comparable in $12/64$ scenarios and worse in only $3/64$ scenarios. These results further go to show that sampling matters in episodic training.

The results in the previous subsections suggest that online Uniform yields a simple and universally applicable method to improve episode sampling. To validate that state-of-the-art methods can also benefit from better sampling, we take the recently proposed FEAT [64] algorithm and augment it with our IS-based implementation of online Uniform. Concretely, we use their open-source implementation⁴⁴4Available at: https://github.com/Sha-Lab/FEAT to train both baseline and online Uniform sampling. We use the prescribed hyper-parameters without any modifications. Results for ResNet-$12$ on Mini-ImageNet and Tiered-ImageNet are reported in Table˜4, where online Uniform outperforms baseline sampling on $3/4$ scenarios and is matched on the remaining one. Thus, better episodic sampling can improve few-shot classification even for the very best methods.