A No-Free-Lunch Theorem for MultiTask Learning

The bulk of theoretical work on multitask and related areas build on early work on domain adaptation (i.e., $N=1$) such as BDBCP (07); CMRR (08); BDBC⁺ (10), which introduce notions of discrepancy such as the $d_{\cal A}$-divergence, $\cal Y$-discrepancy, that specialize the total-variation metric to the setting of domain adaptation. These notions often result in bounds of the form (1), starting with CKW (08); BDBC⁺ (10). Such bounds can in fact be shown to be tight w.r.t. the discrepancy term $\text{disc}\!\left(\left\{P_{t}\right\};{\cal D}\right)$, for given distributional settings and sample sizes, owing for instance to early work on the limits of learning under distribution drift (see e.g. Bar (92)). However, the rates of (1) appear pessimistic when we consider settings of interest here where $h^{\!*}$’s remain the same (or nearly so) across tasks, as they suggest no general improvement on risk with larger sample size. Consider for instance simple situations as depicted on the right, where a source $P$ and target ${\cal D}$ (with respective supports $\mathcal{X}_{P},\mathcal{X}_{\cal D}$) might differ considerably in the mass they assign to regions of data space, thus inducing large discrepancies, but where both assign sufficient mass to decision boundaries to help identify $h^{\!*}$ with enough samples from either distribution. Proper parametrization resolves this issue, as we show through new multitask rates $\mathcal{E}_{\cal D}(\hat{h})\to 0$ in natural situations with $\text{disc}\!\left(\left\{P_{t}\right\};{\cal D}\right)>0$, and even with no target sample.

We remark that other notions of discrepancy, e.g., Maximum Mean Discrepancy GSH⁺ (09), Wasserstein distance RHS (17); SQZY (18) are employed in domain adaptation; however they appear relatively less often in the theoretical literature on multitask and related areas. For multitask, the work of BDB (08) proposes a more-structured notion of task relatedness, whereby a source $P_{t}$ is induced from a target ${\cal D}$ through a transformation of the domain $\mathcal{X}_{\cal D}$; that work also incurs an $n^{-1/2}$ term in the risk bound, but no discrepancy term. The work of MMR (09) considers Rényi divergences in the context of optimal aggregation in multitask under population risk, but does not study sample complexity.

The use of a non-metric discrepancy such as Rényi divergence brings back an important point: two distributions might have asymmetric information on each other w.r.t. domain adaptation. Such insight was raised recently, independently in KM (18); HK (19); APMS (19), with various natural examples therein (see also Section 3.1). In particular, it motivates a more unified view of multitask and multisource domain adaptation, which are often treated separately. Namely, if the goal in multitask is to perform as well as possible on each task $P_{t}$ in our set, then as we show, such asymmetry in information between tasks calls for different aggregation of datasets for each target $P_{t}$: in other words, treating multitask as separate multisource problems, even if the optimal $h^{\!*}$ is the same across tasks.

In contrast, a frequent aim in multitask, dating back to Car (97), has been to ideally arrive at a single aggregation of task datasets that simultaneously benefits all tasks $P_{t}$’s. Following this spirit, many theoretical works on the subject are concerned with bounds on average risk across tasks Bax (97); AZ (05); MPRP (13); PM (13); PL (14); YHC (13); MPRP (16), rather than bounding the supremum risk accross tasks – i.e., treating multitask as separate multisources, as of interest here. Some of these average bounds, e.g., MPRP (13, 16); PM (13), remove the dependence on discrepancy inherent in bounds of the form (1), but maintain a term of the form $n^{-\alpha}$; in other words, any bound on supremum risk derived from such results would be in terms of a single dataset size $n$. The work of BHPQ (17) directly addresses the problem of bounding the supremum risk, but also incurs a term of the form $n^{-\alpha}$ in the risk bound.

In the context of multisource domain adaptation, it has been recognized in practice that some datasets that are too far from the ideal target might hurt target performance and should be downweighted accordingly, a situation that has been termed negative transfer. These situations further motivate the need for adaptive procedures that can automatically identify good datasets. As far as theoretical insights, it is clear that negative transfer might happen for instance, under ERM, if optimal classifiers $h^{\!*}$’s are considerably different across tasks. Interestingly, even when $h^{\!*}$’s are allowed arbitrarily close (but not equal), BDLLP (10) shows that, for $N=1$, the source dataset can be useless without labeled target data. We later derived minimax lower-bounds for the case $N=1$ with or without labeled target data in HK (19) for a range of situations including those considered in BDLLP (10). Such results however do not quite confirm negative transfer, as they allow the possibility that useless datasets might remain safe to include. For the multisource case $N\gg 1$, to the best of our knowledge, situations of negative transfer have only been described in adversarial settings with corrupted labels. For instance, the recent papers of Qia (18); MMM (19); KFAL (20) show limits of multitask under various adversarial corruption of labels in datasets, while SZ (19) derives a positive result, i.e., rates (for Lipschitz loss) decreasing in both $N$ and $n$, up to excluded or downweighted datasets. The procedure of SZ (19) is however nonadaptive as it requires known noise proportions.

KFAL (20) is of particular interest as they are concerned with adaptivity under label corruption. They show that, even if at most $N/2$ of the datasets are corrupted, no procedure can get a rate better than $1/n$. In contrast, in the stochastic setting considered here, there is always an adaptive procedure with significantly faster rates than $1/n$ if at most $N/2$ (in fact any fixed fraction) of distributions $P_{t}$’s are far from the target ${\cal D}$ (Theorem 9 of Appendix B). This dichotomy is due to the strength of adversarial corruptions they consider which in effect can flip optimal $h^{\!*}$’s on corrupted sources. What we show here is that, even when $h^{\!*}$’s are fixed across tasks, and datasets are sampled i.i.d. from each $P_{t}$, i.e., non-adversarially, no algorithm can achieve a rate better than $1/\sqrt{n}$ while a non-adaptive oracle procedure can (see Theorem 5). In other words, some datasets are indeed unsafe for any multisource procedure even in nonadversarial settings, as they can force suboptimal choices w.r.t. a target, absent additional restrictions on the problem setup.

As discussed earlier, such favorable restrictions concern for instance situations where information is available on how sources rank in distance to a target. In particular, the case $n=1$ intersects with the so-called distribution drift setting where each distribution $P_{t},t\in[N]$ has bounded discrepancy $\sup_{t\geq 1}\text{dist}(P_{t},P_{t+1})$ w.r.t. the next distribution $P_{t+1}$ as time $t$ varies. While the notion of discrepancy is typically taken to be total-variation BDBM (89); Bar (92); BL (97) or related notions MM (12); ZL (19); HY (19), both our upper and lower-bounds, specialized to the case $n=1$, provide a new perspective for distribution drift under our distinct parametrization of how $P_{t}$’s relate to each other. For instance, our results on multisource under ranking (Theorem 4) imply new rates for distribution drift, when all $h^{\!*}$’s are the same across time, with excess error at time $N$ going to $0$ with $N$, even in situations where $\sup_{t\geq 1}\text{dist}(P_{t},P_{t+1})>0$ in total variation. Such consistency in $N$ is unavailable in prior work on distribution drift.

Finally we note that there has been much recent theoretical efforts towards other formalisms of relations between distributions in a multitask or multisource scenario, with particular emphasis on distributions sharing common latent substructures, both as applied to classification MBS (13); MB (17); AKK⁺ (19), or to regression settings JSRR (10); LPVDG⁺ (11); NW (11); DHK⁺ (20); TJJ (20). The present work does not address such settings.

Clearly, how well one can do in multitask depend on how distributions relate to each other. The following parametrization will serve to capture the relation between sources and target distributions.

Notice that the above always holds with at least $\rho=\infty$.

We have shown in earlier work HK (19) that the transfer exponent manages to tightly capture the minimax rates of transfer in various situations with a single source $P$ and target ${\cal D}$ ($N=1$), including ones where the best hypotheseses $h^{\!*}_{P},h^{\!*}_{\cal D}$ are different for source and target; in the case of main interest here where $P$ and ${\cal D}$ share a same best in class $h^{\!*}\doteq h^{\!*}_{P}\doteq h^{\!*}_{\cal D}$, for respective data sizes $n_{P},n_{\cal D}$, the best possible excess risk $\mathcal{E}_{\cal D}$ is of order $(n_{P}^{1/\rho}+n_{{\cal D}})^{-1/(2-\beta)}$ which is tight for any values of $\rho$ and $\beta$ (a Bernstein class parameter on $P$ and ${\cal D}$). In other words, $\rho$ captures an effective data size $n_{P}^{1/\rho}$ contributed by the source to the target: this decreases as $\rho\to\infty$, delineating a continuum from easy to hard transfer. Interestingly, $\rho<1$ reveals the fact that source data could be more useful than target data, for instance if the classification problem is easier under the source (e.g., $P$ has more mass at the decision boundary).

Altogether, the transfer exponent $\rho$ appears as a more optimistic measure of discrepancy between source and target, as it reveals the possibility of transfer – even at fast rates – in many situations where traditional measures are pessimistically large. To illustrate this, we recall some of the examples from HK (19). In all examples below, we assume for simplicity that $Y=h^{\!*}(X)$ for some $h^{\!*}\doteq h^{\!*}_{P}\doteq h^{\!*}_{\cal D}$.

Let $h^{\!*}$ be thresholded at $1/2$. We then see that for all $h_{\tau}$ thresholded at $\tau\in[0,1]$, $2P_{X}(h_{\tau}\neq h^{\!*})={\cal D}_{X}(h_{\tau}\neq h^{\!*})$, where for $\tau>1$, $P_{X}(h_{\tau}\neq h^{\!*})=\frac{1}{2}(\tau-1/2)\geq\frac{1}{2}{\cal D}_{X}(h_{\tau}\neq h^{\!*})=\frac{1}{4}$. Thus, the transfer exponent $\rho=1$ with $C_{\rho}=2$, so we have fast transfer at the same rate $1/n_{P}$ as if we were sampling from ${\cal D}$.

On the other hand, recall that the $d_{\cal A}$-divergence takes the form³³3Note that these divergences are often defined w.r.t. every pair $h,h^{\prime}\in\mathcal{H}$ rather than w.r.t. $h^{\!*}$ which makes them smaller. $d_{\cal A}(P,{\cal D})\doteq\sup_{h\in\mathcal{H}}\left|P_{X}(h\neq h^{\!*})-{\cal D}_{X}(h\neq h^{\!*})\right|$, while the $\cal Y$-discrepancy takes the form $d_{\cal Y}(P,{\cal D})\doteq\sup_{h\in\mathcal{H}}\left|\mathcal{E}_{P}(h)-\mathcal{E}_{\cal D}(h)\right|$. The two coincide when $Y=h^{\!*}(X)$.

Now, take $h_{\tau}$ as the threshold at $\tau=1/2$, and $d_{\cal A}=d_{\cal Y}=\frac{1}{4}$ which would wrongly imply that transfer is not feasible at a rate faster than $\frac{1}{4}$; we can in fact make this situation worse, i.e., let $d_{\cal A}=d_{\cal Y}\to\frac{1}{2}$ by letting $h^{\!*}$ correspond to a threshold close to $0$. A first issue is that these divergences get large in large disagreement regions; this is somewhat mitigated by localization – i.e., defining these discrepancies w.r.t. $h$’s in a vicinity of $h^{\!*}$, but does not quite resolve the issue, as discussed in earlier work HK (19).

Suppose $\mathcal{H}$ is the class of one-sided thresholds on the line, $h^{\!*}$ is a threshold at $0$. The marginal ${\cal D}_{X}$ has uniform density $f_{\cal D}$ (on an interval containing $0$), while, for some $\rho\geq 1$, $P_{X}$ has density $f_{P}(\tau)\propto\tau^{\rho-1}$ on $\tau>0$ (and uniform on the rest of the support of ${\cal D}$, not shown). Consider any $h_{\tau}$ at threshold $\tau>0$, we have $P_{X}(h_{\tau}\neq h^{\!*})=\int_{0}^{\tau}f_{P}\propto\tau^{\rho}$, while ${\cal D}_{X}(h_{\tau}\neq h^{\!*})\propto\tau$. Notice that for any fixed $\epsilon>0$, $\lim\limits_{\tau>0,\,\tau\to 0}\frac{{\cal D}_{X}(h_{\tau}\neq h^{\!*})^{\rho-\epsilon}}{P_{X}(h_{\tau}\neq h^{\!*})}=\lim\limits_{\tau>0,\,\tau\to 0}C\frac{\tau^{\rho-\epsilon}}{\tau^{\rho}}=\infty.$

We therefore see that $\rho$ is the smallest possible transfer-exponent. Interestingly, now consider transferring instead from ${\cal D}$ to $P$: we would have $\rho({\cal D}\to P)=1\leq\rho\doteq\rho(P\to{\cal D})$; in other words, there are natural situations where it is easier to transfer from ${\cal D}$ to $P$ than from $P$ to ${\cal D}$, as in the case here where $P$ gives relatively little mass to the decision boundary. This is not captured by symmetric notions of distance, e.g., metrics or semi-metrics such as $d_{\cal A}$, $d_{\cal Y}$, MMD, TV, or Wasserstein.

Finally note that the above examples can be extended to more general hypothesis classes $\mathcal{H}$ as the examples merely play on how fast $f_{P}$ decreases w.r.t. $f_{\cal D}$ in regions of space.

We are now ready to formalize the main class of distributional settings considered in this work.

For notational simplicity, we will often let $P_{N+1}\doteq{\cal D}$. Also, although it is not a parameter of the class, we will also refer to $\rho_{N+1}=1$, as ${\cal D}$ always has transfer exponent $\left(C_{\rho},1\right)$ w.r.t. itself.

Implicit target ${\cal D}={\cal D}(\Pi)$. Every time we write $\Pi$, we will implicitly mean $\Pi=\prod_{t\in[N]}P_{t}^{n_{t}}\times{\cal D}^{n_{N+1}}$, so that in any context where a $\Pi$ is introduced, we may write, for instance, $\mathcal{E}_{{\cal D}}(h)$, which then refers to the ${\cal D}$ distribution in $\Pi$.

Indices and Order Statistics. For any $Z=\left\{Z_{t}\right\}_{t\in[N+1]}\sim\Pi$, and indices $I\subset[N+1]$, we let $Z^{I}\doteq\bigcup_{s\in I}Z_{s}$.

We will often be interested in order statistics $\rho_{(1)}\leq\rho_{(2)}\ldots\leq\rho_{(t)}$ of ordered $\rho_{t},t\in[N+1]$ values, in which case $P_{(t)},Z_{(t)},n_{(t)}$ will denote distribution, sample and sample size at index ${(t)}$. We will then let $Z^{(t)}\doteq Z^{\left\{(1),\ldots,(t)\right\}}$.

Average transfer exponent. For any $t\in[N+1]$, define $\bar{\rho}_{t}\doteq\sum_{s\in[t]}\alpha_{(s)}\cdot\rho_{(s)}$, where $\alpha_{(s)}\doteq\frac{n_{(s)}}{\sum_{r\in[t]}n_{(r)}}$.

Aggregate ERM. For any $Z^{I},I\subset[N+1]$, we let $\hat{h}_{Z^{I}}\doteq\operatorname*{argmin}_{h\in\mathcal{H}}\hat{R}_{Z^{I}}(h)$, and correspondingly we also define $\hat{h}_{Z^{(t)}}$, $t\in[N+1]$ as the ERM over $Z^{(t)}$. When $t=N+1$ we simply write $\hat{h}_{Z}$ for $\hat{h}_{Z^{(N+1)}}$.

Min and Max. We often use the short notations $a\land b\doteq\min\left\{a,b\right\}$, $a\lor b\doteq\max\left\{a,b\right\}$.

Positive Logarithm. For any $x\geq 0$, define $\log(x)\doteq\max\{\ln(x),1\}$.

$\boldsymbol{1/0}$ Convention. We adopt the convention that $1/0=\infty$.

Asymptotic Order. We often write $a\lesssim b$ or $a\asymp b$ in the statements of key results, to indicate inequality, respectively, equality, up to constants and logarithmic factors. The precise constants and logarithmic factors are always presented in supporting results.

We also note that the constraint that $\rho_{t}\geq 1$ is only needed for the lower bound (Theorem 6), whereas all of our upper bounds (Theorems 7, 3, and 4) hold for any values $\rho_{t}>0$. Moreover, there exist classes $\mathcal{H}$ where the lower bound also holds for all $\rho_{t}>0$, so that the form of the bound is generally not improvable. The case $\rho_{t}\in(0,1)$ represents a kind of super transfer, where the source samples are actually more informative than target samples.

It follows from Theorem 1 that, despite there being $2^{N+1}$ possible ways of aggregating datasets (or more if we consider general weightings of datasets), it is sufficient to search over $N+1$ possible aggregations – defined by the ranking $\rho_{(1)}\leq\rho_{(2)}\ldots\leq\rho_{(N+1)}$ – to nearly achieve the minimax rate.

The lower-bound (Theorem 6) relies on constructing a subset (of $\mathcal{M}$) of product distributions $\Pi_{h},h\in\mathcal{H}$, which are mutually close in KL-divergence, but far under the pseudo-metric ${\cal D}_{X}(h\neq h^{\prime})$. For illustration, considering the case $\beta=1$, for any $h,h^{\prime}$ that are sufficiently far under ${\cal D}_{X}$ so that $\mathcal{E}_{{\cal D}}(h^{\prime};h)\gtrsim\epsilon$, the lower-bound construction is such that

ensuring that $\Pi_{h},\Pi_{h^{\prime}}$ are hard to distinguish from finite sample. Thus, the largest $\epsilon$ satisfying the second inequality above, say $\epsilon_{N+1}$ is a minimax lower-bound. On the other-hand, the upper-bound (Theorem 7) relies on a uniform Bernstein’s inequality that holds for non-identically distributed r.v.s (Lemma 1); in particular, by accounting for variance in the risk, such Bernstein-type inequality allows us to extend (to the multisource setting) usual fixed-point arguments that capture the effect of the noise parameter $\beta$. Now, again for illustration, let $\beta=1$, and consider the ERM $\hat{h}\doteq\hat{h}_{Z}$ combining all $N+1$ datasets. Let $\mathcal{E}_{\alpha}\doteq\sum_{t=1}^{N+1}\alpha_{t}\mathcal{E}_{t}(\hat{h})$, $\alpha_{t}\doteq n_{t}/\left(\sum_{s}n_{s}\right)$, then the concentration arguments described above ensure that $\mathcal{E}_{\alpha}(\hat{h})\lesssim 1/\left(\sum_{t}n_{t}\right)$. Now notice that, by definition of $\rho_{t}$, $\mathcal{E}_{\alpha}(\hat{h})\gtrsim\sum_{t}\alpha_{t}\mathcal{E}_{{\cal D}}^{\rho_{t}}(\hat{h})$, in other words, $\mathcal{E}_{{\cal D}}^{\rho_{t}}(\hat{h})$ satisfies the second inequality in (3), and must therefore be at most of order $\epsilon_{N+1}$. This establishes the tightness of $\epsilon_{N+1}$ as a minimax rate, all that is left being to elucidate its exact form in terms of sample sizes. Similar, but somewhat more involved arguments apply for general $\beta$, though in that case we find that pooling all of the data does not suffice to achieve the minimax rate.

Notice that the rates of Theorem 1 immediately imply minimax rates for multitask under the assumption (A1) of sharing a same $h^{\!*}$ (with appropriate $\rho_{t}$’s w.r.t. any target ${\cal D}\doteq P_{s},s\in[N+1]$). It is then natural to ask whether the minimax rate for various targets $P_{s}$ might be achieved by the same algorithm, i.e., the same aggregation of tasks, in light of a common approach in the literature (and practice) of optimizing for a single classifier that does well simultaneously on all tasks. We show that even when all $h^{\!*}$’s are the same, the optimal aggregation might differ across targets $P_{s}$, simply due to the inherent assymmetry of transfer. We have the following theorem, proved in Appendix C.

It is natural to ask whether the above minimax rates for $\mathcal{M}$ are attainable by adaptive procedures, i.e., a reasonable procedure with no access to prior information on (the parameters) of $\mathcal{M}$, but only access to a multisample $Z\sim\Pi$ for some unknown $\Pi\in\mathcal{M}$. As we will see in Section 4.3, this is not possible in general, i.e., outside of the regimes considered here. Our work however leaves open the existence of more refined regimes of adaptivity.

$\bullet$ Low Noise $\beta=1$. To start, when the Bernstein class parameter $\beta=1$ (which would often be a priori unknown to the learner), pooling of all datasets is near minimax optimal as stated in the next result. This corresponds to low noise situations, e.g., so-called Massart’s noise (where $\mathbb{P}(Y\neq h^{\!*}(X)|X)\leq(1/2)-\tau$, for some $\tau>0$), including the realizable case (where $Y=h^{\!*}(X)$ deterministically). Note that $\rho_{t}$’s are nonetheless nontrivial (see examples of Section 3.1), however the distributions $P_{t}$’s are then sufficiently related that their datasets are mutually valuable.

The theorem is proven in Section 7.1. We also state a general bound on $\mathcal{E}_{{\cal D}}(\hat{h}_{Z})$, holding for any $\beta$, in Corollary 2 of Appendix B; the implied rates are not always optimal, though interestingly they are near-optimal in the case that $\sum_{t=1}^{t^{*}}n_{(t)}\propto\sum_{t=1}^{N}n_{t}$, where $t^{*}$ is the minimizer of the r.h.s. in Theorem 1. We note that, unlike in the oracle upper-bounds of Theorem 7, the logarithmic term in the above result is in terms of the entire sample size, rather than the sample size at which the minimizer in $t\in[N+1]$ is attained.

$\bullet$ Available ranking information. Now, assume that on top of $Z\sim\Pi\in\mathcal{M}$, we have access to ranking information $\rho_{(1)}\leq\rho_{(2)}\leq\ldots\leq\rho_{(N+1)}$, but no additional information on $\mathcal{M}$. Namely, $C_{\beta},\beta$, and the actual values of $\rho_{t},t\in[N]$ are unknown to the learner. We show that, in this case, a simple rank-based procedure achieves the minimax rates of Theorem 1, without knowledge of the additional distributional parameters.

Define $\varepsilon(m,\delta)\doteq\frac{d_{\mathcal{H}}}{m}\log\!\left(\frac{m}{d_{\mathcal{H}}}\right)+\frac{1}{m}\log\!\left(\frac{1}{\delta}\right)$, for $\delta\in(0,1)$ and $m\in\mathbb{N}$. Let $\mathbf{n}_{t}\doteq\sum\limits_{s=1}^{t}n_{(s)}$ for each $t\in[N+1]$, ansd recall that $\hat{h}_{Z^{(t)}}$ denotes the ERM over the aggregate sample $Z^{(t)}$.

Rank-based Procedure $\hat{h}$: let $\delta_{t}\doteq\delta/(6t^{2})$, and $C_{0}$ as in Lemma 1. For any $t\in[N+1]$, define:

$$\mathcal{H}_{(t)}\doteq\left\{h\in\mathcal{H}:\hat{\mathcal{E}}_{Z^{(t)}}(h;\hat{h}_{Z^{(t)}})\leq C_{0}\sqrt{\hat{\mathbb{P}}_{Z^{(t)}}(h\neq\hat{h}_{Z^{(t)}})\varepsilon(\mathbf{n}_{t},\delta_{t})}+C_{0}\varepsilon(\mathbf{n}_{t},\delta_{t})\right\},$$

(4)

Return any $h$ in $\bigcap_{s=1}^{N+1}\mathcal{H}_{(s)}$, if not empty, otherwise return any $h$.

We have the following result for this learning algorithm.

The result is shown in Section 7.2 by arguing that each $h$ in $\mathcal{H}_{(t)}$ has ${\mathcal{E}_{{\cal D}}(h)}$ of order $\left(\sum_{s\in[t]}n_{(s)}\right)^{-1/(2-\beta)\bar{\rho}_{t}}$, so the minimizer is attained whenever $\hat{t}=N+1$ (which is probable, as $h^{\!*}$ is likely to be in each $\mathcal{H}_{(t)}$).

Recalling our earlier discussion of Section 1, the above result applies to the case of drifting distributions by letting $n_{t}=1$, and $(t)=N+2-t$, i.e., the $t^{{\rm th}}$ previous example is considered the $t^{{\rm th}}$ most-relevant to the target ${\cal D}$. In this case, in contrast to the lower bounds proven by Bar (92), the above result of Theorem 4 reveals situations where the risk at time $N$ approaches $0$ as $N\to\infty$, even though the total-variation distance between adjacent distributions may be bounded away from zero (see examples of Section 3.1). In other words, by constraining the sequence of $P_{t}$ distributions by the sequence of $\rho_{t}$ values, we can describe scenarios where the traditional upper bound analysis of drifting distributions from Bar (92); BDBM (89); BL (97) can be improved.

Adaptivity in general is not possible even though the rates of Theorem 1 appear to reduce the exponential search space over aggregations to a more manageable one based on rankings of data. As seen in the previous section, easier settings such as ones with ranking information, or low noise, allow adaptivity to the remaining unknown distributional parameters. The result below states that, outside of these regimes, no algorithm can guarantee a rate better than a dependence on the number of samples $n_{\cal D}$ from the target task, even when a semi-adaptive procedure using ranking information can achieve any desired rate $\epsilon$.

The first part of the result follows from Theorem 8 of Section 8, while the second part follows from both Theorem 8 and Theorem 4 of Section 8. The main idea of the proof is to inject enough randomness into the choice of ranking, while at the same time allowing bad datasets from distributions with large $\rho_{t}$ – which would force a wrong choice of $h^{\!*}$ – to appear as benign as good samples from distributions with small $\rho_{t}=1$. Hence, we let $N$ large enough in our constructions so that the bulk of bad datasets would significantly overwhelm the information from good datasets.

As a technical point, no knowledge of $\mathcal{M}$ simply means that a minimax analysis is performed over a larger family containing such $\mathcal{M}$’s, indexed over choices of ranking each corresponding to a fixed $\mathcal{M}$.

Finally, we note that the result leaves open the possibility of adaptivity under further distributional restrictions on $\mathcal{M}$, for instance requiring that the number of samples $n$ per source task be large w.r.t. other parameters such as $\beta$ and $N$; although this remains unclear, large values of $n$ could perhaps yield sufficient information to compare and rank tasks.

Another possible restriction towards adaptivity is to require that a large proportion of the samples are from good datasets w.r.t. $N+1$. In particular, we show in Theorem 9 of Appendix B that the ERM $\hat{h}_{Z}$ which pools all datasets achieves a target excess risk $\mathcal{E}_{{\cal D}}(\hat{h}_{Z})$ depending on the (weighted) median of the $\rho_{t}$ values (or more generally, any quantile); in other words as long as a constant fraction of all datapoints (pooled from all datasets) are from tasks with relatively small $\rho_{t}$, the bound will be small. However, this is not a safe algorithm in general, as per Theorem 2.