Estimator selection in the Gaussian setting

We consider the Gaussian regression framework

where $f=(f_{1},\ldots,f_{n})$ is an unknown vector of ${\mathbb{R}}^{n}$ and the ${\varepsilon}_{i}$ are independent centered Gaussian random variables with common variance $\sigma^{2}$. Throughout the paper, $\sigma^{2}$ is assumed to be unknown which corresponds to the practical case. Our aim is to estimate $f$ from the observation of $Y$. For specific forms of $f$, this setting allows to deal simultaneously with the following problems.

Our estimation strategy is based on estimator selection. More precisely, we start with an arbitrary collection ${\mathbb{F}}=\{\widehat{f}_{\lambda},\ \lambda\in\Lambda\}$ of estimators of $f$ based on $Y$ and aim at selecting the one with the smallest Euclidean risk by using the same observation $Y$. The way the estimators $\widehat{f}_{\lambda}$ depend on $Y$ may be arbitrary and possibly unknown. For example, the $\widehat{f}_{\lambda}$ may be obtained from the minimization of a criterion, a Bayesian procedure or the guess of some experts.

The problem of choosing some best estimator among a family of candidate ones is central in Statistics. Let us present some examples.

Another dilemma for Statisticians is the choice of a procedure to solve a given problem. In the context of Example 3, there exist many competitors to the Lasso estimator and one may alternatively choose a procedure based on ridge regression (see Hoerl and Kennard (1970)), random forest or PLS (see Tenenhaus (1998), Helland (2001) and Helland (2006)). Similarly, for the problem of signal denoising as described in Example 1, popular approaches include spline smoothing, wavelet decompositions and kernel estimators. The choice of a kernel may be possibly tricky.

A common way to address the above issues is to use some cross-validation scheme such as leave-one-out or $V$-fold. Even though these resampling techniques are widely used in practice, little is known on their theoretical performances. For more details, we refer to Arlot and Celisse (2010) for a survey on cross-validation technics applied to model selection. Compared to these approaches, as we shall see, the procedure we propose is less time consuming and easier to implement. Moreover, it does not require to know how the estimators depend on the data $Y$ and we can therefore handle the following problem.

Given a selection rule among ${\mathbb{F}}$, an important issue is to compare the risk of the selected estimator to those of the candidate ones. Results in this direction are available in the context of model selection, which can be seen as a particular case of estimator selection. More precisely, for the purpose of selecting a suitable model one starts with a collection $\SS$ of those, typically linear spaces chosen for their approximation properties with respect to $f$, and one associates to each model $S\in\SS$ a suitable estimator $\widehat{f}_{S}$ with values in $S$. Selecting a model then amounts to selecting an estimator among the collection ${\mathbb{F}}=\{\widehat{f}_{S},\ S\in\SS\}$. For this problem, selection rules based on the minimization of a penalized criterion have been proposed in the regression setting by Yang (1999), Baraud (2000), Birgé and Massart (2001) and Baraud et al (2009). Another way, usually called Lepski’s method, appears in a series of papers by Lepski (1990; 1991; ; ) and was originally designed to perform model selection among collections of nested models. Finally, we mention that other procedures based on resampling have interestingly emerged from the work of Arlot (2007; 2009) and Célisse (2008). A common feature of those approaches lies in the fact that the proposed selection rules apply to specific collections of estimators only.

An alternative to estimator selection is aggregation which aims at designing a suitable combination of given estimators in order to outperform each of these separately (and even the best combination of these) up to a remaining term. Aggregation techniques can be found in Catoni (1997; 2004), Juditsky and Nemirovski (2000), Nemirovski (2000), Yang (),  (),  (2001), Tsybakov (2003), Wegkamp (2003), Birgé (2006), Rigollet and Tsybakov (2007), Bunea, Tsybakov and Wegkamp (2007) and Goldenshluger (2009) for ${\mathbb{L}}_{p}$-losses. Most of the aggregation procedures are based on a sample splitting, one part of the data being used for building the estimators, the remaining part for selecting among these. Such a device requires that the observations be i.i.d. or at least that one has at disposal two independent copies of the data. From this point of view our procedure differs from classical aggregation procedures since we use the whole data $Y$ to build and select. In the Gaussian regression setting that is considered here, we mention the results of Leung and Barron (2006) for the problem of mixing least-squares estimators. Their procedure uses the same data $Y$ to estimate and to aggregate but requires the variance to be known. Giraud (2008) extends their results to the case where it is unknown.

Our approach for solving the problem of estimator selection is new. We introduce a collection $\SS$ of linear subspaces of ${\mathbb{R}}^{n}$ for approximating the estimators in ${\mathbb{F}}$ and use a penalized criterion to compare them. As already mentioned and as we shall see, this approach requires no assumption on the family of estimators at hand and is easy to implement, an R-package being available on

http://w3.jouy.inra.fr/unites/miaj/public/perso/SylvieHuet_en.html.

A general way of comparing estimators in various statistical settings has been described in Baraud (2010). However, the procedure proposed there is mainly abstract and inadequate in the Gaussian framework we consider.

We prove a non-asymptotic risk bound for the estimator we select and show that this bound is optimal in the sense that it essentially cannot be improved (except for numerical constants maybe) by any other selection rule. For the sakes of illustration and comparison, we apply our procedure to various problems among which aggregation, model selection, variable selection and selection among linear estimators. In each of these cases, our approach allows to recover classical results in the areas as well as to establish new ones. In the context of aggregation we compute the aggregation rates for the unknown variance case. These rates turn out to be the same as those for the known variance case. For selecting an estimator among a family of linear ones, we propose a new procedure and establish a risk bound which requires almost no assumption on the considered family. Finally, our approach provides a way of selecting a suitable variable selection procedure among a family of candidate ones. It thus provides an alternative to cross-validation for which little is known.

The paper is organized as follows. In Section 2 we present our selection rule and the theoretical properties of the resulting estimator. For illustration, we show in Sections 3, 4 and 5 respectively, how the procedure can be used to aggregate preliminary estimators, select a linear estimator among a finite collection of candidate ones, or solve the problem of variable selection. Section 6 is devoted to two simulation studies. One aims at comparing the performance of our procedure to the classical $V$-fold in view of selecting a tuning parameter among a grid. In the other, we evaluate the performance of the variable selection procedure we propose to some classical ones such as the Lasso, random forest, and others based on ridge and PLS regression. Finally, the proofs are postponed to Section 7.

Throughout the paper $C$ denotes a constant that may vary from line to line.

Given a collection ${\mathbb{F}}=\{\widehat{f}_{\lambda},\lambda\in\Lambda\}$ of estimators of $f$ based on $Y$, the selection rule we propose is based on the choices of a family $\SS$ of linear subspaces of ${\mathbb{R}}^{n}$, a collection $\{\SS_{\lambda},\ \lambda\in\Lambda\}$ of (possibly random) subsets of $\SS$, a weight function $\Delta$ and a penalty function $\mathop{\rm pen}\nolimits$, both from $\SS$ into ${\mathbb{R}}_{+}$. We introduce those objects below and refer to Sections 3, 4 and 5 for examples.

For all $\lambda\in\Lambda$ let us set

This quantity corresponds to an accuracy index for the estimator $\widehat{f}_{\lambda}$ with respect to the family $\SS_{\lambda}$. The following result holds.

Let us now comment Theorem 1.

It turns out that inequality (10) leaves no place for a substantial improvement in the sense that the bound we get is essentially optimal and cannot be improved (apart from constants) by any other selection rule among ${\mathbb{F}}$. To see this, let us assume for simplicity that ${\mathbb{F}}$ is finite so that a measurable minimizer of ${\rm crit}_{\alpha}$ always exists and $\delta$ can be chosen as 0. Let $K=1.1$, $\alpha=1/2$ (to fix up the ideas), $\SS$ a family of linear spaces satisfying the assumptions of Theorem 1 and $\mathop{\rm pen}\nolimits$, the penalty function achieving equality in (5). Besides, assume that $\SS$ contains a linear space $S$ such that $1\leq\dim(S)\leq n/2$ and associate to $S$ the weight $\Delta(S)=\dim(S)$. If $\SS_{\lambda}=\SS$ for all $\lambda$, we deduce from (4) and (10) that for some universal constant $C^{\prime}$, whatever ${\mathbb{F}}$ and $f\in{\mathbb{R}}^{n}$

In the opposite direction, the following result holds.

We see that, up to the estimator $\widehat{\sigma}_{S}^{2}$ in place of $\sigma^{2}$ and numerical constants, the left-hand sides of (14) and (15) coincide.

In view of commenting (11) further, we continue assuming that ${\mathbb{F}}$ is finite so that we can keep $\delta=0$ in (11). A particular feature of (11) lies in the fact that the risk bound pays no price for considering a large collection ${\mathbb{F}}$ of estimators. In fact, it is actually decreasing with respect to ${\mathbb{F}}$ (or equivalently $\Lambda$) for the inclusion. This means that if one adds a new estimator to the collection ${\mathbb{F}}$ (without changing neither $\SS$ nor the families $\SS_{\lambda}$ associated to the former estimators), the risk bound for $\widehat{f}_{\widehat{\lambda}}$ can only be improved. In contrast, the computation of the estimator $\widehat{f}_{\widehat{\lambda}}$ is all the more difficult that $\left|{{\mathbb{F}}}\right|$ is large. More precisely, if the cardinalities of the families $\SS_{\lambda}$ are not too large, the computation of $\widehat{f}_{\widehat{\lambda}}$ requires around $\left|{{\mathbb{F}}}\right|$ steps.

The selection rule we use does not require to know how the estimators depend on $Y$. In fact, as we shall see, a more important piece of information is the ranges of the estimators $\widehat{f}_{\lambda}=\widehat{f}_{\lambda}(Y)$ as $Y$ varies in ${\mathbb{R}}^{n}$. A situation of special interest occurs when each $\widehat{f}_{\lambda}$ belongs to some (possibly random) linear space $\widehat{S}_{\lambda}$ in $\SS$ with probability one. By taking $\SS_{\lambda}$ such that $\widehat{S}_{\lambda}\in\SS_{\lambda}$ for all $\lambda$, we deduce from Theorem 1 by using (11) and (13) the following corollary.

One may apply this result in the context of model selection. One starts with a collection of models $\SS=\left\{{S_{m},\ m\in\mathcal{M}}\right\}$ and associate to each $S_{m}$ an estimator $\widehat{f}_{m}$ with values in $S_{m}$. By taking ${\mathbb{F}}=\{\widehat{f}_{m},\ m\in\mathcal{M}\}$ (here $\Lambda=\mathcal{M}$) and $\SS_{m}=\left\{{S_{m}}\right\}$ for all $m\in\mathcal{M}$, our selection procedure leads to an estimator $\widehat{f}_{\widehat{m}}$ which satisfies

When $\widehat{f}_{m}=\Pi_{S_{m}}Y$ for all $m\in\mathcal{M}$, our selection rule becomes

and turns out to coincide with that described in Baraud et al (2009). Interestingly, Corollary 1 shows that this selection rule can still be used for families ${\mathbb{F}}$ of (non-linear) estimators of the form $\Pi_{S_{\widehat{m}}}Y$ where the $S_{\widehat{m}}$ are chosen randomly among $\SS$ on the basis of $Y$, doing thus as if the linear spaces $S_{\widehat{m}}$ were non-random. An estimator of the form $\Pi_{S_{\widehat{m}}}Y$ can be interpreted as resulting from a model selection procedures among the family of projection estimators $\{\Pi_{m}Y,\ m\in\mathcal{M}\}$ and hence, (18) can be used to choose some best model selection rule among a collection of candidate ones.

Consider now three estimators $\widehat{f}_{{\rm L}},\widehat{f}_{{\rm MS}},\widehat{f}_{{\mathrm{Cv}}}$ with values respectively in $S_{\{1,\ldots,M\}}$, $\bigcup_{j=1}^{M}S_{\{j\}}$ and the convex hull ${\mathcal{C}}$ of the $\phi_{j}$ (we use a new notation for this convex hull to avoid ambiguity). One may take the estimators defined in Section 3.1 but any others would suit. The aim of this section is to select the one with the smallest risk to estimate $f$. To do so, we apply our selection procedure with ${\mathbb{F}}=\{\widehat{f}_{{\rm L}},\widehat{f}_{{\rm MS}},\widehat{f}_{{\mathrm{Cv}}}\}$, taking thus $\Lambda=\left\{{{\rm L},{\rm MS},{\mathrm{Cv}}}\right\}$, and associate to each of these three estimators the families $\SS_{{\rm L}},\SS_{{\rm MS}},\SS_{{\mathrm{Cv}}}$ defined by (21), (22) and (25) respectively and choose $\SS=\SS_{{\rm L}}\cup\SS_{{\rm MS}}\cup\SS_{{\mathrm{Cv}}}$.

To apply our selection procedure, we need to associate to each $A_{\lambda}$ a suitable collection of approximation spaces $\SS_{\lambda}$. To do so, we introduce below a linear space $S_{\lambda}$ which plays a key role in our analysis.

For the sake of simplicity, let us first consider the case where $A_{\lambda}$ is non-singular. Then $S_{\lambda}$ is defined as the linear span of the right-singular vectors of $A^{-1}_{\lambda}-I$ associated to singular values smaller than 1. When $A_{\lambda}$ is symmetric, $S_{\lambda}$ is merely the linear span of the eigenvectors of $A_{\lambda}$ associated to eigenvalues not smaller than 1/2. If none of the singular values are smaller than 1, then $S_{\lambda}=\left\{{0}\right\}$.

Let us now extend the definition of $S_{\lambda}$ to singular operators $A_{\lambda}$. Let us recall that ${\mathbb{R}}^{n}=\ker(A_{\lambda})\oplus\mathrm{rg}(A_{\lambda}^{*})$ where $A_{\lambda}^{*}$ stands for the transpose of $A_{\lambda}$ and $\mathrm{rg}(A_{\lambda}^{*})$ for its range. The operator $A_{\lambda}$ then induces a one to one operator between $\mathrm{rg}(A^{*}_{\lambda})$ and $\mathrm{rg}(A_{\lambda})$. Write $A_{\lambda}^{+}$ for the inverse of this operator from $\mathrm{rg}(A_{\lambda})$ to $\mathrm{rg}(A_{\lambda}^{*})$. The orthogonal projection operator from ${\mathbb{R}}^{n}$ onto $\mathrm{rg}(A_{\lambda}^{*})$ induces a linear operator from $\mathrm{rg}(A_{\lambda})$ into $\mathrm{rg}(A_{\lambda}^{*})$, denoted $\overline{\Pi}_{\lambda}$. Then $S_{\lambda}$ is defined as the linear span of the right-singular vectors of $A^{+}_{\lambda}-\overline{\Pi}_{\lambda}$ associated to singular values smaller than 1. Again if this set is empty, $S_{\lambda}=\left\{{0}\right\}$. When $A_{\lambda}$ is non-singular or symmetric, we recover the definition of $S_{\lambda}$ given above.

For each $\lambda\in\Lambda$, take $\SS_{\lambda}$ such that $\SS_{\lambda}\supset\left\{{S_{\lambda}}\right\}$. From a theoretical point of view, it is enough to take $\SS_{\lambda}=\left\{{S_{\lambda}}\right\}$ but practically it may be wise to use a larger set and by doing so, to possibly improve the approximation of $\widehat{f}_{\lambda}$ by elements of $\SS_{\lambda}$. One may for example take $\SS_{\lambda}=\left\{{S_{\lambda}^{1},\ldots,S_{\lambda}^{n-2}}\right\}$ where $S_{\lambda}^{k}$ is the linear span of the right-singular vectors associated to the $k$ smallest singular values of $A^{+}_{\lambda}-\overline{\Pi}_{\lambda}$.

Take $\SS=\bigcup_{\lambda\in\Lambda}\SS_{\lambda}$ and $\Delta$ of the form

where $a\geq 1$ satisfies Assumption 2 with $\Sigma\leq 1$. One may take $a=(\log\left|{\Lambda}\right|)\vee 1$ even though this choice is not necessarily the best. Finally, for some $K>1$, take $\mathop{\rm pen}\nolimits(S)=K{\mathop{\rm pen}_{\Delta}\nolimits}(S)$ for all $S\in\SS$ and select $\widehat{f}_{\widehat{\lambda}}$ by minimizing the criterion given by (6), taking thus $\delta=0$ in (9).

The following holds.

The problem of selecting some best linear estimator among a family of those have also been considered in Arlot and Bach (2009) in the Gaussian regression framework, and in Goldenshluger and Lepski (2009) in the multidimensional Gaussian white noise model. Arlot and Bach proposed a penalized procedure based on random penalties. Unlike ours, their approach requires that the operators be symmetric with eigenvalues in $[0,1]$ and that the cardinality of $\Lambda$ is at most polynomial with respect to $n$. Goldenshluger and Lepski proposed a selection rule among families of kernel estimators to solve the problem of structural adaptation. Their approach requires suitable assumptions on the kernels while ours requires nothing. Nevertheless, we restrict to the case of the Euclidean loss whereas Goldenshluger and Lepski considered more general ${\mathbb{L}}_{p}$ ones.

Start by choosing a family ${\mathcal{L}}$ of variable selection procedures. Examples of such procedures are the Lasso, the Dantzig selector, the elastic net, among others. If necessary, associate to each $\ell\in{\mathcal{L}}$ a family of tuning parameters $H_{\ell}$. For example, in order to use the Lasso procedure one needs to choose a tuning parameter $h>0$ among a grid $H_{{\rm Lasso}}\subset{\mathbb{R}}_{+}$. If a selection procedure $\ell$ requires no choice of tuning parameters, then one may take $H_{\ell}=\left\{{0}\right\}$. Let us denote by $\widehat{m}(\ell,h)$ the subset of $\{1,\ldots,p\}$ corresponding to the predictors selected by the procedure $\ell$ for the choice of the tuning parameter $h$. For $m\subset\left\{{1,\ldots,p}\right\}$, let $S_{m}$ be the linear span of the column vectors $X_{.,j}$ for $j\in m$ (with the convention $S_{\varnothing}=\left\{{0}\right\}$). For $\ell\in{\mathcal{L}}$ and $h\in H_{\ell}$, associate to the subset $\widehat{m}(\ell,h)$ an estimator $\widehat{f}_{(\ell,h)}$ of $f$ with values in $S_{\widehat{m}(\ell,h)}$ (one may for example take the projection of $Y$ onto the random linear space $S_{\widehat{m}(\ell,h)}$ but any other choice would suit). Finally, consider the family ${\mathbb{F}}=\{\widehat{f}_{\lambda},\ \lambda\in\Lambda\}$ of these estimators by taking $\Lambda=\bigcup_{\ell\in{\mathcal{L}}}(\{\ell\}\times H_{\ell})$ and set $\widehat{\mathcal{M}}=\left\{{\widehat{m}(\lambda),\ \lambda\in\Lambda}\right\}$. All along we assume that $\Lambda$ is finite (so that we take $\delta=0$ in (9)).