Minimum complexity interpolation in random features models

Kernel methods are among the most fundamental tools in machine learning. Over the last two years they attracted renewed interest because of their connection with neural networks in the linear regime (a.k.a. neural tangent kernel or lazy regime) [JGH18, LR20, LRZ19, GMMM19].

Consider a general covariates space, namely a probability space $(\mathcal{X},\mathbb{P})$ (our results will concern the case $\mathcal{X}={\mathbb{R}}^{d}$, but it is useful to start from a more general viewpoint). A reproducing kernel Hilbert space (RKHS) is usually constructed starting from a positive semidefinite kernel $K:\mathcal{X}\times\mathcal{X}\to{\mathbb{R}}$. Here it will be more convenient to start from a weight space, i.e. a probability space $({\mathcal{V}},\mu)$, and a featurization map $\phi(\cdot;{\bm{w}}):\mathcal{X}\to{\mathbb{R}}$ parametrized by the weight vector¹¹1A sufficient condition for this construction to be well defined is $\phi\in L^{2}(\mathbb{P}\otimes\mu)$. ${\bm{w}}\in{\mathcal{V}}$. The RKHS is then defined as the space of functions of the form

$$\hat{f}({\bm{x}};a)=\int_{{\mathcal{V}}}\phi({\bm{x}};{\bm{w}})\,a({\bm{w}})\,\mu({\rm d}{\bm{w}})\,,$$

(1)

with $a\in L^{2}({\mathcal{V}};\mu)$. To give a concrete example, we might consider ${\mathcal{V}}={\mathbb{R}}^{d}$, and $\phi({\bm{x}};{\bm{w}})=\sigma(\langle{\bm{w}},{\bm{x}}\rangle)$: this is the featurization map arising from two-layers neural networks with random first layer weights.

The radius-$R$ ball in this space is defined as

$${\mathcal{F}}_{2}(R)=\Big{\{}\hat{f}({\bm{x}};a)=\int_{{\mathcal{V}}}\phi({\bm{x}};{\bm{w}})\,a({\bm{w}})\,\mu({\rm d}{\bm{w}}):\;\;\int_{{\mathcal{V}}}|a({\bm{w}})|^{2}\mu({\rm d}{\bm{w}})\leq R^{2}\Big{\}}\,.$$

(2)

This construction is equivalent to the more standard one, with associated kernel $K({\bm{x}}_{1},{\bm{x}}_{2}):=\int_{{\mathcal{V}}}\phi({\bm{x}}_{1};{\bm{w}})\phi({\bm{x}}_{2};{\bm{w}})\,\mu({\rm d}{\bm{w}})$. Vice versa, for any positive semidefinite kernel $K$, we can construct a corresponding featurization map $\phi$, and proceed as above.

It is a basic fact that, although ${\mathcal{F}}_{2}(R)$ is infinite-dimensional, learning in ${\mathcal{F}}_{2}(R)$ can be performed efficiently [SSBD14]: it is useful to recall the reason here. Consider a supervised learning setting in which we are given $n$ samples $(y_{i},{\bm{x}}_{i})$, $i\leq n$, with $y_{i}\in\mathbb{R}$ and ${\bm{x}}_{i}\in\mathcal{X}$. Given a convex loss $\ell:\mathbb{R}\times\mathbb{R}\to\mathbb{R}$, the RKHS-norm regularized empirical risk minimization problem reads:

$\displaystyle\mbox{\rm minimize}\;\;\;\sum_{i=1}^{n}\ell\big{(}y_{i},\hat{f}({\bm{x}}_{i};a)\big{)}+\lambda\int_{{\mathcal{V}}}|a({\bm{w}})|^{2}\mu({\rm d}{\bm{w}})\,.$

(3)

Conceptually, this problem can be solved in two steps: $(1)$ find the minimum RKHS norm subject to interpolation constraints $\hat{f}({\bm{x}}_{i};a)=\hat{y}_{i}$, and $(2)$ minimize the sum of this quantity and the empirical loss $\sum_{i=1}^{n}\ell\big{(}y_{i},\hat{y}_{i})$. Since step $(2)$ is convex and finite-dimensional, the critical problem is the interpolation problem:

	minimize	$\displaystyle\int_{{\mathcal{V}}}|a({\bm{w}})|^{2}\mu({\rm d}{\bm{w}})\,,$
	subj. to	$\displaystyle\hat{f}({\bm{x}}_{i};a)=\hat{y}_{i},\;\;\;\forall i\leq n\,.$

While this is an infinite-dimensional problem, convex duality (the ‘representer theorem’) guarantees that the solution belongs to a fixed $n$-dimensional subspace, $a_{*}({\bm{w}})=\sum_{i=1}^{n}c_{i}\phi({\bm{x}}_{i};{\bm{w}})$.

Unfortunately, kernel methods suffer from the curse of dimensionality. To give a simple example, consider ${\bm{x}}\sim{\rm Unif}(\mathbb{S}^{d-1}(\sqrt{d}))$, ${\bm{w}}\sim{\rm Unif}(\mathbb{S}^{d-1}(1))$, and $\phi({\bm{x}};{\bm{w}})=\sigma(\langle{\bm{w}},{\bm{x}}\rangle)$ with $\sigma:{\mathbb{R}}\to{\mathbb{R}}$ a non-polynomial activation function. Consider noiseless data $y_{i}=f_{*}({\bm{x}}_{i})$, with $f_{*}({\bm{x}})=\sigma(\langle{\bm{w}}_{*},{\bm{x}}\rangle)$ for a fixed ${\bm{w}}_{*}\in\mathbb{S}^{d-1}(1)$. Then, $\|f_{*}\|_{K}=\infty$ (with $\|\,\cdot\,\|_{K}$ denoting the RKHS norm). Correspondingly, [YS19, GMMM19] show that for any fixed $k$, if $n\leq d^{k}$, then any kernel method of the form (3) returns $\hat{f}$ with $\|\hat{f}-f_{*}\|_{L^{2}(\mathbb{P})}$ bounded away from zero.

The curse of dimensionality suggests to seek functions of the form (1) where $a$ is sparse, in a suitable sense [Bac17]. In this paper we consider a generalization in which the RKHS ball of Eq. (2) is replaced by

$${\mathcal{F}}_{p}(R)=\Big{\{}\hat{f}({\bm{x}};a)=\int_{{\mathcal{V}}}\phi({\bm{x}};{\bm{w}})\,a({\bm{w}})\,\mu({\rm d}{\bm{w}}):\;\;\Big{(}\int_{{\mathcal{V}}}|a({\bm{w}})|^{p}\mu({\rm d}{\bm{w}})\Big{)}^{1/p}\leq R\Big{\}}\,.$$

(4)

For $p\in[1,2)$ this comprises a richer function class than the original ${\mathcal{F}}_{2}(R)$, since ${\mathcal{F}}_{2}(R)\subset{\mathcal{F}}_{p}(R)\subset{\mathcal{F}}_{1}(R)$, and it is easy to see that the inclusion is strict. The case $\rho(x)=|x|$ is also known as ‘convex neural network’ [Bac17].

Although the penalty $\int_{{\mathcal{V}}}|a({\bm{w}})|^{p}\mu({\rm d}{\bm{w}})$ is convex, it is far from clear that the learning problem is tractable. Indeed for $p\neq 2$, the classical representer theorem does not hold anymore and we cannot reduce the infinite dimensional problem to solving a finite dimensional quadratic program (see Appendix E.2 for a representer-type theorem for $p>1$).

By the same reduction discussed above, it is sufficient to consider the following minimum-complexity interpolation problem:

	minimize	$\displaystyle\int_{{\mathcal{V}}}\rho(a({\bm{w}}))\mu({\rm d}{\bm{w}})\,,$		(5)
	subj. to	$\displaystyle\hat{f}({\bm{x}}_{i};a)=y_{i},\;\;\;\forall i\leq n\,.$

(We denote the values to be interpolated by $y_{i}$ instead of $\hat{y}_{i}$ because we will focus on the interpolation problem hereafter.) We will take $\rho(x)$ to be a convex function minimized at $x=0$. We establish two main results. First, we establish tractability for a subset of strictly convex penalties $\rho$ which include, as special cases, $\rho(x)=|x|^{p}/p$, $p>1$. Our approach is based on a random features approximation which we discuss next. Second, we establish $\mathsf{NP}$-hardness under randomized reduction for the case $\rho(x)=|x|$.

We sample independently ${\bm{w}}_{j}\sim\mu$, $j\leq N$, and fit a model

$$\hat{f}_{N}({\bm{x}};{\bm{a}})=\frac{1}{N}\sum_{j=1}^{N}a_{j}\phi({\bm{x}};{\bm{w}}_{j})\,.$$

(6)

We determine the coefficients ${\bm{a}}=(a_{i})_{i\leq N}$ by solving the interpolation problem

	minimize	$\displaystyle\sum_{j=1}^{N}\rho(a_{j})\,,$		(7)
	subj. to	$\displaystyle\hat{f}_{N}({\bm{x}}_{i};{\bm{a}})=y_{i},\;\;\;\forall i\leq n\,.$

Notice that this is equivalent to replacing the measure $\mu$ in Eq. (5) by its empirical version $\hat{\mu}_{N}=N^{-1}\sum_{i=1}^{N}\delta_{{\bm{w}}_{i}}$. Borrowing the terminology from neural network theory, we will refer to (7) as the “finite width” problem, and to the original problem (5) as the “infinite width” problem. We will denote by $\hat{\bm{a}}_{N}$ the solution of the finite width problem (7) and by $\hat{a}$ the solution of the infinite width problem (5).

Our main result establishes that, under suitable assumptions on the penalty $\rho$ and the featurization map $\phi$, the random features approach provides a good approximation of the minimum complexity interpolation problem (5). Crucially, this is achieved with a number of features that is polynomial in the sample size $n$. In particular we show that for $\rho(x)=|x|^{p}/p$, $1<p\leq 2$, setting $Q=p/(p-1)$ (and assuming $\|{\bm{y}}\|_{2}\leq C\sqrt{n}$)

$$\|\hat{f}_{N}(\cdot;\hat{\bm{a}}_{N})-\hat{f}(\cdot;\hat{a})\|_{L^{2}}\leq C\Big{(}\sqrt{\frac{n^{2}\log(N)}{N}}\vee\frac{(n\log N)^{(Q+1)/2)}}{N}\Big{)}\,\,.$$

(8)

Hence, for $N\gg(n\log n)^{2}\vee(n\log n)^{(Q+1)/2}$, the random features approach yields a function that is a good approximation of the original problem (5).

Let us emphasize that the scaling of the number of random features $N$ in our bound is not optimal. In particular, for the RKHS case $p=2$, the above result requires $N\gg n^{2}$, while we know from [MMM21] that $N\gg n\log n$ features are often sufficient. We also note that the exponent $Q$ in the polynomial diverges as $p\to 1$. Hence, our results do not guarantee tractability for the case $p=1$. Indeed this is a fundamental limitation: as discussed in Section 4, a bound such as Eq. (8) with finite $Q$ cannot hold for $p=1$, under some standard hardness assumptions. We show instead that no polynomial time algorithm is guaranteed to achieve accuracy $n^{-C}$ for some absolute constant $C$ in Eq. (8). This hardness result is based on a randomized reduction from the problem of learning halfspaces with noise, which was proved to be $\mathsf{NP}$-hard in [FGKP06, GR09].

Our proof that the random features model $\hat{f}_{N}(\cdot;\hat{\bm{a}}_{N})$ is a good approximation of the infinite width model $\hat{f}(\cdot;\hat{a})$ (cf. Eq. (8)) is based on a simple approach which is potentially of independent interest. We notice that, while the optimization problems (5) and (7) are overparametrized and hence difficult to control, their duals are underparametrized and can be studied using uniform convergence arguments.

Let $\rho^{*}$ be the convex conjugate of $\rho$:

$$\rho^{*}(x)=\sup_{y\in\mathbb{R}}\{xy-\rho(y)\}\,.$$

Then, the dual problems of (5) and (7) are given —respectively— by the following optimization problems over ${\bm{\lambda}}\in\mathbb{R}^{n}$:

	$\displaystyle\mbox{maximize}\;\;\;\;\langle{\bm{\lambda}},{\bm{y}}\rangle-\int_{\mathcal{V}}\rho^{*}(\langle\bm{\phi}_{n}({\bm{w}}),{\bm{\lambda}}\rangle)\mu({\rm d}{\bm{w}})\,,$		(9)
	$\displaystyle\mbox{maximize}\;\;\;\;\langle{\bm{\lambda}},{\bm{y}}\rangle-\frac{1}{N}\sum_{j=1}^{N}\rho^{*}(\langle\bm{\phi}_{n}({\bm{w}}_{j}),{\bm{\lambda}}\rangle)\,.$		(10)

Here we denoted ${\bm{y}}=(y_{1},\ldots,y_{n})\in\mathbb{R}^{n}$ the vector of responses and by $\bm{\phi}_{n}:{\mathcal{V}}\to\mathbb{R}^{n}$ the evaluation of the feature map at the $n$ data points $\bm{\phi}_{n}({\bm{w}})=(\phi({\bm{x}}_{1};{\bm{w}}),\ldots,\phi({\bm{x}}_{n};{\bm{w}}))\in\mathbb{R}^{n}$.

We will prove that, under suitable assumptions on the penalty $\rho$, the optimizer of the finite-width dual (10) concentrates around the optimizer of the infinite-width dual (9). Our results hold conditionally on the realization of the data $\{({\bm{x}}_{i},y_{i})\}_{i\leq n}$ and instead exploit the randomness of the weights $\{{\bm{w}}_{i}\}_{i\leq N}$.

The rest of the paper is organized as follows. After briefly discussing related work in Section 2, we state our assumptions and results for strictly convex penalties in Section 3. In Section 4, we show that the problem with ${\mathcal{F}}_{1}$ norm is $\mathsf{NP}$-hard under randomized reduction. In Section 5, we describe a few examples in which we can apply our general results. The proof of our main result is outlined in Section 6, with most technical work deferred to the appendices.

We consider a slightly more general setting than the one discussed in the introduction, whereby we allow the featurization map functions to be randomized. More precisely, conditional on the data $({\bm{x}}_{i})_{i\leq n}$ and weight vectors $({\bm{w}}_{j})_{j\leq N}$, the feature values $\{\phi({\bm{x}}_{i};{\bm{w}}_{j})\}_{i\leq n,j\leq N}$ are independent random variables. Explicitly, such randomized features can be constructed by letting $\phi({\bm{x}};{\bm{w}},z)$ depend on an additional variable $z\in\mathcal{Z}$, and setting (with an abuse of notation) $\phi({\bm{x}}_{i};{\bm{w}}_{j})=\phi({\bm{x}}_{i};{\bm{w}}_{j},z_{ij})$ for $\{z_{ij}\}_{i\leq n,j\leq N}$ a collection of independent random variables with common law $z_{ij}\sim\nu$ (with $\nu$ a probability distribution over $\mathcal{Z}$). Without loss of generality²²2For instance, this is the case as long as the weights take values in a Polish space, by which the conditional probabilities $\mathbb{P}(\phi({\bm{x}};{\bm{w}})\in S|{\bm{w}})=P(S;{\bm{w}})$ exist. we can assume $z_{ij}\sim{\rm Unif}([0,1])$.

We denote by $\overline{\phi}({\bm{x}};{\bm{w}})=\mathbb{E}_{\phi}[\phi({\bm{x}};{\bm{w}})]=\mathbb{E}_{z}[\phi({\bm{x}};{\bm{w}},z)]$ the expectation of the features with respect to this additional randomness. In what follows, we will omit to write the dependency on $z$ explicitly, unless required for clarity. The additional freedom afforded by randomized features is useful in multiple scenarios:

• •

  We only have access to noisy measurements $\phi({\bm{x}}_{i};{\bm{w}}_{j})$ of the true features $\overline{\phi}({\bm{x}}_{i};{\bm{w}}_{j})$.

• •

  We deliberately introduce noise in the featurization mechanism. This is known to have a regularizing effect [Bis95].

• •

  We do not explicitly introduce noise in the featurization mechanism. However, the noiseless featurization mechanism is asymptotically equivalent to one in which noise has been added. Examples of this type are given in [MM19, GLK⁺20, HL20]

At prediction time we use the average features³³3Our results do not change significantly if we use randomized features also at prediction time.

$$\hat{f}({\bm{x}};a)=\int_{{\mathcal{V}}}\overline{\phi}({\bm{x}};{\bm{w}})a({\bm{w}})\,\mu({\rm d}{\bm{w}})\,,\qquad\hat{f}_{N}({\bm{x}};{\bm{a}})=\frac{1}{N}\sum_{j=1}^{N}a_{j}\overline{\phi}({\bm{x}};{\bm{w}}_{j})\,.$$

(12)

The dual of the finite-width problem (7) is given by the problem (10), for which we introduce the following notations:

	$\displaystyle F_{N}({\bm{\lambda}}):=$	$\displaystyle~{}\langle{\bm{\lambda}},{\bm{y}}\rangle-\frac{1}{N}\sum_{j=1}^{N}\rho^{*}(\langle\bm{\phi}_{n,j},{\bm{\lambda}}\rangle)\,,$		(13)
	$\displaystyle\hat{\bm{\lambda}}_{N}=$	$\displaystyle~{}\arg\max_{{\bm{\lambda}}\in\mathbb{R}^{n}}F_{N}({\bm{\lambda}})\,.$

Notice that $\bm{\phi}_{n,j}=\bm{\phi}_{n}({\bm{w}}_{j})=(\phi({\bm{x}}_{1};{\bm{w}}_{j}),\ldots,\phi_{{\bm{w}}}({\bm{x}}_{n};{\bm{w}}_{j}))^{{\mathsf{T}}}$ is now a random vector. In the case of random features, the dual of the infinite width problem (5) has to be modified with respect to (9), and takes instead the form

	$\displaystyle F({\bm{\lambda}}):=$	$\displaystyle~{}\langle{\bm{\lambda}},{\bm{y}}\rangle-\int_{\mathcal{V}}\mathbb{E}_{\phi}[\rho^{*}(\langle\bm{\phi}_{n}({\bm{w}}),{\bm{\lambda}}\rangle)]\mu({\rm d}{\bm{w}})\,,$		(14)
	$\displaystyle\hat{\bm{\lambda}}=$	$\displaystyle~{}\arg\max_{{\bm{\lambda}}\in\mathbb{R}^{n}}F({\bm{\lambda}})\,.$

Note the expectation $\mathbb{E}_{\phi}$ with respect to the randomness in the features (equivalently, this is the expectation with respect to the independent randomness $z$, which is not noted explicitly). The most direct way to see that this is the correct infinite width dual is to notice that this is obtained from Eq. (13) by taking expectation with respect to the weights $\{{\bm{w}}_{j}\}_{j\leq N}$ and the features randomness. We will further discuss the connection between (14) and the infinite width primal problem in Section E.1 below.

In order to discuss the dual optimality conditions, let $s:\mathbb{R}\to\mathbb{R}$ be the derivative of the convex conjugate of $\rho$, i.e., $s(x)=(\rho^{*})^{\prime}(x)=(\rho^{\prime})^{-1}(x)$. Since $\rho$ is assumed to be strictly convex, $s(x)$ exists and is continuous and non-decreasing. With these definitions, the dual optimality condition reads

$\displaystyle{\bm{y}}=\frac{1}{N}\sum_{j=1}^{N}\bm{\phi}_{n,j}\,s(\langle\bm{\phi}_{n,j},{\hat{\bm{\lambda}}}_{N}\rangle)\,.$

(15)

The primal solution is then given by $(\hat{\bm{a}}_{N})_{j}=s(\langle\bm{\phi}_{n,j},\hat{\bm{\lambda}}_{N}\rangle)$ and the resulting predictor is

$$\hat{f}_{N}({\bm{x}};\hat{\bm{a}}_{N})=\frac{1}{N}\sum_{j=1}^{N}\overline{\phi}({\bm{x}};{\bm{w}}_{j})\,s(\langle\bm{\phi}_{n,j},\hat{\bm{\lambda}}_{N}\rangle)\,.$$

In the following, with an abuse of notation, we will write $\hat{f}_{N}({\bm{x}};{\bm{\lambda}})=N^{-1}\sum_{j=1}^{N}\overline{\phi}({\bm{x}};{\bm{w}}_{j})\,s(\langle\bm{\phi}_{n,j},{\bm{\lambda}}\rangle)$ for the model at dual parameter ${\bm{\lambda}}$. The corresponding infinite width predictor reads

$\displaystyle\hat{f}({\bm{x}};{\bm{\lambda}}):=\int_{{\mathcal{V}}}\overline{\phi}({\bm{x}};{\bm{w}})\,\mathbb{E}_{\phi}[s(\langle\bm{\phi}_{n}({\bm{w}}),{\bm{\lambda}}\rangle)]\,\mu({\rm d}{\bm{w}})\,.$

(16)

We will show that conditional on the realization of ${\bm{y}},{\bm{X}}$, the distance between the infinite width interpolating model $\hat{f}(\,\cdot\,;{\hat{\bm{\lambda}}})$ and the finite width one $\hat{f}_{N}(\,\cdot\,;{\hat{\bm{\lambda}}}_{N})$ is small with high probability as soon as $N$ is large enough. Throughout this section, ${\bm{y}},{\bm{X}}$ are viewed as fixed, and we assume certain conditions to hold on the distribution of the features $\phi_{n}({\bm{w}})$. In Section 5 we will check that these conditions hold for a few models of interest, for typical realizations of ${\bm{y}},{\bm{X}}$.

We first state our assumptions on the feature distribution. We define the whitened features

$\displaystyle{\bm{\psi}}_{n,j}:={\bm{K}}_{n}^{-1/2}\bm{\phi}_{n,j},\;\;\;\;\;\;\;{\bm{K}}_{n}:=\int\mathbb{E}_{\phi}[\bm{\phi}_{n}({\bm{w}})\bm{\phi}_{n}({\bm{w}})^{\top}]\,\mu({\rm d}{\bm{w}})\,.$

(17)

Here expectation is over the randomization in the features, and ${\bm{K}}_{n}\in{\mathbb{R}}^{n\times n}$ is the empirical kernel matrix. By construction, ${\bm{\psi}}_{n,j}$ are isotropic: $\mathbb{E}_{{\bm{w}},\phi}[{\bm{\psi}}_{n,j}{\bm{\psi}}_{n,j}^{\top}]={\mathbf{I}}_{n}$.

We will assume the following conditions to hold for the featurization map $({\bm{x}},{\bm{w}})\mapsto\phi({\bm{x}};{\bm{w}})$ and the penalty $x\mapsto\rho(x)$. Throughout we will follow the convention of denoting by Greek letters constants that we track of in our calculations, and by $C,c,r_{0}$ constants that we do not track (and therefore our statements depend on an unspecified way on the latter). We also recall that a random vector ${\bm{Z}}$ is $\gamma^{2}$-subgaussian if $\mathbb{E}[\exp(\langle{\bm{v}},{\bm{Z}}\rangle)]\leq\exp(\gamma^{2}\|{\bm{v}}\|^{2}/2)$ for all vectors ${\bm{v}}$.

FEAT1

(Sub-gaussianity) For some $0<\tau\leq n^{C}$ and any fixed $\|{\bm{x}}\|_{2}\leq r_{0}\sqrt{d}$, $\overline{\phi}({\bm{x}};{\bm{w}})$ is $\tau^{2}$-sub-Gaussian when ${\bm{w}}\sim\mu$. Further, the feature vector ${\bm{\psi}}_{n}:={\bm{K}}_{n}^{-1/2}[\phi(x_{1};{\bm{w}},z_{1}),\dots,\phi(x_{n};{\bm{w}},z_{n})]^{{\mathsf{T}}}$ is $\tau^{2}$-sub-Gaussian when ${\bm{w}}\sim\mu$, $(z_{i})_{i\leq n}\sim_{iid}\nu$. Without loss of generality we assume $\tau\geq 1$.

FEAT2

(Lipschitz continuity) For any ${\bm{w}}\in{\mathcal{V}}$, assume that $\overline{\phi}({\bm{x}};{\bm{w}})$ is $L({\bm{w}})$-Lipschitz with respect to ${\bm{x}}$ and $\bar{\phi}({\bm{0}};{\bm{w}})\leq L({\bm{w}})$, where $\mathbb{P}(L({\bm{w}})\geq t)\leq Ce^{-t^{2}/(2\tau^{2})}$, $\tau\geq 1$.

FEAT3

(Small ball) There exists $\eta\geq n^{-C}$ such that

$\displaystyle\inf_{\|{\bm{u}}\|_{2}=1,\|{\bm{v}}\|_{2}=1}\mathbb{P}\big{(}|\langle{\bm{u}},{\bm{\psi}}_{n}\rangle|\geq\eta,\,|\langle{\bm{v}},{\bm{\psi}}_{n}\rangle|\geq\eta\big{)}\geq c\,,$

(18)

for some strictly positive constants $C,c$. By union bound, this is implied by the stronger condition

$$\sup_{\|{\bm{v}}\|_{2}=1}\mathbb{P}\big{(}|\langle{\bm{v}},{\bm{\psi}}_{n}\rangle|\leq\eta\big{)}\leq\frac{1}{2}(1-c)\,.$$

Without loss of generality, we will assume $\eta\leq 1$.

PEN

(Polynomial growth) We assume that $\rho$ is strictly convex and minimized at 0, so that $s(x)$ is continuous and $s(0)=0$. Because $s$ is non-decreasing, we have that $s(x)$ has a derivative almost everywhere. We assume there exists $Q_{1},Q_{2},q_{1},q_{2}>1$ and $C,c>0$ such that for $|x_{1}|,|x_{2}|>0$,

$$\displaystyle c|s(x_{2})/x_{2}|\,(|x_{1}/x_{2}|^{q_{1}-2}\wedge|x_{1}/x_{2}|^{q_{2}-2})\leq s^{\prime}(x_{1})\leq C|s(x_{2})/x_{2}|\,(|x_{1}/x_{2}|^{Q_{1}-2}\vee|x_{1}/x_{2}|^{Q_{2}-2}),$$

and

$$\displaystyle c|s(x_{2})|\,(|x_{1}/x_{2}|^{q_{1}-1}\wedge|x_{1}/x_{2}|^{q_{2}-1})\leq|s(x_{1})|\leq C|s(x_{2})|\,(|x_{1}/x_{2}|^{Q_{1}-1}\vee|x_{1}/x_{2}|^{Q_{2}-1}).$$

(19)

If $s^{\prime}(x)=(\rho^{*})^{\prime\prime}(x)$ is unbounded as $x\to 0$, we will require a stronger assumption FEAT3’ which will ensure that $\nabla^{2}F({\bm{\lambda}})$ exists and is finite.

FEAT3’

(Small ball) For every $r\in(-1,0)$ and every $\|{\bm{v}}\|_{2}=1$, $\mathbb{E}[|\langle{\bm{\psi}}_{n},{\bm{v}}\rangle|^{r}]\leq C_{r}\eta^{r}<\infty$, $\eta\leq 1$.

Our main theorem establishes that the infinite-dimensional problem of Eq. (5) (and its generalization to randomized featurization maps) is well-approximated by its finite random features counterpart. For this approximation to be good, it is sufficient that the number of random features scales polynomially with the sample size $n$.

In order to interpret Theorem 1, we remark that we expect typically $\|{\bm{K}}_{n}^{-1/2}{\bm{y}}\|_{2}$ to be of order $\sqrt{n}$. In this case $\hat{f}_{N}(\,\cdot\,;\hat{\bm{\lambda}}_{N})$ differs negligibly from $\hat{f}(\,\cdot\,;{\hat{\bm{\lambda}}})$ when $N\gg(n\log(n))^{2\vee(Q+1)/2}$.

It is instructive to specialize Theorem 1 to the case of $p$-norms, which is covered by taking $\rho(x)=|x|^{p}/p$, $p\in(1,2]$. In this case $q_{1}=q_{2}=Q_{1}=Q_{2}=Q$, with $Q=p/(p-1)\geq 2$ and hence the formulas are simpler.

Note that the exponent $Q$ on the right hand side of Eq. (22) diverges as $p\to 1$. Instead of being a feature of our proof technique, we show in the next section (Section 4), that this is unavoidable under standard hardness assumptions.