Superpolynomial Lower Bounds for Learning One-Layer Neural Networks using Gradient Descent

In this paper we give the first superpolynomial lower bounds for learning neural networks using gradient descent in arguably the simplest possible setting: we assume the marginal distribution is a spherical Gaussian, the labels are noiseless and are exactly equal to the output of a one-layer neural network (a linear combination of say ReLU or sigmoid activations), and the goal is to output a classifier whose test error (measured by square-loss) is small. We prove—unconditionally—that gradient descent fails to produce a classifier with small square-loss if it is required to run in polynomial time in the dimension. Our lower bound depends only on the algorithm used (gradient descent) and not on the architecture of the underlying classifier. That is, our results imply that current popular heuristics such as running gradient descent on an overparameterized network (for example, working in the NTK regime [JHG18]) will require superpolynomial time to achieve small test error.

We prove our lower bounds in the now well-studied statistical query (SQ) model of [Kea98] that captures most learning algorithms used in practice. For a loss function $\ell$ and a hypothesis $h_{\theta}$ parameterized by $\theta$, the true population loss with respect to joint distribution $D$ on $X\times Y$ is given by $\mathbb{E}_{(x,y)\sim D}[\ell(h_{\theta}(x),y)]$, and the gradient with respect to $\theta$ is given by $\mathbb{E}_{(x,y)\sim D}[\ell^{\prime}(h_{\theta}(x),y)\nabla_{\theta}h_{\theta}(x)]$. In the SQ model, we specify a query function $\phi(x,y)$ and receive an estimate of $|\mathbb{E}_{(x,y)\sim D}[\phi(x,y)]|$ to within some tolerance parameter $\tau$. An important special class of queries are correlational or inner-product queries, where the query function $\phi$ is defined only on $X$ and we receive an estimate of $|\mathbb{E}_{(x,y)\sim D}[\phi(x)\cdot y]|$ within some tolerance $\tau$. It is not difficult to see that (1) the gradient of a population loss can be approximated to within $\tau$ using statistical queries of tolerance $\tau$ and (2) for square-loss only inner-product queries are required.

Since the convergence analysis of gradient descent holds given sufficiently strong approximations of the gradient, lower bounds for learning in the SQ model [Kea98, BFJ⁺94, Szö09, Fel12, Fel17] directly imply unconditional lower bounds on the running time for gradient descent to achieve small error. We give the first superpolynomial lower bounds for learning one-layer networks with respect to any Gaussian distribution for any SQ algorithm that uses inner product queries:

In particular, this rules out any approach for learning one-layer neural networks in polynomial-time that performs gradient descent on any polynomial-size classifier with respect to square-loss or logistic loss. For classification, we obtain significantly stronger results and rule out general SQ algorithms that run in polynomial-time (e.g., gradient descent with respect to any polynomial-size classifier and any polynomial-time computable loss). In this setting, our labels are $\{\pm 1\}$ and correspond to the softmax of an unknown one-layer neural network. We prove the following:

The above lower bound for classification rules out the commonly used approach of training a polynomial-size, real-valued neural network using gradient descent (with respect to any polynomial-time computable loss) and then taking the sign of the output of the resulting network.

At the core of all SQ lower bounds is the construction of a family of functions that are pairwise approximately orthogonal with respect to the underlying marginal distribution. Typically, these constructions embed $2^{n}$ parity functions over the discrete hypercube $\{-1,1\}^{n}$. Since parity functions are perfectly orthogonal, the resulting lower bound can be quite strong. Here we wish to give lower bounds for more natural families of distributions, namely Gaussians, and it is unclear how to embed parity.

Instead, we use an alternate construction. For activation functions $\phi,\psi:{\mathbb{R}}\rightarrow{\mathbb{R}}$, define

Enumerating over every $S\subseteq[n]$ of size $k$ gives a family of functions of size $n^{O(k)}$. Here $x_{S}$ denotes the vector of $x_{i}$ for $i\in S$ (typically we choose $k=\log m$ to produce a family of one-layer neural networks with $m$ hidden units). Each of the $2^{k}=m$ inner weight vectors are all of unit norm, and all of the $m$ outer weights have absolute value one. Note also that our construction uses activations with zero bias term.

We give a complete characterization of the class of nonlinear activations for which these functions are orthogonal. In particular, the family is orthogonal for any activation with a nonzero Hermite coefficient of degree $k$ or higher.

Apart from showing orthogonality, we must also prove that functions in these classes are nontrivial (i.e., are not exponentially close to the constant zero function). This reduces to proving certain lower bounds on the norms of one-layer neural networks. The analysis requires tools from Hermite and complex analysis.

Another major challenge is that our function family is real-valued as opposed to boolean. Given an orthogonal family of (deterministic) boolean functions, it is straightforward to apply known results and obtain general SQ lower bounds for learning with respect to $0/1$ loss. For the case of real-valued functions, the situation is considerably more complicated. For example, the class of orthogonal Hermite polynomials on $n$ variables of degree $d$ has size $n^{O(d)}$, yet there is an SQ algorithm due to [APVZ14] that learns this class with respect to the Gaussian distribution in time $2^{O(d)}$. More recent work due to [ADHV19] shows that Hermite polynomials can be learned by an SQ algorithm in time polynomial in $n$ and $\log d$.

As such, it is impossible to rule out general polynomial-time SQ algorithms for learning real-valued functions based solely on orthogonal function families. Fortunately, it is not difficult to see that the SQ reductions due to [Szö09] hold in the real-valued setting as long as the learning algorithm uses only inner-product queries (and the norms of the functions are sufficiently large). Since performing gradient descent with respect to square-loss or logistic loss can be implemented using inner-product queries, we obtain our first set of desired results¹¹1The algorithms of [APVZ14] and [ADHV19] do not use inner-product queries..

Still, we would like rule out general SQ algorithms for learning simple classes of neural networks. To that end, we consider the classification problem for one-layer neural networks and output labels after performing a softmax on a one-layer network. Concretely, consider a distribution on ${\mathbb{R}}^{n}\times\{-1,1\}$ where $\mathbb{E}[Y|X]=\sigma(c(X))$ for some $c\in{\mathcal{C}}$ and $\sigma:{\mathbb{R}}\to[-1,1]$ (for example, $\sigma$ could be tanh). We describe two goals. The first is to estimate the conditional mean function, i.e., output a classifier $h$ such that $\mathbb{E}[(h(x)-c(x))^{2}]\leq\epsilon$. The second is to directly minimize classification loss, i.e., output a boolean classifier $h$ such that $\mathbb{P}_{X,Y\sim D}[h(X)\neq Y]\leq 1/2-\epsilon.$

We give superpolynomial lower bounds for both of these problems in the general SQ model by making a new connection to probabilistic concepts, a learning model due to [KS94]. Our key theorem gives a superpolynomial SQ lower bound for the problem of distinguishing probabilistic concepts induced by our one-layer neural networks from truly random labels. A final complication we overcome is that we must prove orthogonality and norm bounds on one-layer neural networks that have been composed with a nonlinear activation (e.g., tanh).

It is easy to see that our results also imply lower bounds for algorithms where the learner adds noise to the estimate of the gradient (e.g., Langevin dynamics). On the other hand, for technical reasons, it is known that SGD is not a statistical query algorithm (because it examines training points individually) and does not fall into our framework. That said, recent work by [AS20] shows that SGD is universal in the sense that it can encode all polynomial-time learners. This implies that proving unconditional lower bounds for SGD would give a proof that $\P\neq\NP$. Thus, we cannot hope to prove unconditional lower bounds on SGD (unless we can prove $\P\neq\NP$).

Independently, Diakonikolas et al. [DKKZ20] have given stronger correlational SQ lower bounds for the same class of functions with respect to the Gaussian distribution. Their bounds are exponential in the number of hidden units while ours is quasipolynomial. We can plug in their result and obtain exponential general SQ lower bounds for the associated probabilistic concept using our framework.

There is a large literature of results proving hardness results (or unconditional lower bounds in some cases) for learning various classes of neural networks [BR89, Vu98, KS09, LSSS14, GKKT17].

The most relevant prior work is due to [SVWX17], who addressed learning one-layer neural networks under logconcave distributions using Lipschitz queries. Specifically, let $n$ be the input dimension, and let $m$ be the number of hidden $s$-Lipschitz sigmoid units. For $m=\tilde{O}(s\sqrt{n})$, they construct a family of neural networks such that any learner using $\lambda$-Lipschitz queries with tolerance greater than $\Omega(1/(s^{2}n))$ needs at least $2^{\Omega(n)}/(\lambda s^{2})$ queries.

Roughly speaking, their lower bounds hold for $\lambda$-Lipschitz queries due to the composition of their one-layer neural networks with a $\delta$-function in order make the family more “boolean.” Because of their restriction on the tolerance parameter, they cannot rule out gradient descent with large batch sizes. Further, the slope of the activations they require in their constructions scales inversely with the Lipschitz and tolerance parameters.

To contrast with [SVWX17], note that our lower bounds hold for any inverse-polynomial tolerance parameter (i.e., will hold for polynomially-large batch sizes), do not require a Lipschitz constraint on the queries, and use only standard $1$-Lipschitz ReLU and/or sigmoid activations (with zero bias) for the construction of the hard family. Our lower bounds are typically quasipolynomial in the number of hidden units; improving this to an exponential lower bound is an interesting open question. Both of our models capture square-loss and logistic loss.

In terms of techniques, [SVWX17] build an orthogonal function family using univariate, periodic “wave” functions. Our construction takes a different approach, adding and subtracting activation functions with respect to overlapping “masks.” Finally, aside from the (black-box) use of a theorem from complex analysis, our construction and analysis are considerably simpler than the proof in [SVWX17].

A follow-up work [VW19] gave SQ lower bounds for learning classes of degree $d$ orthogonal polynomials in $n$ variables with respect to the uniform distribution on the unit sphere (as opposed to Gaussians) using inner product queries of bounded tolerance (roughly $1/n^{d}$). To obtain superpolynomial lower bounds, each function in the family requires superpolynomial description length (their polynomials also take on very small values, $1/n^{d}$, with high probability).

Shamir [Sha18] (see also the related work of [SSSS17]) proves hardness results (and lower bounds) for learning neural networks using gradient descent with respect to square-loss. His results are separated into two categories: (1) hardness for learning “natural” target families (one layer ReLU networks) or (2) lower bounds for “natural” input distributions (Gaussians). We achieve lower bounds for learning problems with both natural target families and natural input distributions. Additionally, our lower bounds hold for any nonlinear activations (as opposed to just ReLUs) and for broader classes of algorithms (SQ).

Recent work due to [GKK19] gives hardness results for learning a ReLU with respect to Gaussian distributions. Their results require the learner to output a single ReLU as its output hypothesis and require the learner to succeed in the agnostic model of learning. [KK14] prove hardness results for learning a threshold function with respect to Gaussian distributions, but they also require the learner to succeed in the agnostic model. Very recent work due to Daniely and Vardi [DV20] gives hardness results for learning randomly chosen two-layer networks. The hard distributions in their case are not Gaussians, and they require a nonlinear clipping output activation.

A statistical query is specified by a query function $h:{\mathbb{R}}^{d}\times Y\to{\mathbb{R}}$. The SQ model allows access to an SQ oracle that accepts a query $h$ of specified tolerance $\tau$, and responds with a value in $[\mathbb{E}_{x,y}[h(x,y)]-\tau,\mathbb{E}_{x,y}[h(x,y)]+\tau]$.²²2In the SQ literature, this is referred to as the STAT oracle. A variant called VSTAT is also sometimes used, known to be equivalent up to small polynomial factors [Fel17]. While it makes no substantive difference to our superpolynomial lower bounds, our arguments can be extended to VSTAT as well. To disallow arbitrary scaling, we will require that for each $y$, the function $x\mapsto h(x,y)$ has norm at most 1. In the real-valued setting, a query $h$ is called a correlational or inner product query if it is of the form $h(x,y)=g(x)\cdot y$ for some function $g$, so that $\mathbb{E}_{D_{c}}[h]=\mathbb{E}_{D}[gc]=\langle g,c\rangle_{D}$. Here we assume $\|g\|\leq 1$ when stating lower bounds, again to disallow arbitrary scaling.

Gradient descent with respect to squared loss is captured by inner product queries, since the gradient is given by

Here the first term can be estimated directly using knowledge of the distribution, while the latter is a vector each of whose elements is an inner product query.

We now formally define the learning problems we consider.

For the classification setting, we consider two different notions of learning $p$-concepts. One is learning the target up to small $L_{2}$ error, to be thought of as a strong form of learning. The other, weaker form, is achieving a nontrivial inner product (i.e. unnormalized correlation) with the target. We prove lower bounds on both in order to capture different learning goals.

Note that the best achievable advantage is $\mathbb{E}_{x\sim D}[|c(x)|]$, achieved by $\tilde{c}(x)=\operatorname{sign}(c(x))$. Note also that $\|c\|_{D}^{2}\leq\mathbb{E}_{D}[|c|]\leq\|c\|_{D}$, and therefore a norm lower bound on functions in ${\mathcal{C}}$ implies an upper bound on the achievable advantage.

The difficulty of learning a concept class in the SQ model is captured by a parameter known as the statistical dimension of the class.

We will mainly be interested in the statistical dimension in a setting where bounds on pairwise correlations are known. In that case the following lemma holds.

Prior work [Szö09, Fel12] has already established the following fundamental result, which we phrase in terms of our definition of statistical dimension. For the reader’s convenience, we include a proof in Appendix B.

It turns out to be fruitful to view our learning problem in terms of a decision problem over distributions. We define the problem of distinguishing a valid labeled distribution from a randomly labeled one, and show a lower bound for this problem. We then show that learning is at least as hard as distinguishing, thereby extending the lower bound to learning as well. Our analysis closely follows that of [FGR⁺17].

Considering the distinguishing problem is useful because if we can show that distinguishing itself is hard, then any reasonable notion of learning will be hard as well, including weak learning. We give simple reductions for both our notions of learning.

We now prove the main lower bound on distinguishing.

The final lower bounds on learning thus obtained are stated as a corollary for convenience. The proof follows directly from Lemma 4.4 and Theorem 4.5.