Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape

The optimization landscape of neural networks has been a topic of enormous interest over the years. A particularly puzzling question is why bad local minima do not seem to be a problem for training. In this context, an important observation is that overparameterization can yield a more benevolent optimization landscapes. While this type of observation can be traced back at least to the works of Poston et al. (1991); Gori & Tesi (1992), it has increasingly become a focus of investigation in recent years. In this article we follow this line of work and describe the optimization landscape of a moderately overparameterized ReLU network with a view on the different activation regions of the parameter space.

Before going into the details of our results we first provide some context. The existence of non-global local minima has been documented in numerous works. This is the case even for networks without hidden units (Sontag & Sussmann, 1989) and single units (Auer et al., 1995). For shallow networks, Fukumizu & Amari (2000) showed how local minima and plateaus can arise from the hierarchical structure of the model for general types of targets, loss functions and activation functions. Other works have also constructed concrete examples of non-global local minima for several architectures (Swirszcz et al., 2017). While these works considered finite datasets, Safran & Shamir (2018) observed that spurious minima are also common for the student-teacher population loss of a two-layer ReLU network with unit output weights. In fact, for ReLU networks Yun et al. (2019b); Goldblum et al. (2020) show that if linear models cannot perfectly fit the data, one can construct local minima that are not global. He et al. (2020) shows the existence of spurious minima for arbitrary piecewise linear (non-linear) activations and arbitrary continuously differentiable loss functions.

A number of works suggest overparameterized networks have a more benevolent optimization landscape than underparameterized ones. For instance, Soudry & Carmon (2016) show for mildly overparameterized networks with leaky ReLUs that for almost every dataset every differentiable local minimum of the squared error training loss is a zero-loss global minimum, provided a certain type of dropout noise is used. In addition, for the case of one hidden layer they show this is the case whenever the number of weights in the first layer matches the number of training samples, $d_{0}d_{1}\geq n$. In another work, Safran & Shamir (2016) used a computer aided approach to study the existence of spurious local minima, arriving at the key finding that when the student and teacher networks are equal in width spurious local minima are commonly encountered. However, under only mild overparameterization, no spurious local minima were identified, implying again that overparameterization leads to a more benign loss landscape. In the work of Tian (2017), the student and teacher networks are assumed to be equal in width and it is shown that if the dimension of the data is sufficiently large relative to the width, then the critical points lying outside the span of the ground truth weight vectors of the teacher network form manifolds. Another related line of works studies the connectivity of sublevel sets (Nguyen, 2019) and bounds on the overparameterization that ensures existence of descent paths (Sharifnassab et al., 2020).

The key objects in our investigation will be the rank of the Jacobian of the parametrization map over a finite input dataset, and the combinatorics of the corresponding subdivision of parameter space into pieces where the loss is differentiable. The Jacobian map captures the local dependency of the functions represented over the dataset on their parameters. The notion of connecting parameters and functions is prevalent in old and new studies of neural networks: for instance, in discussions of parameter symmetries (Chen et al., 1993), functional equivalence (Phuong & Lampert, 2020), functional dimension (Grigsby et al., 2022), the question of when a ReLU network is a universal approximator over a finite input dataset (Yun et al., 2019a), as well as in studies involving the neural tangent kernel (Jacot et al., 2018).

We highlight a few works that take a geometric or combinatorial perspective to discuss the optimization landscape. Using dimension counting arguments, Cooper (2021) showed that, under suitable overparameterization and smoothness assumptions, the set of zero-training loss parameters has the expected codimension, equal to the number of training data points. In the case of ReLU networks, Laurent & von Brecht (2018) used the piecewise multilinear structure of the parameterization map to describe the location of the minimizers of the hinge loss. Further, for piecewise linear activation functions Zhou & Liang (2018) partition the parameter space into cones corresponding to different activation patterns of the network to show that, while linear networks have no spurious minima, shallow ReLU networks do. In a similar vein, Liu (2021) study one hidden layer ReLU networks and show for a convex loss that differentiable local minima in an activation region are global minima within the region, as well as providing conditions for the existence of differentiable local minima, saddles and non differentiable local minima. Considering the parameter symmetries, Simsek et al. (2021) show that the level of overparameterization changes the relative number of subspaces that make up the set of global minima. As a result, overparameterization implies a larger set of global minima relative to the set of critical points, and underparameterization the reverse. Wang et al. (2022) studied the optimization landscape in two-layer ReLU networks and showed that all optimal solutions of the non-convex loss can be found via the optimal solutions of a convex program.

Before specializing to specific neural network architectures, we introduce general definitions which encompass all of the models we will study. For any $d\in\mathbb{N}$ we will write $[d]:=\{1,\ldots,d\}$. We will write $\bm{1}_{d}$ for a vector of $d$ ones, and drop the subscript when the dimension is clear from the context. The Hadamard (entry-wise) product of two matrices $A$ and $B$ of the same dimension is defined as $A\odot B:=(A_{ij}\cdot B_{ij})$. The Kronecker product of two vectors $u\in\mathbb{R}^{n}$ and $v\in\mathbb{R}^{m}$ is defined as $u\otimes v:=(u_{i}v_{j})\in\mathbb{R}^{nm}$.

For more details on the above facts, we refer the reader to Harris (2013). Justified by Lemma 1, we say that a property $P$ depending on $x\in\mathbb{R}^{n}$ holds for generic $x$ if there exists a proper algebraic set $\mathcal{V}$ such that $P$ holds whenever $x\notin\mathcal{V}$. So if $P$ is a property that holds for generic $x$, then in particular it holds for a set of full Lebesgue measure.

We consider input data

and output data

A parameterized model with parameter space $\mathbb{R}^{m}$ is a mapping

We overload notation and also define $F$ as a map from $\mathbb{R}^{m}\times\mathbb{R}^{d\times n}$ to $\mathbb{R}^{1\times n}$ by

Whether we are thinking of $F$ as a mapping on individual data points or on datasets will be clear from the context. We define the mean squared error loss $L\colon\mathbb{R}^{m}\times\mathbb{R}^{d\times n}\times\mathbb{R}^{1\times n}\to\mathbb{R}^{1}$ by

For a fixed dataset $(X,y)$, let $\mathcal{G}_{X,y}\subseteq\mathbb{R}^{m}$ denote the set of global minima of the loss $L$; that is,

If there exists $\theta^{*}\in\mathbb{R}^{m}$ such that $F(\theta^{*},X)=y$, then $L(\theta^{*},X,y)=0$, so $\theta^{*}$ is a global minimum. In such a case,

For a dataset $(X,y)$, we say that $\theta\in\mathbb{R}^{m}$ is a local minimum if there exists an open set $\mathcal{U}\subseteq\mathbb{R}^{m}$ containing $\theta$ such that $L(\theta,X,y)\leq L(\phi,X,y)$ for all $\phi\in\mathcal{U}$. We say that $\theta$ is a bad local minimum if it is a local minimum but not a global minimum.

A key indicator of the local optimization landscape of a neural network is the rank of the Jacobian of the map $F$ with respect to the parameters $\theta$. We will use the following observation.

Finally, in what follows given data $X$ we study activation regions in parameter space. Informally, these are sets of parameters which give rise to particular activation patterns of the neurons of a network over the dataset. We will define this notion formally for each setting we study in the subsequent sections.

Using algebraic techniques, we show that for generic $X$, the rank of the Jacobian is determined by $A$. Then by the above results, Lemma 2 concludes that for most activation regions the smooth critical points are global minima (see full details in Appendix A):

The takeaway of Theorem 5 is that most activation regions do no contain local minima. However, for a given dataset $X$ not all activation regions, each encoded by a binary matrix $A$, may be realizable by the network. Equivalently, $A$ is not realizable if and only if $\mathcal{S}^{A}_{X}=\emptyset$ and we call such activation regions empty. We therefore turn our attention to evaluating the number of non-empty activation regions. Different activation regions in parameter space are separated by the hyperplanes $\{W\colon\langle w^{(i)},x^{(j)}\rangle=0\}$, $i\in[d_{1}]$, $j\in[n]$. We say that a set of vectors in a $d$-dimensional space is in general position if any $k\leq d$ of them are linearly independent, which is a generic property. Standard results on hyperplane arrangements give the following proposition.

The formula provided above in Proposition 6 is equal to $2^{nd_{1}}$ if and only if $n\leq d$, and is $O(n^{dd_{1}})$ if $n>d$. We therefore observe that if $d$ is large in relation to $n$ and the data is in general position then by Proposition 6 most activation regions are non-empty. Thus we obtain the following corollary of Theorem 5.

More generally, for an arbitrary dataset that is not necessarily in general position the regions can be enumerated using a celebrated formula by Zaslavsky (1975), in terms of the intersection poset of the hyperplanes. Moreover, importantly one can show that the maximal number of non-empty regions is attained when the dataset is in general position.

From the above discussion we conclude we have a relatively good understanding of how many activation regions are non-empty. Notably, the number is the same for any dataset that is in general position. However, it is worth reflecting on the fact that the identity of the non-empty regions depends more closely on the specific dataset and is harder to catalogue. For a given dataset $X$ the non-empty regions correspond to the vertices of the Minkowski sum of the line segments with end points $0,x^{(j)}$, for $j\in[n]$, as can be inferred from results in tropical geometry (see Joswig, 2021).

This provides a sense of which activation regions are non-empty, depending on $X$. The explicit list of non-empty regions is known in the literature as the list of maximal covectors of an oriented matroid (see Björner et al., 1999), which can be interpreted as a combinatorial type of the dataset.

In Section 3 we showed that mild overparameterization suffices to ensure that most activation regions do not contain bad local minima. This result however does not discuss the relative scarcity of activation regions with global versus local minima. To this end in this section we take a closer look at the case of a single input dimension, $d_{0}=1$. Importantly, in the univariate setting we are able to entirely characterize the realizable (non-empty) activation regions.

Our strategy for proving Theorem 10 hinges on the following observation. For the sake of example, consider the step vectors $\xi^{(1,1)},\xi^{(2,1)},\ldots,\xi^{(n,1)}$. This set of vectors forms a basis of $\mathbb{R}^{n}$, so if each of these vectors was a row of the activation matrix $A$, it would have full rank. This observation generalizes to cases where some of the step vectors are taken to be $\xi^{(k,0)}$ instead of $\xi^{(k,1)}$. If enough step vectors are “collected” by rows of the activation matrix, it will be of full rank. We can interpret this condition in a probabilistic way. Suppose that the rows of $A$ are sampled randomly from the set of step vectors. We wish to determine the probability that after $d_{1}$ samples, enough step vectors have been sampled to cover a certain set. We use the following lemma.

This gives us a bound for the probability that a randomly sampled region is of full rank. We finally convert this into a combinatorial statement to obtain Theorem 10. For the details and a complete proof, see Appendix C.

In one-dimensional input space, the existence of global minima within a region requires similar conditions to the region having a full rank Jacobian. Both of them depend on having many different step vectors represented among the rows of the activation matrix. The condition we need to check for the existence of global minima is slightly more stringent, and depends on there being enough step vectors for both the positive and negative entries of $v$.

We provide the proof of this statement in Appendix C. To conclude, for shallow ReLU networks and univariate data we have shown that mild overparameterization suffices to ensure that the loss landscape is favorable in the sense that most activation regions contain a set of global minima of significant codimension and do not contain any local minima. Furthermore, considering both Sections 3 and 4, we have shown in both the one-dimensional and higher-dimensional cases that most activation regions have a full rank Jacobian and contain no bad local minima. This suggests that a large fraction of parameter space by volume should also have a full rank Jacobian. This indeed turns out to be the case and is discussed in Section 6. Finally, we highlight to the reader that we provide a discussion of the function space perspective of this setting in Appendix D.

We provide a proof of this statement in Appendix E.

Our main results so far have concerned the number of activation regions containing bad local versus global minima. Here we compute bounds for the volume of the union of these bad versus good activation regions, which recall are subsets of parameter space. Proofs of all results in this section are provided in Appendix F.

Here we empirically demonstrate that most regions of parameter space have a good optimization landscape by computing the rank of the Jacobian for two-layer neural networks. We initialize our network with random weights and biases sampled iid uniformly on $\left[-\frac{1}{\sqrt{d_{1}}},\frac{1}{\sqrt{d_{1}}}\right]$. We evaluate the network on a random dataset $X\in\mathbb{R}^{d_{0}\times n}$ whose entries are sampled iid Gaussian with mean 0 and variance 1. This gives us an activation region corresponding to the network evaluated at $X$, and we record the rank of the Jacobian of that matrix. For each choice of $d_{0},d_{1}$, and $n$, we run 100 trials and record the fraction of them which resulted in a Jacobian of full rank. The results are shown in Figures 1 and 2.

For different scalings of $n$ and $d_{0}$, we observe different minimal widths $d_{1}$ needed for the Jacobian to achieve full rank with high probability. Figure 1 suggests that the minimum value of $d_{1}$ needed to achieve full rank increases linearly in the dataset size $n$, and that the slope of this linear dependence decreases as the input dimension $d_{0}$ increases. This is exactly the behavior predicted by Theorem 2, which finds full rank regions for $d_{1}\gtrsim\frac{n}{d_{0}}$. Figure 2 operates in the regime $d_{0}\sim n$, and shows that the necessary hidden dimension $d_{1}$ remains constant in the dataset size $n$. This is again consistent with Theorem 5, whose bounds depend only on the ratio $\frac{n}{d_{0}}$. Further supporting experiments, including those involving real-world data, are provided in Appendix G.

In this work we studied the loss landscape of both shallow and deep ReLU networks. The key takeaway is that mildly overparameterization in terms of network width suffices to ensure the loss landscape is favorable in the sense that bad local minima exist in only a small fraction of parameter space. In particular, using random matrix theory and combinatorial techniques we showed that most activation regions have no bad differentiable local minima by determining which regions have a full rank Jacobian. For univariate data we further proved that most regions contain a high-dimensional set of global minimizers and also showed illustrated that the same takeaway is true when also considering potential bad non-differentiable local minima on the boundaries between regions. Finally the combinatorial approach we adopted allowed us to prove results independent of the specific choice of initialization of parameters, or on the distribution of the dataset. There are a number of directions for improvement. Notably we obtain our strongest results for shallow one-dimensional input case, where we have a concrete grasp of the possible activation patterns; the case $1<d_{1}<n$ remains open. We leave strengthening our results concerning deep networks and high dimensional data to future work, and hope that our contributions inspire further exploration to address them.