Exploring the Complexity of Deep Neural Networks through Functional Equivalence

The issue of redundancy or identification of parameterization of neural networks has been noted since 1990 in Hecht-Nielsen, (1990). Subsequent studies for neural networks with Tanh and sigmoid activations (Chen et al., , 1993; Fefferman and Markel, , 1993; Kůrkov’a and Kainen, , 1994) have proved that given the input-output mapping of a Tanh neural network, its architecture can be determined and weights are identified up to permutations and sign flips. Recently, the identifiability of parameterization in deep neural networks, particularly ReLU networks, has received considerable attention (Elbrächter et al., , 2019; Bui Thi Mai and Lampert, , 2020; Bona-Pellissier et al., , 2021; Dereich and Kassing, , 2022; Stock and Gribonval, , 2022; Grigsby et al., 2022b, ; Grigsby et al., 2022a, ). Most recently, Bui Thi Mai and Lampert, (2020) demonstrated that ReLU networks with non-increasing widths are identifiable up to permutation and scaling of weight matrices. With redundant parameterization, the weight space of deep neural networks can exhibit symmetric structures, which leads to implications for optimization (Neyshabur et al., 2015a, ; Badrinarayanan et al., 2015a, ; Stock et al., , 2019). These studies suggest that naive loss gradient is sensitive to reparameterization by scaling, and proposed alternative, scaling-invariant optimization procedures. In addition, the redundancy or identification properties of neural networks are closely related to the study of inverse stability (Elbrächter et al., , 2019; Rolnick and Kording, , 2020; Bona-Pellissier et al., , 2021; Petersen et al., , 2021; Stock and Gribonval, , 2022), which investigates the possibility for one to recover the parameters (weights and biases) of a neural network.

The complexity of neural networks in terms of their parameterization redundancy has received limited attention. Among the few relevant studies, Grigsby et al., 2022a and Grigsby et al., 2022b are worth mentioning. In Grigsby et al., 2022a , the authors investigated local and global notions of topological complexity for fully-connected feedforward ReLU neural network functions. On the other hand, Grigsby et al., 2022b defined the functional dimension of ReLU neural networks based on perturbations in parameter space. They explored functional redundancy and conditions under which the functional dimension reaches its theoretical maximum. However, it should be noted that these results on functional dimension do not directly translate into generalization error bounds for deep learning algorithms.

The complexity of a class of functions is closely related to generalization error, with larger complexities often leading to larger generalization error (Bartlett et al., , 2017; Mohri et al., , 2018). Various complexity upper bounds have been studied for deep neural networks using different measurements, such as Rademacher complexity (Neyshabur et al., 2015b, ; Golowich et al., , 2018; Li et al., , 2018), VC-dimension and Pseudo dimension (Baum and Haussler, , 1988; Goldberg and Jerrum, , 1993; Anthony et al., , 1999; Bartlett et al., , 2019), and covering number (Anthony et al., , 1999; Neyshabur et al., 2017b, ; Bartlett et al., , 2017; Lin and Zhang, , 2019). These measurements characterize the complexity of the class of neural networks and are influenced by hyperparameters like network depth, width, number of weights and bias vectors, and corresponding norm bounds. While these bounds are not directly comparable in magnitude, they are closely related and can be converted to facilitate comparisons (Anthony et al., , 1999; Mohri et al., , 2018).

We summarize our contributions as follows:

• 1.

  We make use of the permutation equivalence property to firstly obtain a tighter upper bound on the covering number of neural networks, which improves existing results by factorials of the network widths and provides unprecedented insights into the intricate relationship between network complexity and layer width.

• 2.

  We improve existing covering number bounds in the sense that our results hold for neural networks with bias vectors and general activation functions. Since bias terms are indispensable for the approximation power of neural networks, our results are useful in both theory and practice. Additionally, we express our bound explicitly in terms of the network’s width, depth, size, and the norm of the parameters.

• 3.

  We discuss the implications of our findings for understanding generalization and optimization. In particular, we found that overparameterized networks tend to be easier to train in the sense that increasing the width of neural networks leads to a vanishing volume of the effective parameter space.

The remainder of the paper is organized as follows. In section 2, we introduce the concept of functional equivalence and investigate the permutation invariance property of general feedforward neural networks. In section LABEL:sec_complexity, we derive novel covering number bounds for shallow and deep neural networks by exploiting the permutation invariance property and compare our results with existing ones. In section 3, we discuss the extension to convolutional, residual and attention-based networks. In section 4, we demonstrate the theoretical implication of permutation invariance on the optimization complexity in deep learning and discuss the implications of our results on generalization. Finally, we discuss the limitations of this study and future research directions in section 5. All technical proofs are included in the Appendix.

The remainder of the paper is organized as follows. In section 2, we introduce the concept of functional equivalence and investigate the permutation invariance property of general feedforward neural networks. In section LABEL:sec_complexity, we derive novel covering number bounds for shallow and deep neural networks by exploiting the permutation invariance property and compare our results with existing ones. In section 3, we discuss the extension to convolutional, residual and attention-based networks. In section 4, we demonstrate the theoretical implication of permutation invariance on the optimization complexity in deep learning and discuss the implications of our results on generalization. Finally, we discuss the limitations of this study and future research directions in section 5. All technical proofs are included in the Appendix.

It is well-known that a shallow neural network with a single hidden layer has universal approximation properties (Cybenko, , 1989; Hornik, , 1991) and is sufficient for many learning tasks (Hanin and Rolnick, , 2019; Hertrich et al., , 2021). We begin our investigation by considering shallow neural networks of the form

where the total number of parameters is given by $\mathcal{S}=(d_{0}+2)\times d_{1}+1$. By Theorem 1, for any $\theta=(W^{(1)},b^{(1)},W^{(2)},b^{(2)})\in\Theta$ and any permutation matrix $P$, $\tilde{\theta}=(PW^{(1)},Pb^{(1)},W^{(2)}P^{\top},b^{(2)})\in\Theta$ will produce the same input-out function. Actually, permutation invariance leads to equivalence classes of parameters that yield the same realization, and we can obtain a set of representatives from equivalence classes. A canonical choice is

where the set of representatives is by restricting the bias vector $b^{(1)}=(b^{(1)}_{1},\ldots,b^{(1)}_{d_{1}})^{\top}$ to have descending components. Alternatively, we can sort the first component of the rows of $W^{(1)}$ to obtain a set of representatives. It is worth mentioning that $\Theta_{0}$ may not be the minimal set of representatives since there may be other symmetries within $\Theta_{0}$. We did not further utilize these additional symmetries to reduce $\Theta_{0}$ to be smaller since other symmetries can depend on the specific properties of the activation functions in the neural networks. In this work, we specifically employ permutation invariance (which holds for networks with any activations) to obtain a representative set $\Theta_{0}$.

The set of representatives $\Theta_{0}$ has two important properties

• •

  Neural networks $\{f(\cdot;\theta):\theta\in\Theta_{0}\}$ parameterized by $\Theta_{0}$ contains all the functions in $\{f(\cdot;\theta):\theta\in\Theta\}$, i.e.,

  $$\{f(\cdot;\theta):\theta\in\Theta_{0}\}=\{f(\cdot;\theta):\theta\in\Theta\}.$$

• •

  The volume (in terms of Lebesgue measure) of the set of representatives $\Theta_{0}$ is $(1/d_{1}!)$ times smaller that that of the parameter space $\Theta$, i.e.,

  $${\rm Volume}(\Theta)=d_{1}!\times{\rm Volume}(\Theta_{0}).$$

The first property holds naturally since any parameter $\theta\in\Theta$ has a permuted version in $\Theta_{0}$. Regarding the second property, we note that both $\Theta$ and $\Theta_{0}$ belong to the Euclidean space $\mathbb{R}^{\mathcal{S}}$ (where $\mathcal{S}$ denotes the number of parameters), then their volumes in terms of Lebesgue measure can be calculated. For any parameter $\theta$ with distinct components in the bias vector $b^{(1)}=(b^{(1)}_{1},\ldots,b^{(1)}_{d_{1}})^{\top}$, permutation of the bias vector and corresponding weights can lead to $(d_{1}!)$ distinct equivalent parameters $\theta_{1},\ldots,\theta_{d_{1}!}$. Then this implies a $(d_{1}!)$ times smaller volume of representative set $\Theta_{0}$ compared to that of $\Theta$.

Subsequently, the two properties suggest that $\Theta_{0}$ can be a representative parameterization of neural networks when they are viewed only as input-output functions. Based on these observations, we derive improved complexities of the class of neural networks in terms of its covering number.

The covering number $\mathcal{N}(\mathcal{F},\epsilon,\|\cdot\|_{\infty})$ provides a quantitative measure of the complexity of the class of functions $\mathcal{F}$ under the supremum norm, with smaller values indicating simpler classes. Covering numbers, along with Rademacher complexity, VC dimension, and Pseudo dimension, are essential complexity measures in the analysis of learning theories and in estimating generalization errors. Although these measures are different, they are correlated with each other, and we introduce the detailed correlations in Appendix.

Our upper bound on the covering number firstly takes advantage of permutation invariance, resulting in a reduced complexity (by a factorial term $d_{1}!$ in the denominator) compared to existing studies (Neyshabur et al., 2015b, ; Bartlett et al., , 2017; Neyshabur et al., 2017b, ; Neyshabur, , 2017; Lin and Zhang, , 2019). The factorial reduction $d_{!}!$ can be significant. For instance, for a shallow ReLU network with a hidden dimension of $d_{1}=128$, the factorial $128!\approx 10^{215}$, which is far larger than $10^{82}$, the upper estimate on the number of atoms in the known universe. This reduction can be substantial and can enhance theoretical analysis and results that rely on covering numbers. In addition, it’s worth noting that bounds in Theorem 1 holds true for networks with bias vectors. This is also an improvement over existing studies that didn’t consider bias vectors in neural networks, since bias terms are crucial for the approximation power of neural networks (Yarotsky, , 2017; Lu et al., 2021b, ; Shen et al., , 2022).

Stirling’s formula can be used to approximate the factorial term as $\sqrt{2\pi d_{1}}(d_{1}/e)^{d_{1}}\exp(1/(12d_{1}+1))<d_{1}!<\sqrt{2\pi d_{1}}(d_{1}/e)^{d_{1}}\exp(1/12d_{1})$ when $d_{1}\geq 1$. This reduces the covering number approximately by a factor of $(d_{1}/e)^{d_{1}}$ and the bound in (5) is $(C_{B,B_{x},d_{0},\rho}/\epsilon)^{\mathcal{S}}\times d_{1}^{\mathcal{S}-\mathcal{U}}$, where $\mathcal{U}=d_{1}$ denotes the number of hidden neurons and $C_{B,B_{x},d_{0},\rho}>0$ is a constant depending only on $B,B_{x},d_{0}$ and $\rho$. Notably, $\mathcal{S}-\mathcal{U}$ is the number of weights (excluding bias), and the reduced bound is basically that for a single neural network without bias vectors and considering permutation invariance. Lastly, we note that increasing the number of neurons in a shallow neural network enlarges its approximation power (Lu et al., , 2017; Ongie et al., , 2019), but at a smaller increase in complexity according to our results.

For deep neural networks, we can also analyze its effective parameter space based on permutation invariance properties. By Theorem 1, a set of representatives $\Theta_{0}=\Theta_{0}^{(1)}\times\Theta_{0}^{(2)}\times\cdots\times\Theta_{0}^{(L)}\times\Theta_{0}^{(L+1)}$ can be constructed where

Then we can obtain an upper bound of the covering number of deep feedforward neural networks.

Theorem 2 provides a novel upper bound for the covering number of deep neural networks based on permutation invariance, which reduces the complexity compared to previous results (Neyshabur et al., 2015b, ; Neyshabur et al., 2017b, ; Bartlett et al., , 2017; Lin and Zhang, , 2019) by approximately a factor of $(d_{1}!d_{2}!\cdots d_{L}!)$. According to Theorem 2, increasing the depth of a neural network increases its complexity. However, it is interesting to note that the increased hidden layer $l$ will have a $(d_{l}!)$ discount on the complexity. If the hidden layers have equal width ($d=d_{1}=\cdots=d_{L}$), the bound reduces to $(C_{B,B_{x},d_{0},\rho}/\epsilon)^{\mathcal{S}}\times d^{\mathcal{S}-\mathcal{U}}$, where $\mathcal{U}=Ld$ denotes the number of hidden neurons and $C_{B,B_{x},d_{0},\rho}>0$ is a constant depending only on $B,B_{x},d_{0}$ and $\rho_{i},i=1,\ldots,L$. As with shallow neural networks, the improved rate $(\mathcal{S}-\mathcal{U})$ denotes the number of weights but excluding biases, which assures the approximation power of neural networks, but grows its complexity in a rate free of number of bias.

The complexity upper bounds in terms of covering number on the class of deep neural networks have been studied in Anthony et al., (1999); Neyshabur et al., 2017b ; Bartlett et al., (2017); Lin and Zhang, (2019). These results are proved by using similar approaches of mathematical induction (e.g. Lemma A.7 in Bartlett et al., (2017), Lemma 2 in Neyshabur et al., 2017b and Lemma 14 of Lin and Zhang, (2019)). Compared with these results, we improve upon their results in three ways.

• •

  First, we consider the generally defined neural networks where the bias vector is allowed to appear. Bias terms are indispensable for the approximation power of neural networks (Yarotsky, , 2017; Lu et al., 2021b, ; Shen et al., , 2022). Neural networks without bias vectors may be disqualified in theory and practice.

• •

  Second, we explicitly express our bound in terms of the width, depth, and size of the neural network, as well as the infinity norm of the parameters.

• •

  Third, we utilize the permutation equivalence property to obtain a tighter upper bound on the covering number. Notably, this bound improved existing results by a factorial of the network widths, providing unprecedented insights into the intricate relationship between network complexity and layer width.

Various complexity upper bounds for deep neural networks in other measurements have also been studied, including Rademacher complexity (Neyshabur et al., 2015b, ; Golowich et al., , 2018; Li et al., , 2018), VC-dimension, and Pseudo dimension (Baum and Haussler, , 1988; Goldberg and Jerrum, , 1993; Anthony et al., , 1999; Bartlett et al., , 2019). Our bounds in terms of covering number are not directly comparable with these measurements. To make a comparison with these other measurements, we convert the upper bounds into metric entropy, which is equivalent to the logarithm of the covering number $\log(\mathcal{N}(\mathcal{F},\epsilon,\|\cdot\|_{\infty}))$. Specially, let $\bar{\rho}=\Pi_{j=1}^{L}\rho_{j}$ products of Lipschitz constants of activation functions and $\bar{s}=\Pi_{j=1}^{L}s_{j}$ products of spectral norms of hidden layer weight matrices Bartlett et al., (2017) derived an spectral-norm based bound of metric entropy $B_{x}^{2}(\bar{\rho}\bar{s})^{2}\mathcal{U}\log(W)/\epsilon^{2}$, following which Neyshabur et al., 2017b obtained $B_{x}^{2}(\bar{\rho}\bar{s})^{2}\mathcal{S}L^{2}\log(WL)/\epsilon^{2}$ and Lin and Zhang, (2019) obtained $B_{x}(\bar{\rho}\bar{s})\mathcal{S}^{2}L/\epsilon$. Based on Theorem 12.2 in Anthony et al., (1999), the Pseudo dimension bound in Bartlett et al., (2017) leads to $L\mathcal{S}\log(\mathcal{S})\log(\bar{\rho}\bar{s}B_{x}/\epsilon)$. Lastly, our results in Theorem 2 can be presented as $L\mathcal{S}\log(\bar{\rho}\bar{s}B_{x}^{1/L}/((d_{1}!\cdots d_{L}!)^{\mathcal{S}}\epsilon)^{1/L})$ by letting $s_{i}:=B\sqrt{d_{i}d_{i-1}}$ given in our setting. It is important to note that the quantity $B\sqrt{d_{i}d_{i-1}}$ can provide an upper bound for the spectral norm $s_{i}$, but the reverse is not necessarily true. We cannot ensure Theorem 2 to still be true by directly substituting $B\sqrt{d_{i}d_{i-1}}$ with $s_{i}$ in the theorem. In other words, if the covering number bounds are derived in terms of the spectral norm, the reduction factor $d_{1}!\cdots d_{L}!$ may not be obtained, and additional parameters like matrix norm bounds would appear in the upper bound(Bartlett et al., , 2017). We would also mention that even our results improved over the existing ones, all these bounds scale with number of parameters $\mathcal{S}$ and can still result vacuous bounds in error analysis with extremely over-parametrized settings. We present a detailed comparison of the results in Table 1.

Convolutional neural networks (CNNs) are featured by the utilization of convolution and pooling layers. In a convolution layer, the input is convolved with parameter-carrying filters, resembling a linear layer with a sparse weight matrix. A pooling layer is commonly employed for downsampling and summarizing input feature information. It partitions the input into non-overlapping regions and applies a pooling operation to each region. The most prevalent types of pooling operations include max/min/avg pooling, which retain the maximum, minimum, or average value within each region.

As demonstrated in Example 1-3, scaling, sign flip, and permutation equivalence directly apply to convolution layers (linear layer with sparse weight matrix). We also extend the permutation equivalence within pooling regions as follows.

Permutation equivalence within non-overlapping regions in CNNs preserves max/min/avg values, eliminating the need for cancel-off operations in subsequent layers. This further allows for the derivation of complexity bounds of CNNs by identifying their effective parameter space, similar to Theorem 2. In addition, the results on the connections between CNNs and feed-forward networks can be naturally utilized to improve the covering number bounds and capacity estimates (Fang et al., , 2020; Mao et al., , 2021), e.g., (Zhou, 2020b, ; Zhou, 2020a, ) showed that fully connected deep ReLU networks can be realized by deep CNNs with the same order of network parameter numbers,.

Residual Networks are a type of deep CNN architecture that has a significant impact on computer visions (He et al., , 2016). The key feature of ResNet is the use of skip connections that enable networks to learn residual mappings and bypass layers, leading to very deep but trainable networks. Mathematically, a residual layer $f(x;\theta)=x+\text{F}(x;\theta)$ outputs the summation of the input $x$ and its transformation $\text{F}(x;\theta)$. The $F(\cdot;\theta)$ can be any transformation that maps $x$ to the space of itself, and it defines the residual layer. Then the equivalence of $F$ implies that of the residual layer.

Attention-based models, well known for BERT, GPT and many others, have been successful in natural language processing and computer vision tasks (Vaswani et al., , 2017; Devlin et al., , 2018; Radford et al., , 2018). It utilizes an attention mechanism to focus on relevant parts of the input data. Here we focus on the self-attention module due to its effectiveness. Let $X_{n\times d}$ denote the input of a $n$-sequence of $d$-dimensional embeddings, and let $W^{Q}_{d\times d_{q}},W^{K}_{d\times d_{k}}$ and $W^{V}_{d\times d_{v}}$ be the weight matrices where $d_{q}=d_{k}$. Then the self-attention map outputs

where the $Softmax(\cdot)$ is applied to each row of its input and $A^{\prime}$ denotes the transpose of a matrix $A$.

For the attention module, there is no activation function between the key and query matrices. The relevant symmetry can be considered for any equivalent linear maps $W^{Q}(W^{K})^{\prime}$. In addition, the output of $Softmax$ operator is invariant to the row shift of its input, which also leaves the possibility to further reduce the complexity of attention modules to understand the overparameterization of large language models.