Achieve the Minimum Width of Neural Networks for Universal Approximation

• 1)

  Obtained the universal lower bound of width $w^{*}_{\min}$ for feed-forward neural networks (FNNs) that have universal approximation properties.

• 2)

  Achieved the critical width $w^{*}_{\min}$ by leaky-ReLU+ABS networks and UOE+FLOOR networks. (UOE is a continuous function which has universal ordering of extrema. It is introduced to handle $C$-UAP for one-dimensional functions. See Definition 7.)

• 3)

  Proposed a novel construction scheme from a differential geometry perspective that could deepen our understanding of UAP through topology theory.

To obtain the exact minimum width, one must verify the lower and upper bounds. Generally, the upper bounds are obtained by construction, while the lower bounds are obtained by counterexamples.

Lower bounds. For ReLU networks, Lu et al. (2017) utilized the disadvantage brought by the insufficient size of the dimensions and proved a lower bound $w_{\min}\geq d_{x}$ for $L^{1}$-UAP; Hanin & Sellke (2017) considered the compactness of the level set and proved a lower bound $w_{\min}\geq d_{x}+1$ for $C$-UAP. For monotone activation functions or its variants, Johnson (2019) noticed that functions represented by networks with width $d_{x}$ have unbounded level sets, and Beise & Da Cruz (2020) noticed that such functions on a compact domain $\mathcal{K}$ take their maximum value on the boundary $\partial\mathcal{K}$. These properties allow one to construct counterexamples and give a lower bound $w_{\min}\geq d_{x}+1$ for $C$-UAP. For general activation functions, Park et al. (2021) used the volume of simplex in the output space and gave a lower bound $w_{\min}\geq d_{y}$ for either $L^{p}$-UAP or $C$-UAP. Our universal lower bound, $w_{\min}\geq\max(d_{x},d_{y})$, is based on the insufficient size of the dimensions for both the input and output space, which combines the ideas from these references above.

Upper bounds. For ReLU networks, Lu et al. (2017) explicitly constructed a width-$(d_{x}+4)$ network by concatenating a series of blocks so that the whole network can be approximated by scale functions in $L^{1}(\mathbb{R}^{d_{x}},\mathbb{R})$ to any given accuracy. Hanin & Sellke (2017); Hanin (2019) constructed a width-$(d_{x}+d_{y})$ network using the max-min string approach to achieve $C$-UAP for functions on compact domains; Park et al. (2021) proposed an encoder-memorizer-decoder scheme that achieves the optimal bounds $w_{\min}=\max(d_{x}+1,d_{y})$ of the UAP for $L^{p}(\mathbb{R}^{d_{x}},\mathbb{R}^{d_{y}})$. For general activation functions, Kidger & Lyons (2020) proposed a register model construction that gives an upper bound $w_{\min}\leq d_{x}+d_{y}+1$ for $C$-UAP. Based on this result, Park et al. (2021) improved the upper bound to $w_{\min}\leq\max(d_{x}+2,d_{y}+1)$ for $L^{p}$-UAP. In this paper, we adopt the encoder-memorizer-decoder scheme to calculate the universal critical width for $C$-UAP by ReLU+FLOOR activation functions. However, the floor function is discontinuous. For $L^{p}$-UAP, we reach the critical width by leaky-ReLU, which is a continuous network using a novel scheme based on the approximation power of neural ODEs.

ResNet and neural ODEs. Although our original aim is the UAP for feed-forward neural networks, our construction is related to the neural ODEs and residual networks (ResNet, He et al. (2016)), which include skipping connections. Many studies, such as E (2017); Lu et al. (2018); Chen et al. (2018), have emphasized that ResNet can be regarded as the Euler discretization of neural ODEs. The approximation power of ResNet and neural ODEs have also been examined by researchers. To list a few, Li et al. (2022) gave a sufficient condition that covers most networks in practice so that the neural ODE/dynamic systems (without extra dimensions) process $L^{p}$-UAP for continuous functions, provided that the spatial dimension is larger than one; Ruiz-Balet & Zuazua (2021) obtained similar results focused on the case of one-hidden layer fields. Tabuada & Gharesifard (2020) obtained the $C$-UAP for monotone functions, and for continuous functions it was obtained by adding one extra spatial dimension. Recently, Duan et al. (2022) noticed that the FNN could also be a discretization of neural ODEs, which motivates us to construct networks achieving the critical width by inheriting the approximation power of neural ODEs. For the excluded dimension one, we design an approximation scheme with leaky-ReLU+ABS and UOE activation functions.

We formally state the main results and necessary notations in Section 2. The proof ideas are given in Section 3 4, and 5. In Section 3, we consider the case where $N=d_{x}=d_{y}=1$, which is basic for the high-dimensional cases. The construction is based on the properties of monotone functions. In Section 4, we prove the case where $N=d_{x}=d_{y}\geq 2$. The construction is based on the approximation power of neural ODEs. In Section 5, we consider the case where $d_{x}\neq d_{y}$ and discuss the case of more general activation functions. Finally, we conclude the paper in Section 6. All formal proofs of the results are presented in the Appendix.

The theorem indicates that leaky-ReLU networks achieve the critical width $w^{*}_{\min}=\max(d_{x},d_{y})$, except for the case of $d_{x}=d_{y}=1$. The idea is to consider the case where $d_{x}=d_{y}=d>1$ and let the network width equal $d$. According to the results of Duan et al. (2022), leaky-ReLU networks can approximate the flow map of neural ODEs. Thus, we can use the approximation power of neural ODEs to finish the proof. Li et al. (2022) proved that many neural ODEs could approximate continuous functions in the $L^{p}$ norm. This is based on the fact that orientation preserving diffeomorphisms can approximate continuous functions Brenier & Gangbo (2003).

The exclusion of dimension one is because of the monotonicity of leaky ReLU. When we add a nonmonotone activation function such as the absolute value function or sine function, the $L^{p}$-UAP at dimension one can be achieved.

$C$-UAP is more demanding than $L^{p}$-UAP. However, if the activation functions could include discontinuous functions, the same critical width $w^{*}_{\min}$ can be achieved. Following the encoder-memory-decoder approach in Park et al. (2021), the step function is replaced by the floor function, and one can obtain the minimal width $w_{\min}=\max(d_{x},2,d_{y})$.

Since ReLU and FLOOR are monotone functions, the $C$-UAP critical width $w^{*}_{\min}$ does not hold for $C([0,1],\mathbb{R})$. This seems to be the case even if we add ABS or SIN as an additional activator. However, it is still possible to use the UOE function (Definition 12).

Here, we give a sketch of the proof of Lemma 1, which states that $w^{*}_{\min}$ is a universal lower bound over all activation functions. Parts of Lemma 1 have been demonstrated in many papers, such as Park et al. (2021). Here, we give proof by two counterexamples that are simple and easy to understand from the topological perspective. It contains two cases: 1) there is a function $f^{*}$ that cannot be approximated by networks with width $w\leq d_{x}-1$; 2) there is a function $f^{*}$ that cannot be approximated by networks with width $w\leq d_{y}-1$. Figure 3(a)-(b) shows the counterexamples that illustrate the essence of the proof.

For the first case, $w\leq d_{x}-1$, we show that $f^{*}(x)=\|x\|^{2},x\in\mathcal{K}=[-2,2]^{d_{x}},$ is what we want; see Figure 3(a). In fact, we can relax the networks to a function $f(x)=\phi(Wx+b)$, where $Wx+b$ is a transformer from $\mathbb{R}^{d_{x}}$ to $\mathbb{R}^{d_{x}-1}$ and $\phi(x)$ could be any function. A consequence is that there exists a direction $v$ (set as the vector satisfying $Wv=0$, $\|v\|=1$) such that $f(x)=f(x+\lambda v)$ for all $\lambda\in\mathbb{R}$. Then, considering the sets $A=\{x:\|x\|\leq 0.1\}$ and $B=\{x:\|x-v\|\leq 0.1\}$, we have

Since the volume of $A$ is a fixed positive number, the inequality implies that even the $L^{1}$ approximation for $f^{*}$ is impossible. The case of the $L^{p}$ norm and the uniform norm is impossible as well.

For the second case, $w\leq d_{y}-1$, we show the example of $f^{*}$, which is the parametrized curve from 0 to 1 along the edge of the cubic, see Figure 3(b). Relaxing the networks to a function $f(x)=W\psi(x)+b$, $\psi(x)$ could be any function. Since the range of $f$ is in a hyperplane while $f^{*}$ has a positive distance to any hyperplane, the target $f^{*}$ cannot be approximated.

Now, we show that the lower bound $w_{\min}^{*}$ for $L^{p}$-UAP can be achieved by leaky-ReLU+ABS networks. Without loss of generality, we consider $\mathcal{K}=[0,1]^{d_{x}}$.

For any function $f^{*}$ in $L^{p}([0,1]^{d_{x}},\mathbb{R}^{d_{y}})$, we can extend it to a function $\tilde{f}^{*}$ in $L^{p}([0,1]^{d},\mathbb{R}^{d})$ by filling in zeros where $d=\max(d_{x},d_{y})=w^{*}_{\min}$. When $d_{x}>1$ or $d_{y}>1$, the $L^{p}$-UAP for leaky-ReLU networks with width $w^{*}_{\min}$ is obtained by using Corollary 11. Recall that by the Lemma 1, $w_{\min}^{*}$ is optimal, and we obtain our main result Theorem 2.

Combining the case of $d_{x}=d_{y}=d=1$ in Section 3, adding absolute function ABS as an additional activation function, we obtain Theorem 3.

Here, we use the encoder-memorizer-decoder approach proposed in Park et al. (2021) to achieve the minimum width. Without loss of generality, we consider the function class $C([0,1]^{d_{x}},[0,1]^{d_{y}})$. The encoder-memorizer-decoder approach includes three parts:

• 1)

  an encoder maps $[0,1]^{d_{x}}$ to $[0,1]$ which quantizes each coordinate of $x$ by a $K$-bit binary representation and concatenates the quantized coordinates into a single scalar value $\bar{x}$ having a $(d_{x}K)$-bit binary representation;

• 2)

  a memorizer maps each codeword $\bar{x}$ to its target codeword $\bar{y}$;

• 3)

  a decoder maps $\bar{y}$ to the quantized target that approximates the true target.

As illustrated in Figure 3(c), using the floor function instead of a step function, one can construct the encoder by FLOOR networks with width $d_{x}$ and the decoder by FLOOR networks with width $d_{y}$. The memorizer is a one-dimensional scalar function that can be approximated by ReLU networks with a width of two or UOE networks with a width of one. Therefore, the minimal widths $\max(d_{x},2,d_{y})$ and $\max(d_{x},d_{y})$ are obtained, which demonstrate Lemma 4 and Corollary 6, respectively.

Here, we emphasize that our universal bound of the minimal width is optimized over arbitrary activation functions. However, it cannot always be achieved when the activation functions are fixed. Here, we discuss the case of monotone activation functions.

If the activation functions are strictly monotone and continuous (such as leaky-ReLU), a width of at least $d_{x}+1$ is needed for $C$-UAP. This can be understood through topology theory. Leaky-ReLU, the nonsingular linear transformer, and its inverse are continuous and homeomorphic. Since compositions of homeomorphisms are also homeomorphisms, we have the following proposition: If $N=d_{x}=d_{y}=d$ and the weight matrix in leaky-ReLU networks are nonsingular, then the input-output map is a homeomorphism. Note that singular matrices can be approximated by nonsingular matrices; therefore, we can restrict the weight matrix in neural networks to the nonsingular case.

When $d_{x}\geq d_{y}$, we can reformulate the leaky-ReLU network as $f_{L}(x)=W_{L+1}\psi(x)+b_{L+1}$, where $\psi(x)$ is the homeomorphism. Note that considering the case where $d_{y}=1$ is sufficient, according to Hanin & Sellke (2017); Johnson (2019). They proved that the neural network width $d_{x}$ cannot approximate any scalar function with a level set containing a bounded path component. This can be easily understood from the perspective of topology theory. An example is to consider the function $f^{*}(x)=\|x\|^{2},x\in\mathcal{K}=[-2,2]^{d_{x}}$ shown in Figure 4.

The case where $d_{x}<d_{y}$. We present a simple example in Figure 5. The curve ‘4’ corresponding to a continuous function from $[0,1]\subset\mathbb{R}$ to $\mathbb{R}^{2}$ cannot be uniformly approximated. However, the $L^{p}$ approximation is still possible.