Memory capacity of neural networks with threshold and ReLU activations

In this paper we study a general layered, feedforward, fully connected neural architecture with arbitrarily many layers, arbitrarily many nodes in each layer, with either threshold or ReLU activation functions between all layers, and with the threshold activation function at the output node.

Readers unfamiliar with this terminology may think of a neural architecture as a computational device that can compute certain compositions of linear and nonlinear maps. Let us describe precisely the functions computable by a neural architecture. Some of the best studied and most popular nonlinear functions $\phi\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}\to\mathbb{R}$, or “activation functions”, include the threshold function and the rectified linear unit (ReLU), defined by

respectively.¹¹1It should be possible to extend out results for other activation functions. To keep the argument simple, we shall focus on the threshold and ReLU nonlinearities in this paper. We call a map pseudolinear if it is a composition of an affine transformation and a nonlinear transformation $\phi$ applied coordinate-wise. Thus, $\Phi\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n}\to\mathbb{R}^{m}$ is pseudolinear map if it can be expressed as

where $V$ is a $m\times n$ matrix of “weights”, $b\in\mathbb{R}^{m}$ is a vector of “biases”, and $\phi$ is either the threshold or ReLU function (1.1), which we apply to each coordinate of the vector $Wx-b$.

A neural architecture computes compositions of pseudolinear maps, i.e. functions $F\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n_{1}}\to\mathbb{R}$ of the type

where $\Phi_{1}\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n_{1}}\to\mathbb{R}^{n_{2}}$, $\Phi_{2}\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n_{2}}\to\mathbb{R}^{n_{3}}$, …, $\Phi_{L-1}\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n_{L-1}}\to\mathbb{R}^{n_{L}}$, $\Phi_{L}\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n_{L}}\to\mathbb{R}$ are pseudolinear maps. Each of maps $\Phi_{i}$ may be defined using either the threshold or ReLU function, and mix and match is allowed. However, for the purpose of this paper, we require the output function $\Phi_{L}\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n_{L}}\to\mathbb{R}$ to have the threshold activation.²²2General neural architectures used by practitioners and considered in the literature may have more than one output node and have other activation functions at the output node.

We regard the matrices $V$ and $b$ in the definition of each pseudolinear map $\Phi_{i}$ as free parameters of the given neural architecture. Varying these free parameters one can make a given neural architecture compute different functions $F\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n_{1}}\to\{0,1\}$. Let us denote the class of such functions computable by a given architecture by

A neural architecture can be visualized as a directed graph, which consists of $L$ layers each having $n_{i}$ nodes (or neurons), and one output node. Successive layers are connected by bipartite graphs, each of which represents a pseudolinear map $\Phi_{i}$. Each neuron is a little computational device. It sums all inputs from the neurons in the previous layer with certain weights, applies the activation function $\phi$ to the sum, and passes the output to neurons in the next layer. More specifically, the neuron determines if the sum of incoming signals from the previous layers exceeds a certain firing threshold $b$. If so, the neuron fires with either a signal of strength $1$ (if $\phi$ is the threshold activation function) or with strength proportional to the incoming signal (if $\phi$ is the ReLU activation).

When can a given neural architecture remember a given data? Suppose, for example, that we have $K$ digital pictures of cats and dogs, encoded as as vectors $x_{1},\ldots,x_{K}\in\mathbb{R}^{n_{1}}$, and labels $y_{1},\ldots,y_{K}\in\{0,1\}$ where $0$ stands for a cat and $1$ for a dog. Can we train a given neural architecture to memorize which images are cats and which are dogs? Equivalently, does there exist a function $F\in\mathcal{F}(n_{1},\ldots,n_{L},1)$ such that

A common belief is that this should happen for any sufficiently overparametrized network – an architecture that that has significantly more free parameters than the size of the training data. The free parameters of our neural architecture are the $n_{i-1}\times n_{i}$ weight matrices $V_{i}$ and the bias vectors $b_{i}\in\mathbb{R}^{n_{i}}$. The number of biases is negligible compared to the number of weights, and the number of free parameters is approximately the same as the number of connections³³3We dropped the number of output connections, which is negligible compared to $\overline{W}$.

Thus, one can wonder whether a general neural architecture is able to memorize the data as long as the number of connections is bigger than the size of the data, i.e. as long as

Motivated by this question, one can define memory capacity of a given architecture as the largest size of general data the architecture is able to memorize. In other words, the memory capacity is the largest $K$ such that for a general⁴⁴4One may sometimes wish to exclude some kinds of pathological data, so natural assumptions can be placed on the set of data points $x_{k}$. In this paper, for example, we consider unit and separated points $x_{k}$. set of points $x_{1},\ldots,x_{K}\in\mathbb{R}^{n_{1}}$ and for any labels $y_{1},\ldots,y_{K}\in\{0,1\}$ there exists a function $F\in\mathcal{F}(n_{1},\ldots,n_{L})$ that satisfies (1.2).

The memory capacity is clearly bounded above by the vc-dimension,⁵⁵5This is because the vc-dimension is the maximal $K$ for which there exist points $x_{1},\ldots,x_{K}\in\mathbb{R}^{n_{1}}$ so that for any labels $y_{1},\ldots,y_{K}\in\{0,1\}$ there exists a function $F\in\mathcal{F}(n_{1},\ldots,n_{L})$ that satisfies (1.2). The memory capacity requires any general set of points $x_{1},\ldots,x_{K}\in\mathbb{R}^{n_{1}}$ to succeed as above. which is $O(\overline{W}\log\overline{W})$ for neural architectures with threshold activation functions [7] and $O(L\overline{W}\log\overline{W})$ for neural architectures with ReLU activation functions [5]. Thus, our question is whether these bounds are tight – is memory capacity (approximately) proportional to $\overline{W}$?

A version of this question was raised by Baum [6] in 1988. Building on the earlier work of Cover [8], Baum studied the memory capacity of multilayer perceptrons, i.e. feedforward neural architectures with threshold activation functions. He first looked at the network architecture $[n,m,1]$ with one hidden layer consisting of $m$ nodes (and, as notation suggests, $n$ nodes in the hidden layer and one output node). Baum noticed that for data points $x_{k}$ in general position in $\mathbb{R}^{n}$, the memory capacity of the architecture $[n,m,1]$ is about $nm$, i.e. it is proportional to the number of connections. This is not difficult: general position guarantees that the hyperplane spanned by any subset of $n$ data points misses any other data points; this allows one to train each of the $m$ neurons in the hidden layer on its own batch of $n$ data points.

Baum then asked if the same phenomenon persists for deeper neural networks. He asked whether for large $K$ there exists a deep neural architecture with a total of $O(\sqrt{K})$ neurons in the hidden layers and with memory capacity at least $K$. Such result would demonstrate the benefit of depth. Indeed, we just saw that shallow architecture $[n,O(\sqrt{K}),1]$ has capacity just $n\sqrt{K}$, which would be smaller than the hypothetical capacity $K$ of deeper architectures for $n\ll K$.

There was no significant progress on Baum’s problem. As Mitchison and Durbin noted in 1989, “one significant difference between a single threshold unit and a multilayer network is that, in the latter case, the capacity can vary between sets of input vectors, even when the vectors are in general position” [17]. Attempting to count different functions $F$ that a deep network can realize on a given data set $(x_{k})$, Kowalczyk writes in 1997: “One of the complications arising here is that in contrast to the single neuron case even for perceptrons with two hidden units, the number of implementable dichotomies may be different for various $n$-tuples in general position… Extension of this result to the multilayer case is still an open problem” [15].

The memory capacity problem is more tractable for neural architectures in which the threshold activation is replaced by one of its continuous proxies such as ReLU, sigmoid, tanh, or polynomial activation functions. Such activations allow neurons to fire with variable, controllable amplitudes. Heuristically, this ability makes it possible to encode the training data very compactly into the firing amplitudes.

Yamasaki claimed without proof in 1993 that for the sigmoid activation $\phi(t)=1/(1+e^{-t})$ and for data in general position, the memory capacity of a general deep neural architecture is lower bounded $\overline{W}$, the number of connections [23]. A version of Yamasaki’s claim was proved in 2003 by Huang for arbitrary data and neural architectures with two hidden layers [13].

In 2016, Zhang et al. [25] gave a construction of an arbitrarily large (but not fully connected) neural architecture with ReLU activations and whose memory capacity is proportional to both the number of connections and the number of nodes. Hardt and Ma [12] gave a different construction of a residual network with similar properties.

Very recently, Yun et at. [24] removed the requirement that there be more nodes than data, showing that the memory capacity of networks with ReLU and tahn activation functions is proportional to the number of connections. Ge et al. [11] proved a similar result for polynomial activations.

Significant efforts was made in the last two years to justify why for overparametrized networks, the gradient descent and its variants could achieve $100\%$ capacity on the training data [10, 9, 16, 26, 1, 14, 27, 18, 20, 2, panigrahi2019effect]; see [21] for a survey of related developments.

Meanwhile, the original problem of Baum [6] – determine memory capacity of networks with threshold activations – has remained open. In contrast to the neurons with continuous activation functions, neurons with threshold activations either not fire at all of fire with the same unit amplitude. The strength of the incoming signal is lost when transmitted through such neurons, and it is not clear how the data can be encoded.

This is what makes Baum’s question hard. In this paper, we (almost) give a positive answer to this question.

Why “almost”? First, the size of the input layer $n_{1}$ should not affect the capacity bound and should be excluded from the count of the free parameters $\overline{W}$. To see this, consider, for example, the data points $x_{k}\in\mathbb{R}^{n_{1}}$ all laying on one line; with respect to such data, the network is equivalent to one with $n_{1}=1$. Next, ultra-narrow bottlenecks should be excluded at least for the threshold nonlinearity: for example, any layer with just $n_{i}=1$ node make the number of connections that occur in the further layers irrelevant as free parameters.

In our actual result, we make somewhat stronger assumptions: in counting connections, we exclude not only the first layer but also the second; we rule out all exponentially narrow bottlenecks (not just of size one); we assume that the data points $x_{k}$ are unit and separated; finally, we allow logarithmic factors.

In short, Theorem 1.1 states that the memory capacity of a general neural architecture with threshold or ReLU activations (or a mix thereof) is lower bounded by the number of the deep connections. This bound is independent of the depth, bottlenecks (up to exponentially narrow), or any other architectural details.

One can wonder about the necessity of the separation assumption (1.4). Can we just assume that $x_{k}$ are distinct? While this is true for ReLU and tanh activations [24], it is false for threshold activations. A moment’s thought reveals that any pseudolinear map from $\mathbb{R}$ to $\mathbb{R}^{m}$ transforms any line into a finite set such of cardinality $O(m)$. Thus, by pigeonhole principle, any map from layer $1$ to layer $2$ is non-injective on the set of $K$ data points $x_{k}$ – which makes it impossible to memorize some label assignments – unless $K=O(n_{2})$. In other words, if we just assume that the data points $x_{k}$ are distinct, the network must have at least as many nodes in the second layer as the number of data points. Still, the separation assumption (5.2) does not look tight and might be weakened.

Instead of requiring the network to memorize the training data with $100\%$ accuracy as in Theorem 1.1, one can ask to memorize just $1-\varepsilon$ fraction, or just a half of the training data correctly. This corresponds to a relaxed, or fractional memory capacity of neural architectures that was introduced by Cover in 1965 [8] and studied extensively afterwards.

To estimate fractional capacity of a given architecture, one needs to count all functions $F$ this architecture can realize on a given finite set points $x_{k}$. When this set is the Boolean cube $\{0,1\}^{n}$, this amounts to counting all Boolean functions $F\mathrel{\mathop{\mathchar 58\relax}}\{0,1\}^{n}\to\{0,1\}$ the architecture can realize. The binary logarithm of the number of all such Boolean functions was called (expressive) capacity by Baldi and the author [3, 4]. For a neural architecture with all threshold activations and $L$ layers, the expressive capacity is equivalent to the cubic polynomial in the sizes of layers $n_{i}$:

up to an absolute constant factor [4]. The factor $\min(n_{1},\ldots,n_{i})$ quantifies the effect of any bottlenecks that occur before layer $i$.

Similar results can be proved for the restricted expressive capacity where we count the functions $F$ the architecture can realize on a given finite set of $K$ points $x_{k}$ [4, Section 10.5]. Ideally, one might hope to find that all $2^{K}$ functions can be realized on a general $K$-element set, which would imply that the memory capacity is at least $K$. However, the current results on restricted expressive capacity are not tight enough to reach such conclusions.

Our construction of the enrichment maps $\Phi_{1}$ and $\Phi_{2}$ exploits sparsity. Both maps will have the form

where $G$ is an $n\times n$ Gaussian random matrix with all i.i.d. $N(0,1)$ coordinates, $g_{i}\sim N(0,I_{n})$ are independent standard normal random vectors, $\bar{b}$ is the vector whose all coordinates equal some value $b>0$.

If $b$ is large, $\Phi(x)$ is a sparse random vector with i.i.d. Bernoulli coordinates. A key heuristic is that independent sparse random vectors are almost orthogonal. Indeed, if $u$ and $u^{\prime}$ are independent random vectors in $\mathbb{R}^{n}$ whose all coordinates are Bernoulli(p), then $\operatorname{\mathbb{E}}\langle u,u^{\prime}\rangle=np^{2}$ while $\operatorname{\mathbb{E}}\|u\|_{2}=\operatorname{\mathbb{E}}\|u^{\prime}\|_{2}=np$, so we should expect

making $u$ and $u^{\prime}$ almost orthogonal for small $p$.

Unfortunately, the sparse random vectors $\Phi(x)$ and $\Phi(x^{\prime})$ are not independent unless $x$ and $x^{\prime}$ are exactly orthogonal. Nevertheless, our heuristic that sparsity induces orthogonality still works in this setting. To see this, let us check that the correlation of the coefficients $\Phi(x)$ and $\Phi(x^{\prime})$ is small even if $x$ and $x^{\prime}$ are far from being orthogonal. A standard asymptotic analysis of the tails of the normal distribution implies that

if $x$ and $x^{\prime}$ are unit and $\delta$-separated (Proposition 3.1), and

if $x$ and $x^{\prime}$ are unit and $\varepsilon$-orthogonal (Proposition 3.2).

Now we choose $b$ so that the coordinates of $\Phi(x)$ and $\Phi(x^{\prime})$ are sparse enough, i.e.

thus $b\sim\sqrt{\log n}$. Choose $\varepsilon$ sufficiently small to make the factor $\exp(2b^{2}\varepsilon)$ in (2.2) nearly constant, i.e.

Finally, we choose the separation threshold $\delta$ sufficiently large to make the factor $2\exp(-b^{2}\delta^{2}/8)$ in (2.1) less than $\varepsilon$, i.e.

this explains the form of separation condition in Theorem 1.1.

With these choices, (2.1) gives

confirming our claim that $\Phi(x)$ and $\Phi(x^{\prime})$ tend to be $\varepsilon$-orthogonal provided that $x$ and $x^{\prime}$ are $\delta$-separated. Similarly, (2.2) gives

confirming our claim that $\Phi(x)$ and $\Phi(x^{\prime})$ tend to be $(p=1/\sqrt{n})$-orthogonal provided that $x$ and $x^{\prime}$ are $\varepsilon$-orthogonal.

As we just saw, the enrichment process transforms our data points $x_{k}$ into $O(1/\sqrt{n})$-orthogonal vectors $v_{k}$. Let us now find a perception map $\Psi$ that can fit the labels to the data: $\Psi(v_{k})=y_{k}$.

Consider the random vector

where the signs are independently chosen with probability $1/2$ each. Then, separating the $k$-th term from the sum defining $w$ and assuming for simplicity that $v_{k}$ are unit vectors, we get

Taking the expectation over independent signs, we see that

where $y_{i}^{2}\in\{0,1\}$ and $\langle v_{i},v_{k}\rangle^{2}=O(1/n)$ by almost orthogonality. Hence

if $K\ll n$. This yields $\langle w,v_{k}\rangle=\pm y_{k}+o(1)$, or

Since $y_{k}\in\{0,1\}$, the “mirror perceptron”⁶⁶6The mirror perceptron requires not one but two neurons to implement, which is not a problem for us.

fits the data exactly: $\Psi(v_{k})=y_{k}$.

The same argument can be repeated for networks with variable sizes of layers, i.e. $[n,m,d,1]$. Interestingly, the enrichment works fine even if $n\ll m\ll d$, making the lower-dimensional data almost orthogonal even in very high dimensions. This explains why (moderate) bottlenecks – narrow layers – do not restrict memory capacity.

The argument we outlined allows the network $[n,m,d,1]$ to fit around $d$ data points, which is not very surprising, since we expect the memory capacity be proportional to the number of connections and not the number of nodes. However, the power of enrichment allows us to boost the capacity using the standard method of batch learning (or distributed learning).

Let us show, for example, how the network $[n,m,d,r,1]$ with three hidden layers can fit $K\sim dr$ data points $(x_{k},y_{k})$. Partition the data into $r$ batches each having $O(d)$ data points. Use our previous result to train each of the $r$ neurons in the fourth layer on its own batch of $O(d)$ points, while zeroing out all labels outside that batch. Then simply sum up the results. (The details are found in Theorem 7.1.)

This result can be extended to deeper networks using stacking, or unrolling a shallow architecture into a deep architecture, thereby trading width for depth. Figure 2 gives an illustration of stacking, and Theorem 7.2 provides the details. A similar stacking construction was employed in [4].

The success of stacking indicates that depth has no benefit formemorization purposes: a shallow architecture $[n,m,d,r,1]$ can memorize roughly as much data as any deep architecture with the same number of connections. It should be noted, however, that training algorithms commonly used by practitioners, i.e. variants of stochastic gradient descent, do not seem to lead to anything similar to stacking; this leaves the question of benefit of depth open.

As we explained in Section 2.1, the first two hidden layers of the network act as preconditioners: they transform the input vectors $x_{i}$ into vectors $v_{i}$ that are almost orthogonal. Almost orthogonality facilitates memorization process in the deeper layers, as we saw in Section 2.2.

The idea to keep the data well-conditioned as it passes through the network is not new. The learning rate of the stochastic gradient descent (which we are not dealing with here) is related to how well conditioned is the so-called gradient Gram matrix $H$. In the simplest scenario where the activation is ReLU and the network has one hidden layer of infinite size, $H$ is a $K\times K$ matrix with entries

Much effort was made recently to prove that $H$ is well-conditioned, i.e. its smallest singular value of $H$ is bounded away from zero, since this can be used to establishes a good convergence rate for the stochastic gradient descent, see the papers cited in Section 1.3 and especially [1, 10, 9, panigrahi2019effect]. However, existing results that prove that $H$ is well conditioned only hold for very overparametrized networks, requiring at least $n_{1}^{4}$ nodes in the hidden layer [panigrahi2019effect]. This is a much stronger overparametrization requirement than in our Theorem 1.1.

On the other hand, as opposed to many of the results quoted above, Theorem 1.1 does not shed any light on the behavior of stochastic gradient descent, the most popular method for training deep networks. Instead of training the weights, we explicitly compute them from the data. This allows us to avoid dealing with the gradient Gram matrix: our enrichment method provides an explicit way to make the data well conditioned. This is achieved by setting the biases high enough to enforce sparsity. It would be interesting to see if similar preconditioning guarantees can be achieved with small (or even zero) biases and thus without exploiting sparsity.

A different form of enrichment was developed recently in the paper [4] which showed that a neural network can compute a lot of different Boolean functions. Toward this goal, an enrichment map was implemented in the first hidden layer. The objective of this map is to transform the input set (the Boolean cube $\{0,1\}^{n}$) into a set $S\subset\{0,1\}^{m}$ on which there are lots of different threshold functions – so that the next layers can automatically compute lots of different Boolean functions. While the general goal of this enrichment map in [4] is the same as in the present paper – achieve a more robust data representation that is passed to deeper layers – the constructions of these two enrichment maps are quite different.

As we saw in the previous section, we utilized the first two hidden layers of the network to preprocess, or enrich, the data vectors $x_{k}$. This made us skip the sizes of the first two layers when we counted the number of connections $W$. If these vectors are already nice, no enrichment may be necessary, and we have a higher memory capacity.

Suppose, for example, that the data vectors $x_{k}$ in Theorem 1.1 are $O(1/\sqrt{n_{\infty}})$-orthogonal. Then, since no enrichment is needed in this case, the conclusion of the theorem holds with

which is the sum of all connections in the network.

If, on the other hand, the data vectors $x_{k}$ in Theorem 1.1 are only $O(1/\sqrt{\log n_{\infty}})$-orthogonal, just the second enrichment is needed, and so the conclusion of the theorem holds with

which is the sum of all connections between the non-input layers.

This make us wonder: can enrichment be always achieved in one step instead of two? Can one find a pseudolinear map $\Phi\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n}\to\mathbb{R}^{n}$ that transforms a given set of $\delta$-separated vectors $x_{k}$ (say, for $\delta=0.01$) into a set of $O(1/\sqrt{n})$-orthogonal vectors? If this is possible, we would not need to exclude the second layer from the parameter count, and Theorem 1.1 would hold for $W\mathrel{\mathop{\mathchar 58\relax}}=n_{2}n_{3}+\cdots+n_{L-1}n_{L}$.

A related question for further study is to find an optimal separation threshold $\delta$ in the assumption $\|x_{i}-x_{j}\|_{2}\geq\delta$ in Theorem 1.1, and to remove the assumption that $x_{k}$ be unit vectors. Both the logarithmic separation level of $\delta$ and the normalization requirement could be artifacts of the enrichment scheme we used.

There are several ways Theorem 1.1 could be extended. It should not be too difficult, for example, to allow the output layer have more than one node; such multi-output networks are used in classification problems with multiple classes.

Finally, it should be possible to extend the analysis for completely general activation functions. Threshold activations we treated are conceptually the hardest case, since they act as extreme quantizers that restrict the flow of information through the network in the most dramatic way.

In Section 3 we prove bounds (2.1) and (2.2) which control $\operatorname{\mathbb{E}}\Phi(x)_{i}\Phi(x^{\prime})_{i}$, the correlation of the coefficients of the coordinates of $\Phi(x)$ and $\Phi(x^{\prime})$. This immediately controls the expected inner product $\operatorname{\mathbb{E}}\langle\Phi(x),\Phi(x^{\prime})\rangle=\sum_{i}\operatorname{\mathbb{E}}\Phi(x)_{i}\Phi(x^{\prime})_{i}$. In Section 4 we develop a deviation inequality to make sure that the inner product $\langle\Phi(x),\Phi(x^{\prime})\rangle$ is close to its expectation with high probability. In Section 5, we take a union bound over all data points $x_{k}$ and thus control all inner products $\langle\Phi(x_{i}),\Phi(x_{j})\rangle$ simultaneously. This demonstrates how enrichment maps $\Phi$ make the data almost orthogonal – the property we outlined in Section 2.1. In Section 6, we construct a random perception map $\Psi$ as we outlined in Section 2.2. We combine enrichment and perception in Section 7 as we outlined in Section 2.3. We first prove a version of our main result for networks with three hidden layers (Theorem 7.1); then we stack shallow networks into an arbitrarily deep architecture proving a full version of our main result in Theorem 7.2.

In the rest of the paper, positive absolute constants will be denoted $C,c,C_{1},c_{1}$, etc. The notation $f(x)\lesssim g(x)$ means that $f(x)\leq Cg(x)$ for some absolute constant $C$ and for all values of parameter $x$. Similarly, $f(x)\asymp g(x)$ means that $cg(x)\leq f(x)\leq Cg(x)$ where $c$ is another positive absolute constant.

We call a map $E$ almost pseudolinear if $E(x)=\lambda\Phi(x)$ for some nonnegative constant $\lambda$ and some pseudolinear map $\Phi$. For the ReLU nonlinearity, almost pseudolinear maps are automatically pseudolinear, but for the threshold nonlinearity this is not necessarily the case.

Cauchy-Schwarz inequality gives a trivial bound

with equality when $x=x^{\prime}$. Our first result shows that if the vectors $x$ and $x^{\prime}$ are $\delta$-separated, this bound can be dramatically improved, and we have

We continue to study the covariance structure of the random process $Z_{x}$. If $x$ and $x^{\prime}$ are orthogonal, $Z_{x}$ and $Z_{x^{\prime}}$ are independent and we have

In this subsection, we show the stability of this equality. The result below implies that if $x$ and $x^{\prime}$ are almost orthogonal, namely $\mathinner{\!\left\lvert\langle x,x^{\prime}\rangle\right\rvert}\ll b^{-2}$, then

In order to prove this proposition, we first establish a more general stability property: