On the training of sparse and dense deep neural networks: less parameters, same performance.

Automated learning from data via deep neural networks bishop_pattern_2011 ; cover_elements_1991 ; hastie2009elements ; hundred is becoming popular in an ever-increasing number of applications sutton2018reinforcement ; graves2013speech ; sebe2005machine ; grigorescu2020survey . Systems can learn from data, by identifying distinctive features which form the basis of decision making, with minimal human intervention. The weights, which link adjacent nodes across feedforward architectures, follow the optimisation algorithm and store the information needed for the trained network to perform the assigned tasks with unprecedented fidelity chen2014big ; meyers2008using ; caponetti2011biologically . A radically new approach to the training of a deep neural network has been recently proposed which anchors the process to reciprocal space, rather than to the space of the nodes giambagli2021 . Reformulating the learning in reciprocal space enables one to shape key collective modes, the eigenvectors, which are implicated in the process of progressive embedding, from the input layer to the detection point. Even more interestingly, one can assume the eigenmodes of the inter-layer transfer operator to align along suitable random directions and identify the associated eigenvalues as target for the learning scheme. This results in a dramatic compression of the training parameters space, yielding accuracies which are superior to those attained with conventional methods restricted to operate with an identical number of tunable parameters. Nonetheless, neural networks trained in the space of nodes with no restrictions on the set of adjusted weights, achieve better classification scores, as compared to their spectral homologues with quenched eigendirections. In the former case, the number of free parameters grows as the product of the sizes of adjacent layer pairs, thus quadratically in terms of hosted neurons. In the latter, the number of free parameters increases linearly with the size of the layers (hence with the number of neurons), when the eigenvalues are solely allowed to change. Also training the eigenvectors amounts to dealing with a set of free parameters equivalent to that employed when the learning is carried out in direct space: in this case, the two methods yield performances which are therefore comparable.

Starting from this setting, we begin by discussing a straightforward generalisation of the spectral learning scheme presented in giambagli2021 , which proves however effective in securing a significant improvement on the recorded classification scores, while still optimising a number of parameters which scales linearly with the size of the network. The proposed generalisation paves the way to a biomimetic interpretation of the spectral training scheme. The eigenvalues can be tuned so as to magnify/damp the contribution associated to the input nodes. At the same time, they modulate the excitability of the receiving nodes, as a function of the local field. This latter effect is reminiscent of the homeostatic plasticity Surmeier2004 as displayed by living neurons. Further, we will show that the residual gap between conventional and spectral trainings methods can be eventually filled by resorting to apt decompositions of the non trivial block of the eigenvectors matrix, which place the emphasis on a limited set of collective variables. Finally, we will prove that working in reciprocal space turns out to be by far more performant, when aiming at training sparse neural networks. Because of the improvement in terms of computational load, and due to the advantage of operating with collective target variables as we will make clear in the following, it is surmised that modified spectral learning of the type here discussed should be considered as a viable standard for deep neural networks training in artificial intelligence applications.

To test the effectiveness of the proposed method we will consider classification tasks operated on three distinct database of images. The first is the celebrated MNIST database of handwritten digits lecun1998mnist , the second is Fashion-MNIST (F-MNIST), a dataset of Zalando’s article images, the third is CIFAR-10 a collection of images from different classes (airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks). In all considered cases, use can be made of a deep neural network to perform the sought classification, namely to automatically assign the image supplied as an input to the class it belongs to. The neural network is customarily trained via standard backpropagation algorithms to tune the weights that connect consecutive stacks of the multi-layered architecture. The assigned weights, target of the optimisation procedure, bear the information needed to allocate the examined images to their reference category.

Consider a deep feedforward network made of $\ell$ distinct layers and label each layer with the progressive index $i$ $(=1,...,\ell)$. Denote by $N_{i}$ the number of computing units, the neurons, that belong to layer $i$. The total number of parameters that one seeks to optimise in a dense neural network setting (all neurons of any given layer with $i<\ell-1$ are linked to every neurons of the adjacent layer) equals $\sum_{i=1}^{\ell-1}N_{i}N_{i+1}$, when omitting additional bias. As we shall prove in the following, impressive performance can be also achieved by pursuing a markedly different procedure, which requires acting on just $N_{1}+N_{\ell}+2\sum_{i=2}^{\ell-1}N_{i}$ free parameters (not including bias). To this end, let us begin by reviewing the essence of spectral learning method as set forth in giambagli2021 .

Introduce $N=\sum_{i=1}^{\ell}N_{i}$ and create a column vector $\vec{n}_{1}$, of size $N$, whose first $N_{1}$ entries are the intensities (from the top-left to the bottom-right, moving horizontally) as displayed on the pixels of the input image. All other entries of $\vec{n}_{1}$ are set to zero. The ultimate goal is to transform $\vec{n}_{1}$ into an output vector $\vec{n}_{\ell}$ , of size $N$, whose last $N_{\ell}$ elements reflect the intensities of the output nodes where reading takes eventually place. This is achieved with a nested sequence of linear transformations, as exemplified in the following. Consider the generic vector $\vec{n}_{k}$, with $k=1,...,\ell-1$, as obtained after $k$ compositions of the above procedure. This latter vector undergoes a linear transformation to yield $\vec{n}_{k+1}={\mathbf{A}}^{(k)}\vec{n}_{k}$. Further, $\vec{n}_{k+1}$ is processed via a suitably defined non-linear function, denoted by $f\left(\cdot,\beta_{k}\right)$, where $\beta_{k}$ stands for an optional bias. Focus now on ${\mathbf{A}}^{(k)}$, a $N\times N$ matrix with a rather specific structure, as we will highlight hereafter. Posit ${\mathbf{A}}^{(k)}={\mathbf{\Phi}}^{(k)}{\mathbf{\Lambda}}^{(k)}\left({\mathbf{\Phi}}^{(k)}\right)^{-1}$, by invoking a spectral decomposition. ${\mathbf{\Lambda}}^{(k)}$ is the diagonal matrix of the eigenvalues of ${\mathbf{A}}^{(k)}$. By construction we impose, $\left({\mathbf{\Lambda}}^{(k)}\right)_{jj}=1$ for $j<\sum_{i=1}^{k}N_{i}$ and $j>\sum_{i=1}^{k+1}N_{i}$. The remaining $N_{k}$ elements are initially set to random numbers, e.g. extracted from a uniform distribution, and define the target of the learning scheme ¹¹1The only noticeable exception is when $k=1$. In this case, the first $N_{1}$ diagonal elements of ${\mathbf{\Lambda}}^{(1)}$ take part to the training.. Back to the spectral decomposition of ${\mathbf{A}}^{(k)}$, ${\mathbf{\Phi}}^{(k)}$ is assumed to be the identity matrix $\mathbb{1}_{N\times N}$, with the inclusion of a sub-diagonal rectangular block $\phi$^(k) of size $N_{k+1}\times N_{k}$. This choice corresponds to dealing with a feedforward arrangements of nested layers. A straightforward calculation returns $\left({\mathbf{\Phi}}^{(k)}\right)^{-1}=2\mathbb{1}_{N\times N}-{\mathbf{\Phi}}^{(k)}$, which readily yields ${\mathbf{A}}^{(k)}={\mathbf{\Phi}}^{(k)}{\mathbf{\Lambda}}^{(k)}\left(2\mathbb{1}_{N\times N}-{\mathbf{\Phi}}^{(k)}\right)$. In the simplest setting that we shall inspect in the following, the off-diagonal elements of matrix ${\mathbf{\Phi}}^{(k)}$ are frozen to nominal values, selected at random from a given distribution. In this minimal version, the spectral decomposition of the transfer operators ${\mathbf{A}}^{(k)}$ enables one to isolate a total of $N=\sum_{i=1}^{\ell}N_{i}$ adjustable parameters, the full collection of non trivial eigenvalues, which can be self-consistently trained. To implement the learning scheme on these premises, we consider $\vec{n}_{\ell}$, the image on the output layer of the input vector $\vec{n}_{1}$:

and calculate $\vec{z}=\sigma(\vec{n}_{\ell})$ where $\sigma(\cdot)$ stands for the softmax operation. We then introduce the categorical cross-entropy loss function $\text{CE}(l(\vec{n}_{1}),\vec{z})$ where the quantity $l(\vec{n}_{1})$ identifies the label attached to $\vec{n}_{1}$ reflecting the category to which it belongs via one-hot encoding aggarwal2018NNdeeplearn . More specifically, the $k$-th elements of vector $l(\vec{n}_{1})$ is equal to unit (the other entries being identically equal to zero) if the image supplied as an input is associated to the class of items grouped under label $k$.

The loss function can be minimized by acting on a limited set of free parameters, the collection of $N$ non trivial eigenvalues of matrices $\mathbf{A}_{k}$ (i.e. $N_{1}+N_{2}$ eigenvalues of $\mathbf{A}^{(1)}$, $N_{3}$ eigenvalues of $\mathbf{A}^{(2)}$,…, $N_{\ell}$ eigenvalues of $\mathbf{A}^{(\ell-1)}$). In principle, the sub-diagonal blocks $\phi$^(k) (the non orthogonal entries of the basis that diagonalises $\mathbf{A}^{(k)}$) can be optimised in parallel, but this choice nullifies the gain in terms of parameters containment, as achieved via spectral decomposition, when the eigenvalues get solely modulated. The remaining part of this Letter is entirely devoted to overcoming this limitation, while securing the decisive enhancement of the neural network’s performance.

The first idea, as effective as it is simple, is to extend the set of trainable eigenvalues. When mapping layer $k$ into layer $k+1$, we can in principle act on $N_{k}+N_{k+1}$ eigenvalues, without restricting the training to the $N_{k+1}$ elements, which were identified as the sole target of the spectral method in its original conception (except for the first mapping, from the input layer to its adjacent counterpart). As we shall clarify in the following, the eigenvalues can be trained twice, depending on whether they originate from incoming or outcoming nodes, along the successive arrangement of nested layers. The global number of trainable parameters is hence $N_{1}+N_{\ell}+2\sum_{i=2}^{\ell-1}N_{i}$, as anticipated above. A straightforward calculation, carried out in the annexed Supplementary Information, returns a closed analytical expression for $w^{(k)}_{ij}$, the weights of the edges linking nodes $i$ and $j$ in direct space, as a function of the underlying spectral quantities. In formulae:

where $l(i)=\sum_{s=1}^{k}N_{s}+i$ and $m(j)=\sum_{s=1}^{k-1}N_{s}+j$, with $i\in\left(1,...,N_{k+1}\right)$ and $j\in\left(1,...,N_{k}\right)$. More specifically, $j$ runs on the nodes at the departure layer ($k$), whereas $i$ identifies those sitting at destination (layer $k+1$). In the above expression, $\lambda^{(k)}_{m(j)}$ stand for the first $N_{k}$ eigenvalues of ${\mathbf{\Lambda}}^{(k)}$. The remaining $N_{k+1}$ eigenvalues are labelled $\lambda^{(k)}_{l(i)}$. To help comprehension denote by $x^{(k)}_{j}$ the activity on nodes $j$. Then,

The eigenvalues $\lambda^{(k)}_{m(j)}$ modulate the density at the origin, while $\lambda^{(k)}_{l(i)}$ appears to regulate the local node’s excitability relative to the network activity in its neighbourhood. This is the artificial analogue of the homeostatic plasticity, the strategy implemented by living neurons to maintain the synaptic basis for learning, respiration, and locomotion Surmeier2004 .

To illustrate the effectiveness of the proposed methodology we make reference to Fig. 1, which summarises a first set of results obtained for MNIST. To keep the analysis as simple as possible we have here chosen to deal with $\ell=3$. The sizes of the input ($N_{1}$) and output ($N_{3}$) layers are set by the specificity of the considered dataset. Conversely, the size of the intermediate layer ($N_{2}$) can be changed at will. We then monitor the relative accuracy, i.e. the accuracy displayed by the deep neural networks trained according to different strategies, normalised to the accuracy achieved with an identical network trained with conventional methods. In the upper panel of Fig. 1, the performance of the neural networks trained via the modified spectral strategy (referred to as to Spectral) is displayed in blue (triangles). The recorded accuracy is satisfactory (about 90% of that obtained with usual means and few percent more than that obtained with the spectral method of original conception giambagli2021 ), despite the modest number of trained parameters. To exemplify this, in the bottom panel of Fig. 1 we plot the relative ratio of the number of tuned parameters (Spectral) vs. conventional one) against $N_{2}$ (blue triangles) : the reduction in the number of parameters as follows the modified spectral method is staggering. Working with the other employed dataset, respectively F-MNIST and CIFAR-10, yields analogous conclusions (see annexed Supplementary Information).

One further improvement can be achieved by replacing $\phi$^(k) with its equivalent singular value decomposition (SVD), a factorization that generalizes the eigendecomposition to rectangular (in this framework, $N_{k+1}\times N_{k}$) matrices (see Gabri__2019 for an application to neural networks). In formulae, this amounts to postulate $\phi$^(k) $={\mathbf{U}}_{k}\mathbf{\Sigma}_{k}{\mathbf{V}}_{k}^{T}$ where ${\mathbf{V}}_{k}$ and ${\mathbf{U}}_{k}$ are, respectively, $N_{k}\times N_{k}$ and $N_{k+1}\times N_{k+1}$ real orthogonal matrices. On the contrary, $\mathbf{\Sigma}_{k}$ is a $N_{k+1}\times N_{k}$ rectangular diagonal matrix, with non-negative real numbers on the diagonal. The diagonal entries of $\mathbf{\Sigma}_{k}$ are the singular values of $\phi$_k. The symbol $\left(\cdot\right)^{T}$, stands for the transpose operation. The learning scheme can be hence reformulated as follows. For each $k$, generate two orthogonal random matrices ${\mathbf{U}}_{k}$ and ${\mathbf{V}}_{k}$. These latter are not updated during the successive stages of the learning process. At variance, the $M_{k+1}=\min(N_{k},N_{k+1})$ non trivial elements of $\mathbf{\Sigma}_{k}$ take active part to the optimisation process. For each $k$, $M_{k+1}+N_{k}+N_{k+1}$ parameters can be thus modulated to optimize the information transfer, from layer $k$ to layer $k+1$. Stated differently, $M_{k+1}$ free parameters adds up to the $N_{k}+N_{k+1}$ eigenvalues that get modulated under the original spectral approach. One can hence count on a larger set of parameters as compared to that made available via the spectral method, restricted to operate with the eigenvalues. Nonetheless, the total number of parameters scales still with the linear size $N$ of the deep neural network, and not quadratically, as for a standard training carried out in direct space. This addition (referred to as the S-SVD scheme) yields an increase of the recorded classification score, as compared to the setting where the Spectral method is solely employed, which is however not sufficient to fill the gap with conventional schemes (see Fig. 1). Similar scenarios are found for F-MNIST and CIFAR-10 (see Supplementary Information), with varying degree of improvement, which reflects the specificity of the considered dataset.

A decisive leap forward is however accomplished by employing a QR factorization of matrix $\phi$^(k). For $N_{k+1}>N_{k}$, this corresponds to writing the $N_{k+1}\times N_{k}$ matrix $\phi$_k as the product of an orthogonal $N_{k+1}\times N_{k}$ matrix ${\mathbf{Q}}_{k}$ and an upper triangular $N_{k}\times N_{k}$ matrix ${\mathbf{R}}_{k}$. Conversely, when $N_{k+1}<N_{k}$, we factorize $\phi$${}_{k}^{T}$, in such a way that the square matrix ${\mathbf{R}}_{k}$ has linear dimension $N_{k+1}$. In both cases, matrix ${\mathbf{Q}}_{k}$ is randomly generated and stays frozen during gradient descent optimisation. The $M_{k+1}(M_{k+1}+1)/2$ entries of the $M_{k+1}\times M_{k+1}$ matrix ${\mathbf{R}}_{k}$ can be adjusted so as to improve the classification ability of the trained network (this strategy of training, integrated to the Spectral method, is termed S-QR ). Results are depicted in Fig. 1 with (red) diamonds. The achieved performance is practically equivalent to that obtained with a conventional approach to learning. Also in this case $\rho<1$, the gain in parameter reduction being noticeable when $N_{1}$ is substantially different (smaller or larger) than $N_{2}$, for the case at hand. Interestingly enough, for a chief improvement of the performance, over the SVD reference case, it is sufficient to train a portion of the off diagonal elements of ${\mathbf{R}}$. In the Supplementary Information, we report the recorded accuracy against $p$, the probability to train the entries that populate the non null triangular part of ${\mathbf{R}}_{k}$. The value of the accuracy attained with conventional strategies to the training is indeed approached, already at values of $p$ which are significantly different from unit.

The quest for a limited subset of key parameters which define the target of a global approach to the training is also important for its indirect implications, besides the obvious reduction in terms of algorithmic complexity. As a key application to exemplify this point, we shall consider the problem of performing the classification tasks considered above, by training a neural network with a prescribed degree of imposed sparsity. This can be achieved by applying a non linear filter on each individual weight $w_{ij}$. The non linear mask is devised so as to return zero (no link present) when $|w_{ij}|<C$. Here, $C$ is an adaptive cut-off which can be freely adjusted to allow for the trained network to match the requested amount of sparsity. This latter is measured by a scalar quantity, spanning the interval $[0,1]$: when the degree of sparsity is set to zero, the network is dense. At the opposite limit, when the sparsity equals one, the nodes of the network are uncoupled and the information cannot be transported across layers. Working with the usual approach to the training, which seeks to modulate individual weights in direct space, one has to face an obvious problem. When the weight of a given link is turned into zero, then it gets excluded by the subsequent stages of the optimisation process. Consequently, a weight that has been silenced cannot regain an active role in the classification handling. This is not the case when operating under the spectral approach to learning, also when complemented by the supplemental features tested above. The target of the optimisation, the spectral attributes of the transfer operators, are not biased by any filtering masks: as a consequence, acting on them, one can rescue from oblivion weights that are deemed useless at a given iteration (and, as such, silenced), but which might prove of help, at later stages of the training. In Fig. 2, the effect of the imposed sparsity on the classification accuracy is represented for conventional vs. S-QR method. The latter is definitely more performant in terms of displayed accuracy, when the degree of sparsity gets more pronounced. The drop in accuracy as exhibited by the sparse network trained with the S-QR modality is clearly less pronounced, than that reported for an equivalent network optimised in direct space. Deviations between the two proposed methodologies become indeed appreciable in the very sparse limit, i.e. when the residual active links are too few for a proper functioning of the direct scheme. In fact, edges which could prove central to the classification, but that are set silent at the beginning, cannot come back to active. At variance, the method anchored to reciprocal space can identify an optimal pool of links (still constraint to the total allowed for) reversing to the active state, those that were initially set to null. Interestingly, it can be shown that a few hubs emerge in the intermediate layer, which collect and process the information delivered from the input stack.

Taken altogether, it should be unequivocally concluded that a large body of free parameters, usually trained in machine learning applications, is de facto unessential. The spectral learning scheme, supplemented with a QR training of the non trivial portion of eigenvectors’ matrix, enabled us to identify a limited subset of key parameters which prove central to the learning procedure, and reflect back with a global impact on the computed weights in direct space. This observation could materialise in a drastic simplification of current machine learning technologies, a challenge at reach via algorithmic optimisation carried out in dual space. Quite remarkably, working in reciprocal space yields trained networks with better classification scores, when operating at a given degree of imposed sparsity. This finding suggests that shifting the training to the spectral domain might prove beneficial for a wide gallery of deep neural networks applications.

*