Laplacian Networks: Bounding Indicator Function Smoothness for Neural Networks Robustness

Consider a deep neural network architecture. Such a network is obtained by assembling layers of various types. Of particular interest are layers of the form $\mathbf{x}^{\ell}\mapsto\mathbf{x}^{\ell+1}=h^{\ell}(\mathbf{W}^{\ell}\mathbf{x}^{\ell}+\mathbf{b}^{\ell})$, where $h^{\ell}$ is a nonlinear function, typically a ReLU, $\mathbf{W}^{\ell}$ is the weight tensor at layer $\ell$, $\mathbf{x}^{\ell}$ is the intermediate representation of the input at layer $\ell$ and $\mathbf{b}^{\ell}$ is the corresponding bias tensor. Note that strides or pooling may be used. Assembling can be achieved in various ways: composition, concatenation, sums…so that we obtain a global function $f$ that associates an input tensor $\mathbf{x}^{0}$ to an output tensor $\mathbf{y}=f(\mathbf{x}^{0})$.

When computing the output $\mathbf{y}$ associated with the input $\mathbf{x}^{0}$, each layer $\ell$ of the architecture processes some input $\mathbf{x}^{\ell}$ and computes the corresponding output $\mathbf{y}^{\ell}=h^{\ell}(\mathbf{W}^{\ell}\mathbf{x}^{\ell}+\mathbf{b}^{\ell})$. For a given layer $\ell$ and a batch of $b$ inputs $\mathcal{X}=\{\mathbf{x}_{1},\dots,\mathbf{x}_{b}\}$, we can obtain two sets $\mathcal{X}^{\ell}=\{\mathbf{x}^{\ell}_{1},\dots,\mathbf{x}^{\ell}_{b}\}$, called the preset, and $\mathcal{Y}^{\ell}=\{\mathbf{y}^{\ell}_{1},\dots,\mathbf{y}^{\ell}_{b}\}$, called the postset.

Given a similarity measure $s$ on tensors, from a preset we can build the similarity preset matrix: $\mathbf{M}^{\ell}_{pre}[i,j]=s(\mathbf{x}_{i}^{\ell},\mathbf{x}_{j}^{\ell}),\forall 1\leq i,j\leq b$, where $\mathbf{M}[i,j]$ denotes the element at line $i$ and column $j$ in $\mathbf{M}$. The postset matrix is defined similarly.

Consider a similarity (either preset or postset) matrix $\mathbf{M}^{\ell}$. This matrix can be used to build a $k$-nearest neighbor similarity weighted graph $G^{\ell}=\langle V,\mathbf{A}^{\ell}\rangle$, where $V=\{1,\dots,b\}$ is the set of vertices and $\mathbf{A}^{\ell}$ is the weighted adjacency matrix defined as:

$$\displaystyle{\mathbf{A}^{\ell}[i,j]=\left\{\begin{array}[]{ll}\mathbf{M}^{\ell}[i,j]&$if $\mathbf{M}^{\ell}[i,j]\in\arg\max_{i^{\prime}\neq j}{(\mathbf{M}^{\ell}[i^{\prime},j],k)}\\ &\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\bigcup\arg\max_{j^{\prime}\neq i}{(\mathbf{M}^{\ell}[i,j^{\prime}],k)}\\ 0&$otherwise$\end{array}\right.},\forall i,j\in V,$$

(1)

where $\arg\max_{i}(a_{i},k)$ denotes the indices of the $k$ largest elements in $\{a_{1},\dots,a_{b}\}$. Note that by construction $\mathbf{A}^{\ell}$ is symmetric.

Given a weighted graph $G^{\ell}=\langle V,\mathbf{A}^{\ell}\rangle$, we call Laplacian of $G^{\ell}$ the matrix $\mathbf{L}^{\ell}=\mathbf{D}^{\ell}-\mathbf{A}^{\ell}$, where $\mathbf{D}^{\ell}$ is the diagonal matrix such that: $\mathbf{D}^{\ell}[i,i]=\sum_{j}{\mathbf{A}^{\ell}[i,j]},\forall i\in V$. Because $\mathbf{L}^{\ell}$ is symmetric and real-valued, it can be written:

$$\mathbf{L}^{\ell}=\mathbf{F}^{\ell}\mathbf{\Lambda}^{\ell}\mathbf{F}^{\ell\top},$$

(2)

where $\mathbf{F}$ is orthonormal and contains eigenvectors of $\mathbf{L}^{\ell}$ as columns, $\mathbf{F}^{\top}$ denotes the transpose of $\mathbf{F}$, and $\mathbf{\Lambda}$ is diagonal and contains eigenvalues of $\mathbf{L}^{\ell}$ is ascending order. Note that the constant vector $\mathbf{1}\in\mathbb{R}^{b}$ is an eigenvector of $\mathbf{L}^{\ell}$ corresponding to eigenvalue $0$. Moreover, all eigenvalues of $\mathbf{L}^{\ell}$ are nonnegative. Consequently, $\mathbf{1}/\sqrt{n}$ can be chosen as the first column in $\mathbf{F}$.

Consider a vector $\mathbf{s}\in\mathbb{R}^{b}$, we define $\hat{\mathbf{s}}$ the Graph Fourier Transform (GFT) of $\mathbf{s}$ on $G^{\ell}$ as (Shuman et al., 2013):

$$\hat{\mathbf{s}}=\mathbf{F}^{\top}\mathbf{s}.$$

(3)

Because the order of the eigenvectors is chosen so that the corresponding eigenvalues are in ascending order, if only the first few entries of $\hat{\mathbf{s}}$ are nonzero that indicates that $\mathbf{s}$ is low frequency (smooth). In the extreme case where only the first entry of $\hat{\mathbf{s}}$ is nonzero we have that $\mathbf{s}$ is constant (maximum smoothness). More generally, smoothness $\sigma^{\ell}(\mathbf{s})$ of a signal $\mathbf{s}$ can be measured using the quadratic form of the Laplacian:

$$\sigma^{\ell}(\mathbf{s})=\mathbf{s}^{\top}\mathbf{L}^{\ell}\mathbf{s}=\sum_{i,j=1}^{b}{\mathbf{A}^{\ell}[i,j](\mathbf{s}[i]-\mathbf{s}[j])^{2}}=\sum_{i=1}^{b}{\mathbf{\Lambda}^{\ell}[i,i]\hat{\mathbf{s}}[i]^{2}},$$

(4)

where we note that $\mathbf{s}$ is smoother when $\sigma^{\ell}(\mathbf{s})$ is smaller.

In this paper we are particularly interested in smoothness of the label signals. We call label signal $\mathbf{s}_{c}$ associated with class $c$ a binary ($\{0,1\}$) vector whose nonzero coordinates are the ones corresponding to input vectors of class $c$. In other words, $\mathbf{s}_{c}[i]=1\Leftrightarrow(\mathbf{x}_{i}$ is in class $c),\forall 1\leq i\leq b$.

Denote $u$ the last layer of the architecture: $\mathbf{y}^{u}_{i}=\mathbf{y}_{i},\forall i$. Note that in typical settings, where outputs of the networks are one-hot-bit encoded and no regularizer is used, at the end of the learning process it is expected that $\mathbf{y}_{i}^{\top}\mathbf{y}_{j}\approx 1$ if $i$ and $j$ belong to the same class, and $\mathbf{y}_{i}^{\top}\mathbf{y}_{j}\approx 0$ otherwise.

Thus, assuming that cosine similarity is used to build the graph, the last layer smoothness for all $c$ would be $\sigma^{u}_{post}(\mathbf{s}_{c})\approx 0$, since edge weights between nodes having different labels will be close to zero given Equation (4). More generally, smoothness of $\mathbf{s}_{c}$ at the preset or postset of a given layer measures the average similarity between examples in class $c$ and examples in other classes ($\sigma(\mathbf{s}_{c})$ decreases as the weights of edges connecting nodes in different classes decrease). Because the last layer can achieve $\sigma(\mathbf{s}_{c})\approx 0$, we expect the smoothness metric $\sigma$ at each layer to decrease as we go deeper in the network. Next we introduce a regularization strategy that limits how much $\sigma$ can decrease from one layer to the next and can even prevent the last layer from achieving $\sigma(\mathbf{s}_{c})=0$. This will be shown to improve generalization and robustness. The theoretical motivation for this choice is discussed in Section 3.4.

Recent work (Anis et al., 2017) develops an asymptotic analysis of the bandwidth of label signals, $BW(\mathbf{s})$, where bandwidth is defined as the highest non-zero graph frequency of $\mathbf{s}$, i.e., the nonzero entry of $\hat{\mathbf{s}}$ with the highest index. An estimate of the bandwidth can be obtained by computing:

$$BW_{m}(\mathbf{s})=\left(\frac{\mathbf{s}^{\top}\mathbf{L}^{m}\mathbf{s}}{\mathbf{s}^{\top}\mathbf{s}}\right)^{(1/m)}$$

(6)

for large $m$. This can be viewed as a generalization of the smoothness metric of (4). (Anis et al., 2017) shows that, as the number of labeled points $\mathbf{x}$ (assumed drawn from a distribution $p(\mathbf{x})$) grows asymptotically, the bandwidth of the label signal converges in probability to the supremum of $p(\mathbf{x})$ in the region of overlap between classes. This motivates our work in three ways.

First, it provides theoretical justification to use $\sigma^{\ell}(\mathbf{s})$ for regularization, since lower values of $\sigma^{\ell}(\mathbf{s})$ are indicative of better separation between classes. Second, the asymptotic analysis suggests that using higher powers of the Laplacian would lead to better regularization, since estimating bandwidth using $BW_{m}(\mathbf{s})$ becomes increasingly accurate as $m$ increases. Finally, this regularization can be seen to be protective against overfitting by preventing $\sigma^{\ell}(\mathbf{s})$ from decreasing “too fast”. For most problems of interest, given a sufficiently large amount of labeled data available, it would be reasonable to expect the bandwidth of $\mathbf{s}$ not to be arbitrarily small, because the classes cannot be exactly separated, and thus a network that reduces the bandwidth too much can result in overfitting.

In Figure 2 we depict the Laplacian and squared Laplacian of similarity graphs obtained at different layers in a trained vanilla architecture. On the deep layers, we can clearly see blocks corresponding to the classes, while the situation in the middle layer is not as clear. This figure illustrates how using the squared Laplacian helps modifying the distances to improve separation. Note that we normalize the squared Laplacian values by dividing them by the highest absolute value.

In Figure 3, we plot the average evolution of smoothness of label signals over 100 batches, as a function of layer depth in the architecture, and for different choices of the regularizer. In the left part, we look at smoothness measures using the Laplacian. In the right part, we use the squared Laplacian. We can clearly see the effectiveness of the regularizer in enforcing small variations of smoothness across the architecture. Note that for model regularized with $\mathbf{L}^{2}$, changes in smoothness measured by $\mathbf{L}$ are not easy to see. This seems to suggest that some of the gains achieved via $\mathbf{L}^{2}$ regularization come in making changes that would be “invisible” when looking at the layers from the perspective of $\mathbf{L}$ smoothness. The same normalization from Figure 2 is used for $\mathbf{L}^{2}$.

In this scenario we evaluate the robustness of the network function to small isotropic variations of the input. We generate 40 different deformations using random variables $\mathcal{N}(0,0.25)$ which are added to the test set inputs. Note that they are scaled so that $SNR\approx 15$ and $SNR\approx 20$. The middle and right-most plots from Figure 4 show that the proposed method increases the robustness of the network to isotropic deformations. Note that in both scenarios the best results are achieved by combining Parseval training and our proposed method (lower-most box on both figures).

We next evaluate robustness to adversarial inputs, which are specifically built to fool the network function. Such adversarial inputs can be generated and evaluated in multiple ways. Here we implement two approaches: first a mean case of adversarial noise, where the adversary can only use one forward and one backward pass to generate the deformations, and second a worst case scenario, where the adversary can use multiple forward and backward passes to try to find the smallest deformation that will fool the network.

For the first approach, we add the scaled gradient sign (FGSM attack) on the input (Kurakin et al., 2016), so that we obtain a target $SNR$. Results are depicted in the left and center plots of Figure 5. In the left plot the noise is added after normalizing the input whereas on the middle plot it is added before normalizing. As in the isotropic noise case, a combination of the Parseval method and our proposed approach achieves maximum robustness.

In regards to the second approach, where a worst case scenario is considered, we use the Foolbox toolbox (Rauber et al., 2017) implementation of DeepFool  (Moosavi Dezfooli et al., 2016). The conclusions are similar (right plot of Figure 5) to those obtained for the first adversarial attack approach.

Finally, in a third series of experiments we evaluate the robustness of the network functions to faulty implementations. As a result, approximate computations are made during the test phase that consist of random erasures of the memory (dropout) or quantization of the weights (Hubara et al., 2017).

In the dropout case, we compute the test set accuracy when the network has a probability of either $25\%$ or $40\%$ of dropping a neuron’s value after each block. We run each experiment $40$ times. The results are depicted in the left and center plots of Figure 6. It is interesting to note that the Parseval trained functions seem to drop in performance as soon as we reach $40\%$ probability of dropout, providing an average accuracy smaller than the vanilla networks. In contrast, the proposed method is the most robust to these perturbations.

For the quantization of the weights, we consider a scenario where the network size in memory has to be shrink 6 times. We therefore quantize the weights of the networks to 5 bits (instead of 32) and re-evaluate the test set accuracy. The right plot of Figure 6 shows that the proposed method is providing a better robustness to this kind of deformation than the tested counterparts.