On the Robustness and Anomaly Detection of Sparse Neural Networks

With the need to deploy Machine Learning (ML) models in real-life systems such as autonomous vehicles, medical diagnosis, online fraud detection etc., building safe and reliable ML pipelines has become more critical than ever. However, there exist three well-known cases of changes to the in-distribution data – or anomalies – to which models are known to be brittle: adversarial attacks (AA), which are intentionally crafted but imperceptible perturbations that can fool a classifier Goodfellow et al. (2014); distribution shifts (DS), which are naturally occurring perturbations that can also change the model’s output Hendrycks & Dietterich (2019); and out-of-distribution (OOD) samples which come from an entirely different distribution, and that can surprisingly lead to highly confident but wrong predictions Hendrycks & Gimpel (2016). This latter case is notoriously known for generative models, which tend to assign higher likelihoods to samples coming from a distribution that they have not been trained on Nalisnick et al. (2018).

In an orthogonal line of work, Neural Networks (NN) are also known to be highly over-parameterized, which is made evident by the existence of sparse subnetworks that achieve the same accuracy performance as their dense counter-part Han et al. (2015). To find these subnetworks, many works have derived pruning criteria that assign scores to weights, and prune those with the lowest values Lee et al. (2018); Wang et al. (2020). However, the algorithms that attempt to find these subnetworks rarely evaluate them on other challenging tasks such as robustness and anomaly detection.

In this paper, we show that pruning can benefit robustness and anomaly detection by tackling the following four questions: 1) How does sparsity affect various properties of networks, beyond test accuracy? (Sec. 3), 2) Can we find sparse subnetworks within pre-trained dense networks that achieve better robustness and detection? (Sec. 4), 3) Can sparsity benefit tasks beyond deterministic single-network classification such as ensembles, Bayesian NN and generative models? (Sec. 5), and 4) Are pruning criteria useful for the detection of anomalies in the input? (Sec. 6) ¹¹1Project page including code at https://www.cs.cit.tum.de/daml/uncertainty-snn/

Sparse Networks. Pruning can be performed either before training a model Lee et al. (2018); Wang et al. (2020), early in training Rachwan et al. (2022); You et al. (2020), or after the model has been fully trained Frankle et al. (2020a, b); LeCun et al. (1990); Morcos et al. (2019). It can be done either in an unstructured Nalisnick et al. (2018); Wang et al. (2020); Frankle et al. (2020a) or a structured manner Rachwan et al. (2022); Li et al. (2016); You et al. (2019), where unstructured pruning removes weights by setting them to 0 whereas structured pruning removes entire neurons or filters by reducing the size of the weight matrices involved. Additionally, pruning can be done globally Rachwan et al. (2022); Lee et al. (2018); Wang et al. (2020); Frankle et al. (2020a) or locally Ramanujan et al. (2020), where global pruning automatically decides how much to prune from each layer when given a desired total sparsity, whereas local pruning prunes the same percentage of weights at every layer. Unless otherwise specified, we use global pruning throughout the paper. We briefly introduce the 8 pruning algorithms we use in this paper:

Pruning Before Training. SNIP Lee et al. (2018): A sensitivity-based pruning method (i.e. it scores weights based on their sensitivity to a certain metric) that aims at preserving the loss function. The score used is: $s_{i}=|w_{i}\frac{\partial L}{\partial w_{i}}|$, where $L$ is the loss and $w_{i}$ the weight.
GRASP Wang et al. (2020): A sensitivity-based pruning method that aims at increasing the gradient flow. The final score is: $s_{i}=-|w_{i}(\textbf{H}\frac{\partial L}{\partial\textbf{w}})_{i}|$, where H denotes the Hessian.
CroPit Rachwan et al. (2022): A sensitivity-based pruning method that aims at preserving the gradient flow. The final score is: $s_{i}=|w_{i}(\textbf{H}\frac{\partial L}{\partial\textbf{w}})_{i}|$, where H denotes the Hessian. We also use its structured version CroPit-S.
Edge-popup Ramanujan et al. (2020): An optimization-based method that aims at finding sparse network by optimizing the mask directly instead of the weights. The sparse model is not trained further.

Pruning During Training. IMP Frankle et al. (2020a): An iterative-based pruning method that prunes the smallest weights first. It goes through multiple train-prune-rewind cycles until it reaches its desired sparsity, where "rewind" corresponds to resetting the values of the remaining weights after each pruning phase to their value at an early point in training.
EarlyCroP: CroPit performed early in training instead of before training.

Pruning After Training. SNIP-After: SNIP applied on a fully trained network.
Edge-popup-After: Edge-popup applied on a fully trained network instead of randomly initialized one.

Robustness and Anomaly Detection of Networks. While NN are constantly improving in terms of accuracy or generative capability, their weak robustness and anomaly detection pose challenges to their safe deployment in the real world Shafaei et al. (2018). Classifiers have shown to be un-calibrated, i.e. assign high confidence to wrong predictions, and works have tried to improve existing models Hendrycks et al. (2018); Liu et al. (2020); Huang et al. (2021) or come up with new methods that focus solely on the detection of anomalous inputs Tack et al. (2020). Generative models that can compute the log-likelihood of inputs have also been shown to assign higher likelihood to anomalous samples Nalisnick et al. (2018), leading to another body of works that attempts to understand and solve this problem Schirrmeister et al. (2020). Note that, unless otherwise specified, we use the common baseline Maximum-Softmax-Probability (MSP) Hendrycks & Gimpel (2016) to compute the AUC-ROC scores throughout the paper, which uses the predicted probability of an input as the confidence.

Robustness and Anomaly Detection of Sparse Networks. With the growing interest in sparsity of deep NN, the literature started addressing the question of whether sparse models would have different properties than their dense counterparts, beyond their test accuracy. Some works argue that an appropriate level of sparsity can improve robustness of models either to adversarial attacks Guo et al. (2018) or to distribution shifts Diffenderfer et al. (2021), while others argue the contrary Verdenius et al. (2020); Liebenwein et al. (2021). Another body of works focuses on simultaneously optimizing for high accuracy, robustness and sparsity by incorporating objectives that promote the latter two during training Madaan et al. (2020); Zhang et al. (2021); Sehwag et al. (2020). In contrast, we do not directly optimize weights for the tasks of robustness and detection, but evaluate the effect of existing sparsity methods on these tasks and propose new ways to take advantage of them.

We start by addressing the question: How does sparsity affect various properties of networks, beyond test accuracy? Our goal is to give a comprehensive view on the effect of pruning on robustness to and detection of anomalies. We therefore evaluate models pruned with 8 different pruning algorithms, and make the distinction between pruning before, during or after training. Indeed, we expect the pruning time to affect the set of weights being preserved, and therefore their final performance on new tasks.

Setup. We train ResNet18 models on the CIFAR10 dataset. We use L-$\infty$ and L-2 FGSM attacks with $\epsilon=8$, CIFAR10C as the DS dataset, and SVHN and CIFAR100 as the OOD datasets. We report relative accuracy (i.e. normalized with respect to the dense model) and relative AUC-ROC on attacks and shifts, and relative Brier score and relative AUC-ROC on OOD datasets. All models are trained with 3 different seeds and their average performance is reported.

Discussion. Fig. 1 clearly shows that almost all pruning methods improve the robustness to L-$\infty$ FGSM attacks (a), but not to L-2 FGSM attacks (b). Indeed, the relation between L-$\infty$ and sparsity has already been discussed in Guo et al. (2018), and works such as Verdenius et al. (2020) have reported a decrease of robustness on L-2 based attacks such as Carlini-Wagner Carlini & Wagner (2017). We also observe that most methods maintain the robustness to DS (c), with During-Training methods performing slightly better than Before-Training methods at high sparsity.

In terms of detection, Fig. 2 shows that generally all methods except Edge-popup can maintain the AUC-ROC on DS. SNIP-After also appears to provide better improvements at an appropriate sparsity than the remaining methods.

Finally, from Fig. 3 we can see that During or After-Training pruning methods perform better than the dense model in terms of Brier score (a) up until 90% sparsity, but only IMP does so at even higher sparsity. Generally all methods, except Edge-popup, can also maintain the AUC-ROC on the OOD datasets SVHN (b) and CIFAR100 (c). We report additional metrics and results on LeNet5 in App. C.

Next, we address the question: Can we find (sparse) sub-networks within dense pre-trained networks that achieve better robustness and detection? This question has been briefly tackled by Sehwag et al. (2020), but they only evaluated the robustness of the sparse models to adversarial attacks, while we aim to evaluate their robustness to both adversarial attacks and distribution shifts, as well as their OOD detection capability.

Setup. We train ResNet18 models on CIFAR10, and report the accuracy and AUC-ROC on FGSM attacks with $\epsilon=8$ and on CIFAR10C, as well as the AUC-ROC on SVHN. Similarly to Sehwag et al. (2020), we use the Edge-popup algorithm, which trains a mask over the weights instead of the weights themselves (App. B). However instead of using the standard cross-entropy objective $\mathcal{H}$, we augment the objective depending on the task we are trying to solve. In the following, $X$ refers to a clean batch, $Y$ to its ground-truth labels, and $f$ to the model:
Adversarial Attacks: $\mathcal{L}(X)=\mathcal{H}(f(X),Y)+\lambda*\mathcal{KL}(f(X),f(X^{\prime}))$, with $X$with $X^{\prime}$ the adversarial attack of $X$, and $\mathcal{KL}$ the KL divergence between the predictions on $X$ and $X^{\prime}$. The second term in $\mathcal{L}$ ensures that the predictions on $X$ and $X^{\prime}$ do not deviate too much. We use $\lambda=6$.
Distribution Shifts: $\mathcal{L}(X)=\mathcal{H}(f(X^{\prime}),Y)$, with $X^{\prime}$ being a clean batch perturbed with gaussian noise. Indeed, training a model on clean inputs perturbed with gaussian noise can lead to a model more robust to natural perturbations Diffenderfer et al. (2021).
OOD Detection: $\mathcal{L}(X)=\mathcal{H}(f(X),Y)+\mathcal{H}(f(X^{\prime}),U)$, with $X^{\prime}$ being an OOD batch and $U$ a matrix of uniform distributions. The second term in $\mathcal{L}$ ensures the predictions on OOD samples are as close as possible to $U$ (maximum uncertainty).

Discussion. In Tab. 1, we report the metrics of interest for the three objectives on Edge-popup and compare to the original dense model. Each objective improves its corresponding metric the most, but sometimes also benefits others. For example, training on the OOD objective significantly improves the accuracy on FGSM attacks, a correlation which has already been noted in the literature Lee et al. (2020). This experiment shows that subnetworks that achieve much better performance on a certain objective do exist wihtin a pre-trained dense network. This suggests that dense networks focus on non-robust features as was shown in Ilyas et al. (2019), possibly due to their over-parameterization, and pruning eliminates some weights that focus on these features.

In this section, we investigate the effect of sparsity on existing methods that improve the robustness or detection capability of networks, and try to answer the question: Can sparsity benefit tasks beyond deterministic single-network classification? We examine ensembles, Bayesian neural networks and generative models.

Ensembles. One popular method to improve robustness and calibration of classifiers is ensembles. It consists of training multiple networks independently, and then averaging their outputs to obtain more calibrated predictions Lakshminarayanan et al. (2016). Ensembles have been shown to be among the most calibrated methods against distribution shifts Ovadia et al. (2019). However, one major drawback is the expensive nature of this method, both in terms of memory and time. Structured pruning helps in overcoming both of these drawbacks, since it removes entire neurons and channels from the architecture, thereby reducing the training and inference time, but also the space complexity. For this experiment, we use the large model AlexNet trained on CIFAR10, and compare a single dense model and an ensemble of 5 dense models to an ensemble of 5 models pruned to 80% node sparsity each. We can see from Tab. 2 that the sparse ensemble reaches almost the same test accuracy as the dense ensemble, matches it in terms of AUC-ROC on SVHN, and is much more efficient than the dense ensemble in terms of training and inference time, and GPU RAM usage.

Bayesian Neural Networks. Bayesian NN also fall into the category of expensive uncertainty methods. They are both slower to train and infer on, and more memory intensive since they often require to learn more parameters than a dense model. To illustrate the potential of sparsity in such tasks, we turn our attention to the popular method presented in Blundell et al. (2015). It consists of learning both a mean $w_{i}$ and a standard deviation $t_{i}$ for each weight $i$ in the network. At test time, we sample a value for each weight when doing an inference, and repeat the process multiple times to obtain multiple predictions, similar to ensembles. This work suggests to remove weights with a low Signal-to-Noise Ratio (SNR) $s_{i}=|\frac{w_{i}}{t_{i}}|$, but only evaluates the pruned model on the final accuracy. SNR is an intuitive pruning criterion, since it encourages the removal of weights with small means (similar to IMP which removes weights close to 0), and high standard deviation which indicates the uncertainty of the model and therefore the vulnerability of this weight to changes in the input. We extend this criterion to a structured one by simply summing the scores at each activation, which we call SNR-S, to take advantage of the removal of structures to achieve faster training and inference time. We train a dense Conv6 model on CIFAR10, a deterministic one, and two SNR-S models where pruning is done either during or after training. From Tab. 3, we observe that both versions of Bayesian SNR-S achieve better accuracy than the dense Bayesian, and SNR-S After achieves even better AUC-ROC than both dense models, while being more efficient than the dense Bayesian model.

Generative Models. Sparsity in generative models has recently started to gain attention in the literature Kalibhat et al. (2020). However, to the best of our knowledge, its effect on OOD detection has not been investigated yet. Generative models, some of which are capable of computing the log-likelihood of input samples, are known for their counter-intuitive behavior on OOD samples: they assign higher likelihood to OOD samples than to their own training/testing in-distribution samples. In this experiment, starting with the pre-trained generative model Glow Kingma & Dhariwal (2018), we prune it using EarlyCroP locally. We observe that, in the FashionMNIST-MNIST combination, pruning greatly helps in better distinguishing the two types of datasets and correctly assign lower likelihoods to OOD samples. However, when moving to more complex cases (CIFAR10-SVHN or CELEBA-SVHN), we do not observe the same behavior. Nevertheless, the success on the FashionMNIST-MNIST combination encourages further exploration of the benefits of sparsity in generative models. Note that local pruning plays a crucial role here, as global pruning did not result in the same behavior on FashionMNIST. One possible explanation for this is that local pruning creates more bottlenecks in the network, which has been shown to help in OOD detection Kirichenko et al. (2020).

We finally address the question: Are pruning criteria useful for the detection of anomalies in the inputs? Some pruning methods that operate before training have been shown to be independent of the batch being used to score the weights Su et al. (2020). This means that the weights being pruned and the final structure of the network do not depend much on the current task/dataset of interest. However, methods operating during training are believed to use information from the batch to keep the most important weights for the given task Su et al. (2020). Note that this applies to methods that use a sensitivity criterion, i.e. that assign a score to each weight (unlike IMP). Since pruning based on one or more batches will preserve the weights that are relevant for these batches, pruning based on batches that contain anomalies will attempt to preserve a different set of weights. Based on these observations, our key idea is to detect anomalies by measuring the distance between the sensitivity of weights on in-distribution (training) data, and that of each incoming test batch. Intuitively, the larger the distance the more likely the input batch contains anomalous samples.

Method. More formally, we are given a pre-trained model with weights $\mathbf{w}$, a train dataset $D_{train}=\{d_{i}\}_{i=0}^{m}$, a test dataset $D_{test}=\{d^{\prime}_{j}\}_{j=0}^{n}$, and a sensitivity scoring function $f_{B}$ where $B$ indicates one or more batches used to get the weights’ sensitivity. We first compute the sensitivity of weights on the training data $f_{D_{train}}(\mathbf{w})$ by going through all training batches and accumulating the gradients. We also save the element-wise standard deviation over all batches $std(f_{D_{train}}(\mathbf{w}_{i}))$. At test time, for every incoming batch $d^{\prime}_{j}$, we compute $f_{d^{\prime}_{j}}(\mathbf{w})$. Finally, we compute the distance $S_{d^{\prime}_{j}}=\|f_{D_{train}}(\mathbf{w})-f_{d^{\prime}_{j}}(\mathbf{w})\|_{p}$, where $\|.\|_{p}$ is the L$p$-norm (in practice we use $p=5$). If $S_{d^{\prime}_{j}}\geq\tau$, where $\tau$ is some pre-defined threshold, then $d^{\prime}_{j}$ is an anomalous batch; otherwise, it is an in-distribution batch. We refer to our method as SensNorm.

We rely on the SNIP score as our sensitivity score, since its purpose is to preserve the loss of the task at hand. However, we find that dropping the absolute value helps in achieving better performance. Indeed, the sign of a score in SNIP indicates whether a certain weight was increasing or decreasing the loss, which can help distinguish in-distribution from anomalous batches. We also standardize the test batch scores by their mean and standard deviation computed on the training data, so as to reduce the contribution of the weights that are highly variable on the in-distribution data itself. Our weight sensitivity function is therefore:

Discussion. Our method naturally lends itself to batch detection, but can also be applied for single-input prediction. Tab. 4 shows that, with 100 samples, our method can detect all types of inputs easily, and their detection remains high until a batch of size 5. The challenge arises for batches of size 2 and 1, since gradients will be very noisy for these cases. We therefore rely on augmentations to gain more information from the input sample (App. A). Note that, while originally conceived for detecting batches, SensNorm still outperforms MSP on batch size 1. We also compare our results to GradNorm, which uses gradient norms to detect anomalies, and find that SensNorm matches it or even outperforms it on small batch sizes.

In conclusion, we show that the topics of pruning, robustness and detection are closely intertwined. First, most pruning algorithms can maintain the original uncertainty of the dense model on AA, DS and OOD. All pruning methods also greatly improve the accuracy to L-$\infty$ FGSM attacks, but not to the L-$2$ based attacks. Pruning methods that operate during training (i.e. EarlyCroP and IMP) often offer the best trade-off between good test performance and maintaining or even improving robustness and detection metrics. Second, we show that it is possible to find subnetworks within pre-trained dense networks that perform better than the original models in terms of robustness and detection. Third, we find that pruning can be beneficial in tasks such as ensembles, Bayesian NN and generative models, especially in terms of efficiency for the expensive uncertainty estimation methods. Finally, we introduce SensNorm, a new method for detecting batches of anomalous samples. SensNorm shows that weight sensitivity is a good indicator of the presence of anomalies in the input.