Memory Faults in Activation-sparse Quantized Deep Neural Networks: Analysis and Mitigation using Sharpness-aware Training

Activation sparsity refers to the prevalence of a large number of zero-valued DNN activations. A common activation function in DNNs is the rectified linear unit (ReLU), which outputs a zero for every negative input, leading to a high activation sparsity. This motivates the design of sparsity-aware DNN hardware, that can skip the computations involving zero-valued activations, leading to a reduction in the inference latency [10]. Unlike weight sparsity, activation sparsity is dynamic in nature, which means that the number and location of the zero values vary from input to input. When utilized properly, the inherently large activation sparsity can be exploited by sparsity-aware DNN accelerators for sizeable latency improvements over standard accelerators [1] [10].

Consequently, algorithmic techniques to enhance the activation sparsity have gained interest in recent times. For example, the work in [5] adds the $L_{1}$ norm of the activations ($||x_{l,n}||_{1}$) to the original loss function to incentivize sparsity:

Here, $L_{reg}(x,w)$ and $L_{0}(x,w)$ are the new and original loss functions, respectively, $L$ is the number of layers in the DNN, $N$ is the batch size and $\alpha_{l}$ is the regularization constant per layer. By penalizing the dense activations using $L1$ regularization, the work in [5] exhibits up to 60% increase in sparsity with negligible loss in inference accuracy for image classification. The work in [4] uses a Hoyer sparsity metric based regularizer and a variant of ReLU to boost the activation sparsity of DNNs. It should be noted that these techniques are not equivalent to ”pruning” the activations [3]. While pruning permanently sets the parameters to zero for all inputs, the activation sparsification techniques do not remove any of the activations. Rather, they ensure that a low percentage of activations are non-zero for all inputs on an average.

In this paper, we use the $L1$ regularization based approach to train activation-sparse (AS) QDNNs. We will refer to the QDNNs trained without this regularization as standard QDNNs and the ones trained with it as AS QDNNs.

Aggressive scaling and exploration of new memory technologies have heightened the importance of studying memory faults. Faults corrupt a percentage of the stored weights, leading to erroneous computations. Two common memory fault models are bit-flip faults and stuck-at (SA) faults. Stuck-at one (SA1) and stuck-at zero (SA0) faults fix the value of a bitcell unalterably to ’1’ and ’0’, respectively. Conversely, bit-flip faults invert the value in the bitcell (from ’0’ to ’1’ or vice versa). The SA fault model is generally used to describe permanent defects such as fabrication defects, whereas the bit-flip fault model is used to describe transient phenomena such as half-select read disturbance and alpha particle strikes.

Understanding the impact of memory faults on DNN accuracy has attracted attention in recent times. Various works have analyzed the fault tolerance of both floating-point DNNs and QDNNs [11]. Some works have also studied the accuracy of pruned networks in the presence of faults [7], circuit non-idealities and variations [6], showing a larger vulnerability of pruned DNNs to faults and other non-idealities than their unpruned counterparts. However, to the best of our knowledge, no work has analyzed the impact of faults on AS QDNNs.

Multiple hardware and algorithmic solutions have been analyzed for mitigating the impact of faults. From the hardware perspective, the work in [12] proposes a selective duplication strategy to replicate the vulnerable portions of the DNNs to enhance fault tolerance. The work in [13] leverages the natural redundancy present in compute-in-memory based ternary bitcells to reduce fault impact. From the algorithmic perspective, the work in [14] uses error-correcting output codes to reduce DNN sensitivity to variations and SA faults. Works like [15] and [16] use weight transformations to reduce the amount of unmasked faults, leading to increased fault tolerance. Fault-aware training/finetuning is also a promising approach but is difficult to implement when the fault data is unavailable [17].

In this work, we extensively investigate the impact of SA and bit-flip faults on the performance of standard and AS QDNNs. Additionally, we introduce SAQ-based training as a strategy for fault tolerance. SAQ can be used alongside hardware techniques and offers protection against multiple types of faults, as discussed subsequently.

The sharpness of the minima of the loss landscape that a DNN converges to during training has a key impact on its generalization capability [18]. Both theoretically and empirically, it has been shown that convergence to a flat minima improves generalization. Hence, training schemes such as sharpness-aware minimization (SAM) are gaining attention [19]. SAM simultaneously minimizes the loss value and the sharpness of the weight loss landscape to learn the optima which have uniformly low loss values in their neighbourhood. DNNs trained with SAM have displayed state-of-the-art accuracies for several datasets and models. However, SAM is not as effective in QDNNs as it does not account for weight quantization. Sharpness-aware quantization (SAQ) is a variant of SAM designed for QDNNs [8]. SAQ simultaneously minimizes the loss value and flattens the loss curvature near the minima by minimizing the loss function with adversarially perturbed quantized weights. This leads to better generalization and improved accuracy of QDNNs. In this work, we advance the application of SAQ to train fault-tolerant QDNNs, based on the intuition that QDNNs converged at flatter minima will be less sensitive to the weight perturbations due to faults.

To explore the latency benefits and the impact of faults on activation-sparse (AS) QDNNs, we use the following methodology. First, two 4-bit standard QDNNs (baselines), viz. LeNet-5 with the FashionMNIST (FMNIST) dataset and ResNet-18 with the CIFAR-10 dataset are trained using the quantization framework in [2]. Then, we train the AS QDNNs by combining the $L1$ activation regularization [5] with the quantization framework in [2]. We focus on these model architectures because of their compact size, making them suitable for resource-constrained edge accelerators. The standard and AS QDNNs are trained to have nearly equal inference accuracies (with $\leq 0.5\%$ difference) for a fair comparison.

After training, we evaluate their latencies on sparsity-aware hardware based on the work in [10], to quantify the benefits of enhanced sparsity. Then, to study their fault tolerance, bit-flip faults and stuck-at faults (SA0 and SA1 individually) are randomly and uniformly injected into each model. Monte Carlo based fault injection experiments are performed, and the mean value of the inference accuracy is reported for different fault rates and types of faults (see Fig. 5). We perform experiments for fault rates 0% - 5% and 0% - 3% for the LeNet-5 and ResNet-18 QDNNs, respectively. ResNet-18 QDNNs were analyzed up to a fault rate of 3%, as higher rates caused severe accuracy drops, rendering the QDNNs unusable.

To better understand the differences in fault tolerance, we visualize the weight loss landscape for standard and AS QDNNs. Loss landscape visualization has been used in various works to enhance understanding of how well a model generalizes [18]. To our knowledge, this is the first work to examine the shape of the minima in relation to the QDNN’s fault tolerance. We use the normalization-based visualization technique described in [9] to generate the weight loss landscapes.

SAQ flattens the weight loss landscape of the QDNNs, which motivates us to employ it for making the QDNN less sensitive to faults. To evaluate this approach, we utilize SAQ to train both standard and AS QDNNs.

SAQ concurrently minimizes the loss value and loss sharpness by solving the following min-max optimization problem:

The first term in the equation defines the sharpness metric, which is the maximum change in the loss value for some weight perturbation $\epsilon$. The second term is the loss function itself and the third term is the standard $L2$ regularization term. Also, the perturbation $\epsilon$ is an adversarial one and is chosen such that the maximum sharpness is minimized. $\epsilon$ is given by the following equation and can have a maximum value of $\rho$.

Here, $\nabla_{Q_{w}(w,b)}L_{S}(Q_{w}(w,b))$ is the gradient of the loss function with respect to the quantized weights. $\epsilon$ is estimated using a forward and backward pass through the QDNN. Thus, compared to conventional training, SAQ training requires double the number of forward/backward passes per epoch. For a fair comparison of SAQ-based QDNNs with conventionally trained ones, we train SAQ models for half as many epochs. We select the $\rho$ hyperparameter to maximize the fault-free accuracy (fault rate of 0%) for the QDNNs. Using this framework, we analyse the effectiveness of SAQ in increasing the fault tolerance for both standard QDNNs and AS QDNNs.

First, we compare standard QDNNs trained conventionally and with SAQ. Figure 5 shows that SAQ-trained standard QDNNs exhibit larger fault tolerance than their conventionally trained counterparts. For example, for a bit-flip fault rate of 3%, SAQ-training increases the inference accuracy of LeNet-5 and ResNet-18 by 6.70% and 12.49%, respectively.

Next, we compare the accuracies of SAQ-trained and conventionally trained AS QDNNs. The $L1$ regularization constant is kept the same for both SAQ-trained and conventionally trained AS QDNNs, ensuring similar activation sparsity. The trends observed for AS QDNNs are similar to those for standard QDNNs. For a bit-flip fault rate of 3%, SAQ training enhances the inference accuracy of LeNet-5 and ResNet-18 AS QDNNs by 9.34% and 13.96%.

An interesting observation here is that SAQ-trained AS QDNNs exhibit a higher accuracy in the presence of faults than standard QDNNs without SAQ. For example, our experiments show that the SAQ-trained LeNet 5 AS QDNN has a 8.37% higher accuracy than the conventionally trained LeNet 5 standard QDNN at fault rate = 5%. Thus, SAQ-trained AS QDNNs have both the benefits of superior fault tolerance and lower latency (compared to conventionally trained standard QDNNs), making them highly suitable for edge applications. Thus, training with SAQ mitigates the adverse impact that activation sparsity augmentation has on the fault tolerance.