A binary-activation, multi-level weight RNN and training algorithm for ADC-/DAC-free and noise-resilient processing-in-memory inference with eNVM

The NN model structure. RNNs are well-suited to this speech-recognition task. Figure 2a illustrates a 2-layer gated recurrent unit (GRU) (Cho et al., 2014) with BA and MLW. We first perform FFT (window=16ms, stride=8ms) on the raw audio signals and then extract 40 Mel-frequency cepstral coefficients (MFCC) per 8ms timestep. Each MFCC vector passes through an input FC layer that encodes it into a 128 dimensional binary vector as the input to the first layer of a 2-layer stacked GRU (both layers use 128 dimensional vectors). We use the following modified version of GRU equations (Ng et al., 2018):

where $t$ denotes the timestep, $l$ is the layer number, and the gate $G^{l}<t>$ ($l=1,2$), candidate $C^{l}<t>$ ($l=1,2$), hidden state $H^{l}<t>$ ($l=1,2$), and the input encoding $H^{0}<t>$ are all 128 dimensional activation vectors, trained using our NNA algorithm. The activation function $f$ refers to NCN (Equation 3) during training, and NBN (Equation 5) for evaluation, and simply uses the binary step function for PIM deployment (Figure 2a and 2b). Compared with the original GRU equations from (Cho et al., 2014), we remove the reset gate since it has minimal effect on the accuracy of this task, but greatly simplifes circuit design (Figure 2b). After the GRU processes inputs from all timesteps, the final timestep output of the top layer feeds into an output FC layer followed by a 12-way softmax that yields the classification.

PIM circuit design for the GRU. The BA-MLW GRU equations can be mapped into compact PIM circuits shown in Figure 2b. The MLC array implements MAC computations and the column sense-amps resolve the binarized $G^{l}<t>$ and $C^{l}<t>$. $G^{l}<t>$ serves as the multiplexer selection signal (since $1-G^{l}<t>=!G^{l}<t>$ in Equation 11 for binary signals) to choose either to keep the binary hidden state saved from the previous timestep or to update it with the current binary candidate state. Since all analog MAC signals are encapsulated inside the MLC array, all input/output interface signals are binary, and the GRU logic outside the array is totally digital, this PIM design does not require any ADCs or DACs.

Noise-induced loss penalty terms for GRU. For the GRU, we can derive the noise-induced regularization penalty term $P$ in Equation 6 by taking the derivatives of loss $L$ w.r.t. the pre-activations of candidates ($\widetilde{C}^{l}<t>$) and gates ($\widetilde{G}^{l}<t>$).

where a lowercase letter with subscript $i=0\sim 127$ denotes one element of the corresponding uppercase vector, and $Pg$ and $Pc$ regularize gates and candidates, respectively. The forms of Equations 13 and 14 have intuitive interpretations when minimizing them during training. To minimize $Pg$, one way is to reduce the derivative of the gate ($\mathrm{sigmoid}^{\prime}(\widetilde{g}_{i}^{l}<t>)$) by pushing $\widetilde{g}_{i}^{l}<t>$ away from zero (the steep slope region of sigmoid), so that the gate is firmly ON or firmly OFF; alternatively, it can try to make the candidate of the current timestep $c_{i}^{l}<t>$ equal to the hidden state of the previous timestep $h_{i}^{l}<t-1>$, such that the new hidden state $h_{i}^{l}<t>$ would be the same regardless of $\widetilde{g}_{i}^{l}<t>$. Either way, minimizing $Pg$ makes $h_{i}^{l}<t>$ immune to noise injected into $\widetilde{g}_{i}^{l}<t>$. To minimize $Pc$, training will either reduce the gradient of candidate ($\mathrm{sigmoid}^{\prime}(\widetilde{c}_{i}^{l}<t>)$), by pushing $\widetilde{c}_{i}^{l}<t>$ into the saturated flat regions of sigmoid, or try to turn on the gate $g_{i}^{l}<t>$ to preserve the hidden state from the previous timestep $h_{i}^{l}<t-1>$ disregarding the new candidate $c_{i}^{l}<t>$. Either way, it desensitizes $h_{i}^{l}<t>$ to noise injected into $\widetilde{c}_{i}^{l}<t>$. Equations 12–14 provide a quantitative metric to assess the resilience to quantization and noise in a trained GRU, which we use to compare different training schemes.

To quantitatively compare NN accuracy and noise-resilience trained with different methods, we inject the same type of Gaussian training noise during inference and sweep the noise sigma – a common practice adopted in previous NN noise-resilience studies (Yang and Sze, 2019; Klachko et al., 2019; He et al., 2019; Moon et al., 2019; Bennett et al., 2020). We first compare different methods of binarizing activations, then explore different quantization levels for the weights, and, finally, compare RNN training performance using our NNA algorithm versus the popular STE algorithm. Results are summarized in Figure 3.

Comparing training schemes for binarizing activations. As a full-precision (FP) baseline, we first train the NN with FP sigmoid activations and FP weights, without noise injection or quantization. During inference, we add noise $n_{\text{eval}}\sim N(0,\sigma_{\text{eval}}^{2})$ to its pre-activations and evaluate it with both FP sigmoid activations (baseline-$A_{\text{FP}}$-$W_{\text{FP}}$, black), and binarized activations (baseline-$BA$-$W_{\text{FP}}$, yellow), shown in Figure 3a. Trained without noise injection, baseline-$A_{\text{FP}}$-$W_{\text{FP}}$ cannot maintain its high accuracy at large $\sigma_{\text{eval}}$, making it vulnerable to quantization errors, leading to the poor performance of baseline-$BA$-$W_{\text{FP}}$.

To endow the NN with resilience to quantization errors, we use our NNA algorithm and retrain from the FP sigmoid baseline (which is a good initialization point to speed up retraining). Initially, we use large training noise $\sigma_{\text{train}}=\sigma_{L}=1.6$, and plot the inference accuracy with BAs and 7-level weights (NNA($\sigma_{L}$)-$BA$-$W_{7}$, purple) in Figure 3a. Results show high accuracy across a wider range of $\sigma_{\text{eval}}$ than baseline-$BA$-$W_{\text{FP}}$, demonstrating stronger resilience to quantization errors. However, the accuracy peaks at a large $\sigma_{\text{eval}}$ around $\sigma_{L}$, and lower at small $\sigma_{\text{eval}}$, because it is trained to minimize its loss in the presence of this large additive noise. This noise-resilience profile might suit certain noisy circuit environments, e.g., noisy power supply.

In order to also achieve high accuracy at small $\sigma_{\text{eval}}$, we anneal $\sigma_{\text{train}}$ down to a small $\sigma_{\text{train}}=\sigma_{S}$ and further retrain. As shown in Figure 3a (NNA($\sigma_{L}\rightarrow\sigma_{S}$)-$BA$-$W_{7}$, red), peak accuracy is achieved at smaller $\sigma_{\text{eval}}$, demonstrating the efficacy of NNA with annealing (from $\sigma_{L}$ to $\sigma_{S}$). In comparison, directly retraining with $\sigma_{S}$ from the baseline (NNA($\sigma_{S}$)-$BA$-$W_{7}$, green), without the intermediate $\sigma_{L}$ stage, results in much worse accuracy and noise tolerance than NNA with annealing.

In practice, we find the choices of the hyperparameter $\sigma_{\text{train}}$ quite flexible. For the initial large noise, we use a $\sigma_{L}$ to be about 20% of the standard deviation (STD) of the inherent pre-activation distribution, corresponding to $\sigma_{L}=1.6$ for this GRU, though a wide range of values all work similarly well; for the annealed noise, we find $\sigma_{S}=0\sim 0.5$ all achieves optimal results. For temperature $\tau$ in NCN, we find $0.3$ to be optimal: If too small, RNN’s gradient explodes. If too large, BA is not sufficiently approximated.

Loss penalty term interpretations. To understand why the annealing procedure of NNA is critical, we compare the loss penalty terms ($Pg+Pc$ from Equation 13 and 14) of the four networks in Figure 3a. Shown in Table 2, we normalize them by the penalty value of baseline-$A_{\text{FP}}$-$W_{\text{FP}}$, since only the relative values matter for comparison. After retraining with $\sigma_{L}$, NNA($\sigma_{L}$)-$BA$-$W_{7}$’s penalty term reduces to half of the baseline penalty, explaining its higher resilience to noise and quantization errors. After further retraining with annealed $\sigma_{S}$, NNA($\sigma_{L}\rightarrow\sigma_{S}$)-$BA$-$W_{7}$ maintains this small penalty value, even though $\sigma_{S}$ provides less regularization effect – the smaller multiplier $\sigma_{S}^{2}$ in Equation 12 (compared to previous $\sigma_{L}^{2}$) makes the retraining prioritize reducing the raw error (thus higher peak accuracy) over enhancing noise resilience. This means the network can still “memorize” its previous large-noise training regularization effect even after annealing to fine-tune with smaller noise. In contrast, without the intermediate “experience” of large noise training, NNA($\sigma_{S}$)-$BA$-$W_{7}$ has much less regularization effect to reduce its loss penalty.

It should also be pointed out that if evaluated with FP activations and zero noise, all these networks achieve similarly high accuracy as the purely FP sigmoid network, but their accuracy and noise resilience are dramatically different after using binary activations. This implies that trained through different stages, the 4 networks in Figure 3a find distinct regions in the global solution space: retraining with $\sigma_{L}$ finds a promising solution region that’s insensitive to quantization errors, while the further fine-tuning with $\sigma_{S}$ only does a local search to optimize accuracy at the small noise range; in contrast, training without noise injection or only with small noise will not discover the solution region that’s resilient to noise and quantization due to lack of regularization penalty.

Weight quantization levels. Figure 3b shows the GRU’s performance across different numbers of weight quantization levels and confirms the high resilience to weight quantization offered by our NNA algorithm. The resulting performance of 7-level (implemented with a pair of 4-level cells) and 5-level (a pair of 3-level cells) quantization are almost the same as using FP weights. GRUs with 7-level weights achieve the highest peak accuracy at the lower end of $\sigma_{\text{eval}}$ range. Accuracy for GRUs with ternary weights (a pair of 2-level cells) degrades more quickly as $\sigma_{\text{eval}}$ increases. In contrast, GRUs with binary weights, which can be implemented with SRAMs), suffer the most accuracy degradation across the entire $\sigma_{\text{eval}}$ range. Our NNA algorithm’s high resilience to weight quantization makes it possible to use a pair of 3 or 4-level MLCs to achieve performance comparable to GRUs with FP weights. Even GRUs with tenary weights can achieve relatively high inference accuracy for applications with small $\sigma_{\text{eval}}$. Therefore, our optimal PIM architecture (Figure 1c) avoids combining multiple cells to achieve high-resolution weights (Figure 1a, (Shafiee et al., 2016)), thereby saving area and simplifying circuit design.

NNA vs STE. We also experiment with SBN trained with STE, and compare its performance with our NNA algorithm in Figure 3c. We try 3 different settings for slope $s$: $s=1$ (yellow), $s=3$ (blue), and annealing $s$ from $1$ to $3$ ($s=1\rightarrow 3$, green) (Chung et al., 2016). Even with FP weights, all three GRUs trained with STE achieve worse inference accuracy compared to using NNA (NNA($\sigma_{L}\rightarrow\sigma_{S}$)-$BA$-$W_{7}$, red).

We should also point out an important distinction between the forms of SBN (Equation 1) and our NCN (Equation 3): SBN only has one parameter $s$, whereas NCN has two degrees of freedom using $\tau$ and $\sigma_{train}$. On one hand, SBN uses $s$ to control the sigmoid slope (corresponding to $1/\tau$ in NCN), and similar to $\tau$, we find $s$ needs to be no greater than about 3, in order not to run into exploding gradient problem. On the other hand, $s$ also controls the stochasticity of the Bernoulli sampling: a smaller $s$ introduces more randomness thus a higher noise resilience range, as can be seen from Figure 3c, comparing $s=1$, $s=3$ and $s=1\rightarrow 3$. However, SBN cannot seperately control the the sigmoid slope and the stochasticity. In contrast, our NCN has independent controls: $\tau$ is chosen to approximate binary outputs while avoiding exploding gradients, whereas the magnitude of noise injection is seperately controlled by $\sigma_{train}$. This flexiblity enables us to effectively implement NNA algorithm’s annealing procedure using NCN.

From a mathematical rigor point of view, in contrast to the Gaussian RVs used in NCN, the Bernoulli RVs used in SBN do not comply with the “location-scale” distribution required for using the reparameterization trick (Maddison et al., 2016). Therefore, it is mathematically illegal for STE to change the order between taking expectation and taking derivative for Bernoulli RVs (Equation 2). Increasing the slope $s$ can alleviate the discrepancy between the forward and backward pass of SBN (making the math less wrong, which explains the higher accuracy with $s=3$ in Figure 3c), but due to the lack of separate controls, changing $s$ inevitably changes both the sampling randomness and sigmoid’s gradient, and $s$ cannot be too large which will cause exploding gradients.

Need for accurate hardware noise models. So far we have used the same type of additive random Gaussian noise for both training and inference as $N_{\text{train}}$ and $N_{\text{eval}}$ in Figure 3, consistent with prior work that evaluate and compare NN noise resilience (Yang and Sze, 2019; Klachko et al., 2019; He et al., 2019; Moon et al., 2019; Bennett et al., 2020). While prior hardware noise modeling work generally over-simplify by lumping all the noise sources into a single random Gaussian RV, real circuits have a variety of noise sources with properties different from these additive Gaussian RVs, mandating realistic models to faithfully evaluate their impact on hardware inference accuracy.

To evaluate how real circuit noise impacts inference accuracy, we account for the detailed profiles of three important sources of physical noise in PIM circuits shown in Figure 4: (1) the weight noise $N_{\text{MLC}}$ due to variability of eNVM devices, (2) an offset error $N_{\text{OS}}$ due to device mismatch in the sensing circuitry, and (3) a white noise source $N_{\text{white}}$ from thermal and shot noise of the circuits. In the following subsections, we first elaborate on the distinctive characteristics of these three noise sources and present our simulation methodology of the overall noise model; then we introduce the devices and circuit designs of the PIM implementation used for deriving the statistics to build the noise model; finally, we present the hardware validation experiment results.

(1) $N_{\text{MLC}}$ comes from eNVM device manufacturing and programming variability, modeled as noise in programmed MLC weight levels, i.e., per-cell static weight error $\delta W_{k,i}^{+}$ or $\delta W_{k,i}^{-}$ deviating from the intended/ideal value $W_{k,i}^{+}$ or $W_{k,i}^{-}$ (Figure 4). Depending on input activations of each timestep, each cell’s weight error selectively contributes noise to the total weight noise of $N_{\text{MLC}}=\Sigma_{k}(\delta W_{k,i}^{+}-\delta W_{k,i}^{-})\cdot X_{k}^{t}$ added to each corresponding element of MAC pre-activations.

(2) $N_{\text{OS}}$ models transistor-mismatch-induced offset as a sense-amp input-referred static error. Each sense-amp’s random offset is determined after fabrication, which adds a fixed asymmetric bias term into the MAC computation associated with the comparisons of each sense-amp. This offset proves to be detrimental to inference accuracy due to its lack of dynamic randomness, but most prior noise-modeling studies overlook this important source of error.

Among the few papers that address this offset error, (Moon et al., 2019) simplifies it into a Gaussian RV, lumped with other noise sources by adding their variances together, and proposes per-chip, post-fabrication retraining; (Bankman et al., 2019) uses on-chip digital calibration circuitry to cancel the offset after fabrication. In contrast, we dedicate the realistic static noise model $N_{\text{OS}}$ to each offset error, and leverage a simple yet effective circuit technique that avoids the need for individual retraining or the overhead of offset calibration. Our solution is to randomly flip the polarity of the added $N_{\text{OS}}$ using a set of switches controlled by a pseudo random number generator (PRNG), as shown in Figure 4, such that each fixed $N_{\text{OS}}$ magnitude is presented to each element of pre-activations with a random polarity per sense-amp comparison per timestep, thereby converting the effect of each sense-amp offset into a Bernoulli RV. Compared with the smooth “bell-shaped” Gaussian noise with which our NNs are trained, Bernoulli distributions have very different forms, but they can guarantee per-element zero-mean and dynamic randomness across timesteps. These are the traits of noise that NNs trained with our NNA algorithm are surprisingly resilient to, as we will show in our validation experiments.

(3) $N_{\text{white}}$ represents thermal and shot noise from the circuits, which are dynamic random white noise, modeled as an input-referred Gaussian noise source at each sense-amp. This is the only truly random and dynamic source of hardware noise, with exactly the same properties as the Gaussian noise used for training.

The overall noise model accounts for the three aforementioned noise sources, such that the total noise added into each element of pre-activations at each timestep is: $N_{\text{MLC}}\pm N_{\text{OS}}+N_{\text{white}}=\Sigma_{k}(\delta W_{k,i}^{+}-\delta W_{k,i}^{-})\cdot X_{k}^{t}+(-1)^{\text{polarity}}\cdot N_{\text{OS}}+N_{\text{white}}$. When simulating the inference performance of a single chip, $N_{\text{OS}}$, $\delta W_{k,i}^{+}$ and $\delta W_{k,i}^{-}$ are determined/static after fabrication and weight programming, so we only randomly sample them once and fix them during inference on all validation examples; $N_{\text{white}}$ is dynamic and so we randomly generate a new sample for each sense-amp comparison; the PRNG dynamically generates a random selection of $polarity$ to add or subtract the fixed $N_{\text{OS}}$ magnitude for each corresponding sense-amp comparison. We validate hardware performance at the presence of all these noise sources by simulating cycle-accurate inference across GRU timesteps.

Device models and circuit simulations. For eNVM devices, we evaluate three promising emerging eNVMs (ReRAM, PCM, and CMOS-MLC) and model their corresponding $N_{\text{MLC}}$ based on measured data from (Sheu et al., 2011), (Bedeschi et al., 2008), and (Ma et al., 2019), respectively. We design and simulate all peripheral circuits targeting a 16nm FinFET technology (Wu et al., 2014). The sensing circuits that resolve BAs adopt the dynamic bitline discharge technique commonly used in memory reading circuitry, with the self-timed StrongArm-type sense-amp from (Miyahara et al., 2008). Sensing via variable bitline discharge time automatically handles the wide statistical range of bitline currents resulting from PIM-based MAC computations. Transistor sizes in the sense-amp pose a tradeoff between area versus $N_{\text{OS}}$, i.e., enlarging the transistor sizes can reduce $N_{\text{OS}}$ at the expense of a larger sense-amp area overhead. We use Monte Carlo simulations to measure the $N_{\text{OS}}$ of a range of sense-amp sizes, and evaluate their impacts on inference accuracy.

Validation results are summarized in Table 3 and Figure 5. We choose NNA($\sigma_{L}\rightarrow\sigma_{S}$)-$BA$-$W_{7}$ as the optimal design target for circuit implementation and evaluation of the NNA algorithm’s resilience to hardware noise. We establish a baseline accuracy ($mean\pm\textit{STD}=(91.76\pm 0.23)$%) corresponding to when only $N_{\text{white}}$ is present and both $N_{\text{OS}}$ and $N_{\text{MLC}}$ are zero. This baseline accuracy is on par with the peak software inference accuracy in Figure 3. The magnitude of $N_{\text{white}}$ for real circuits turns out to be too small to have measurable impacts on accuracy. We first focus on exploring the performance impact of $N_{\text{OS}}$ and the polarity flipping circuit technique. We then simulate with the entire noise model across three promising eNVM technologies.

As summarized by the upper two rows of Table 3, when we inject the modeled $N_{\text{OS}}$ but fix their polarities, the validation accuracy drops to $mean\pm\textit{STD}=(91.26\pm 0.22)$%. However, by randomly flipping the polarity of each $N_{\text{OS}}$ using the PRNG, validation accuracy of the NN recovers to $mean\pm\textit{STD}=(91.86\pm 0.24)$%, proving the effectiveness of this simple random polarity-flipping technique. The modeled statistical magnitudes of $N_{\text{OS}}$ are derived from our sensing circuitry design with a reasonable sense-amp sizing choice that requires no more than 8 fins of minimum-length FinFETs for each input differential pair transistor. Therefore, our PIM accelerator design ensures minimal peripheral sensing circuitry overhead (mostly from sense-amps) thanks to BAs, in contrast to area- and power-consuming ADCs otherwise needed for higher-precision activations. These validation results also reveal that our NNA algorithm results in NNs that are resilient to noise profiles beyond the additive Gaussian noise with which they are trained (consistent with discoveries in (Joshi et al., 2020; Merolla et al., 2016)) and they are particularly tolerant to dynamic random zero-mean noise, whereas the exact shape of the distribution is less important (no need to be smooth or bell-shaped).

Finally, we evaluate inference performance for PIM designs with binary activations and 7-level weights (each weight implemented with a pair of 2-bit (4-level) MLCs) using the three promising eNVM technologies (PCM, ReRAM, and CMOS-MLC) and with all three types of noise sources (plus random polarity flipping). Figure 5 and the lower two rows of Table 3 summarize the validation results. PCM, ReRAM, and CMOS-MLC achieve hardware inference accuracy of $mean\pm\textit{STD}$ = ($91.60\pm 0.34$)%, ($91.71\pm 0.27$)%, and ($91.60\pm 0.29$)%, respectively, validating that PIM circuits with all three eNVM technologies can achieve performance comparable to the software accuracy (Figure 3) even in the presence of realistic device and circuit non-idealities.