Efficient yet Accurate End-to-End SC Accelerator Design

We refer to the accumulation and activation module as the SC non-linear adder. Typical SC Non-linear adders employ stochastic coding with FSM to implement different activation functions  [6, 7, 8, 9]. FSM-based designs serially process stochastic bitstream inputs, which results in inaccurate outputs (Figure 1) that do not utilize all of the information in the inputs and have random fluctuations in the inputs themselves. Thus, very long bitstreams, e.g., 1024 bits, are used for accuracy and lead to an unacceptable latency, which severely affects the hardware efficiency.

In our work, we employ the deterministic thermometer coding scheme (Table II) and the corresponding accurate SC circuit designs to achieve accurate end-to-end SC acceleration. With thermometer coding, all the 1s appear at the beginning of the bitstream and a value $x$ is represented with a $L$-bit sequence as $x=\alpha x_{q}=\alpha(\sum_{i=0}^{L-1}x[i]-\frac{L}{2}),$ where $x_{q}=\sum_{i=0}^{L-1}x[i]-\frac{L}{2}$ is the quantized value of range $[-\frac{L}{2},\frac{L}{2}]$ and $\alpha$ is a scaling factor obtained by training.

Deterministic coding, in contrast to stochastic coding, achieves hardware-efficient and accurate computations with shorter bitstreams. By employing a 2-bit ternary bitstream, we can realize multiplication with only 5 gates using a deterministic multiplier (Figure 3(a)).

To achieve accurate accumulation and activation functions simultaneously, we employ the bitonic sorting network (BSN). BSN is a parallel sorting network that sorts inputs in thermometer coding, ensuring the output is also in thermometer coding. The sorting process, performed by comparators constructed with AND and OR gates, follows Batcher’s bitonic sorting algorithm [13] (Figure 3(b)). The number of 1’s in the sorted bitstream output from BSN corresponds to the sum of 1’s in all input bitstreams, effectively representing the accumulation result.

By sorting all the bits, the inputs and outputs of the selective interconnect (SI)[14] are deterministic. Therefore, when the SI selects different bits from the BSN directly as outputs based on the selection signals, a deterministic input-output correspondence is generated and different activation functions are realized. The example in Figure 3(b) implements the two-step activation function shown at the bottom when the SI selects the 3rd and 6th bits of the BSN as outputs. We refer interested readers to [3, 4] for more details.

We prototype the proposed SC accelerator with a 28-nm CMOS process. The chip’s measured current consumption and energy efficiency in Figure  4 show a peak of 198.9 TOPS/W at 200 MHz and 650 mV. Compared to state-of-the-art binary-based NN processors [15, 16, 17, 18, 19], the fabricated SC-based NN processor achieves an average energy efficiency improvement of 10.75$\times$ (1.16$\times\sim 17.30\times$). And the area efficiency improves by 4.20$\times$ (2.09$\times\sim 6.76\times$). We also compare the accuracy under varying bit error rates (BER) using a ternary neural network that achieves 98.28% accuracy on the MNIST dataset, as shown in Figure 5. The proposed SC design demonstrates significant fault tolerance, as the average reduction of accuracy loss by 70%. It is the first silicon-proven end-to-end SC accelerator, to the best of the authors’ knowledge.

The SC TNN accelerator in Section II lacks support for batch normalization (BN) and residual connections, limiting its accuracy on complex datasets like CIFAR10 or CIFAR100. Increasing precision can enhance accuracy but compromises hardware efficiency. Figure 2 demonstrates that increasing BSL from 2 to 8 bits improves accuracy at the expense of a 3 to 10 times efficiency overhead. Accurate yet efficient SC acceleration is very challenging.

To understand the origin of the accuracy degradation, we quantize the network weight and activation to low precision separately. Table III shows similar accuracy between low precision weight quantization and the floating point baseline, while 2b BSL activation quantization results in a 10% accuracy drop. Hence, low precision activation is the root cause of the accuracy loss due to its limited representation capacity. After quantization, the range of activations is reduced to $\{-1,0,+1\}$ for 2b BSL encoding, significantly reducing the number of possible configurations.

As a remedy, we add the high-precision activation input through residual connections to the result of the low-precision convolution (Figure 6). By increasing the activation range to $\{-8,-7,\ldots,7,8\}$, we enhance representation capacity to $17^{H\times W\times C}$. This significantly improves inference accuracy while maintaining efficiency by preserving energy-efficient convolution computation.

Besides the high precision residual, another remaining question is how to efficiently process BN. And $BN(x)=\gamma(x-\beta)$, where $\gamma$ and $\beta$ are trainable parameters. We propose to fuse BN with the ReLU activation function as Equation 1. Consequently, we achieve an SC-friendly low precision model with high precision residual fusion depicted in Figure 6(b).

Compared to the proposed accelerator in Section II-B, the model in Figure 6(b) further requires the implementation of SC circuits for BN fusion and residual connection.

The above fused BN and ReLU function can be efficiently and accurately processed in SC, leveraging the selective interconnect described in Section II-B. Figure 7 demonstrates how different BN parameters affect the objective function of the SI.

For the accumulation of residual and multiplication products, the different scaling factors $\alpha$ of residual and convolution results can lead to errors in the accumulation operation. The residual re-scaling block is proposed to align the $\alpha$ before accumulation. In the re-scaling block, we multiply or divide the residual by a factor of $2^{N}$ (where $N$ is an integer). To multiply the residual by $2^{N}$, we replicate it $2^{N}$ times in the buffer. For division by $2^{N}$, we select 1 out of 2 bits of the residual per cycle and generate the final result after $N$ cycles. To maintain a constant BSL for the residual, we append 8 bits of ’11110000’ (equal to 0) per division cycle.

Figure 8 demonstrate significant improvement in network accuracy. With the high precision residual, network accuracy is improved significantly by 8.69% and 8.12% for low precision ResNet18 on CIFAR10 and CIFAR100, respectively. Combined with the novel training techniques, network accuracy can be improved in total by 9.43% and 15.42%. Compared to baseline accelerators, it achieves a 9.4% accuracy improvement with only a 1.3% efficiency overhead compared to the efficient baseline and achieves a 3$\times$ efficiency improvement with comparable accuracy to the accurate baseline design, as shown in Table IV. In this way, the proposed method achieves accurate yet efficient SC acceleration.

BSN accumulates all the input in parallel through sorting, so as to generate an accurate output based on all the information input. However, it also forces the hardware cost to increase super linearly with the accumulation widths (Figure 9(a)). And the BSN has to support the largest accumulation widths among all layers. The large BSN, however, leads to very high hardware redundancy at shallow layers where the accumulation is always small (Figure 9(b)). This makes our previous design still inefficient for SOTA models.

To address the inefficiency and inflexibility of BSN, we find a significant precision gap between the input and output of SI, as revealed in Figure 6(b), making the high precision SC input redundant. We reduce the BSN output BSL, resulting in a small accuracy loss for the tanh function and negligible impact on the ReLU function, as shown in Figure 10(a).

To further reduce hardware cost, we adopt a progressive sorting and sub-sampling approach for the BSN. Figure 10(b) presents a parameterized BSN design space that determines the location, number of sampling times, and method of sampling. The parameterized BSN consists of $N$ stages and in the $i$th stage, there are $m_{i}$ sub-BSN modules, each taking an input bitstream of $l_{i}$-bit BSL. Within each sub-BSN, there is a sub-sampling block that implements truncated quantization. It clips out $c_{i}$ bits on each end of the BSN while sampling 1 bit every $s_{i}$ bit from the remaining. Considering the input distribution resembles a Gaussian distribution with a small variance due to inputs from a large number of multipliers, significant clipping can be performed with negligible errors, as illustrated in Figure 11.

Thanks to the fact that the output BSL of the approximate BSN is much shorter than the input, we can further fold the accumulation temporally to achieve more flexibility. In this case, as shown in Figure 12, a large BSN is implemented by multi-cycle reuse of a single small BSN circuit. In the proposed spatial-temporal BSN architecture, the approximation level of BSN, i.e., the BSL of partial sums, and its corresponding reuse can be controlled through control signals. This allows for flexible handling of various accumulation widths with different approximate configurations.

For the largest convolution in the ResNet18, the two proposed approximate BSN reduced the ADP of BSN by 2.8$\times$ and 4.1$\times$ compared to the baseline, as shown in Table V. When handling the four different sizes of convolutions in ResNet18, the spatial-temporal BSN needs fewer cycles for smaller convolutions and achieved ADP reductions from 8.2$\times$ to 23.3$\times$ with negligible errors, as shown in Figure 13. On average, the spatial-temporal BSN reduces the 2.2$\times$ area of datapath by reducing the average ADP of BSN by 8.5$\times$. This shows that the proposed SC design is more flexible and efficient.