DeepFire2: A Convolutional Spiking Neural Network Accelerator on FPGAs

Figure 1 (d) shows the components used to compose a convolution layer. To precisely adjust the resource utilization and parallelism of the layer, its hardware modules are characterized in two dimensions:

1. 1.

   Number $\kappa$ of kernel units K per layer which compute the convolution of a $3\times 3$ window in the feature map. This is determined by the feature map size.

2. 2.

   Number $\omega$ of parallel weight and threshold units W & T which store weights and thresholds. This depends on the throughput requirements and available hardware resources.

The arrangement of neuron cores C can be visualized as a two-dimensional array of size $\kappa\times\omega$. Each unit W & T is associated with $\kappa$ neuron cores C. Using Figure 1 as an example, the feature map stored in (c) has 5 rows, implying $\kappa=3$ kernel units for (d). For real-world models, however, initial layers can have as much as 224 kernel units for ImageNet. In order to retain a high clock frequency and, hence, high throughput, re-timing registers are introduced to partition large neuron core arrays. In DF2, groups of 8 neuron cores are formed. The grouping for selected values of $\omega$ ranging from 2 to 16 is shown in Figure 1 (d.1 to d.4). (d.1), where each kernel has 2 cores, 4 kernels (K$<$3:0$>$) are grouped. W$<$1:0$>$ and T$<$1:0$>$ signals are re-timed and passed to K$<$7:4$>$. On the other hand, 2 kernels are grouped when each kernel has 4 cores (d.2). When each kernel has more than 8 neuron cores, multiple groupings are formed within the kernel itself as shown in (d.3 and d.4). A group size of 8 allows good timing closure on our test platform, but the optimal number may vary for different platforms. Therefore, the DF2 IP can adapt by changing the grouping parameter.

Split-kernel mapping allows the adjustment of the degree of parallelism to trade-off throughput and hardware resources. A larger $\omega$ instantiates more parallel neuron cores, increasing the utilization of DSPs and LUTs. In addition, $\omega$ influences the shape of memory blocks as demonstrated in Figure 2 (a), (b) and (c). Splitting a layer across multiple SLRs along the $\omega$ dimension becomes necessary if one SLR does not have enough memory to accommodate all parameters. This applies especially to the last layers in a neural network. Given the number of kernel weights (64 in this example), the user can choose different memory depths by cascading memory block primitives. The kernel map in Figure 2 (a) uses shallower memory depth of 4 neurons with 16 weight units W0 to W15 to store the 64 kernel values, leading to 16 parallel cores. On the other end of the spectrum, it can also be reconfigured to 4 weight units with deeper memory and 4 cores to reduce the throughput and conserve computing resources (c). An input beat is the data transfer of input spikes, whose size is fixed to 8 according to the neuron core design (Section 3.2). How the eight spikes are mapped to the neuron cores is highlighted in the figure. Three-way splitting of the kernel across three SLRs is possible as shown in Figure 2 (d), where one third of the weight units and cores are assigned to every SLR. Alternatively, the user can also choose an asymmetric two-way split, where two thirds of the kernel values are assigned to SLR0 and the remaining ones go to SLR1.

When splitting layers across multiple SLRs, the primary SLR accommodates not only the allocated C, W & T, but also the controller and the feature buffer. Secondary SLRs transceive only spiking features through RX and TX bridges. Since all the kernel operations are contained within each SLR, the strain on inter-SLR routing resources is relieved. The bridges are designed with multiple clock roots as suggested by [41] for high-speed data crossing. The output features from the cores are merged at the MUXs in Figure 1 (e), which are scheduled in round-robin style and route data to the feature buffer of the next layer. The the following sections, the pipeline components are explained in detail.

Our DF2 neuron core design shown in Figure 3 is the RTL version of the integrate-and-fire (IF) neuron. It processes eight input spikes si[7:0] and their weights w[63:0] in parallel, where weights are quantized to 8 bits. The operation includes an AND, addition, and accumulate & fire stage. When accelerating large networks, a substantial number of neuron cores are instantiated in the FPGA. This renders the previous DF1 neuron core design [14] unusable as it consumes too many lookup table resources, limiting the scalability of the accelerator. Our improved neuron core design reduces the use of LUTs by routing weights w directly to the register input D. The AND operation is carried out by connecting inverted input spikes si to the synchronous reset R of the registers. If an input spike is not present the reset is asserted, which overrides input D and sets the data output Q to low in the next clock cycle.

Saving a few LUTs on a hardware module can have a great impact when it is instantiated many times to deploy large-scale models. Registers are also more abundant in Xilinx UltraScale+ devices, since each configurable logic block (CLB) contains eight 6-input LUTs, but 16 flip-flops. Moreover, using a 6-input LUT for an AND operation with only two inputs wastes resources. Lastly, these AND registers also act as the re-timing registers. They allow DF2 to be run at a higher clock speed, since they relax the placement distance between the result of the AND operation and the DSP slice of the adder tree.

The adder tree consists of three stages, where the first stage is embedded in DSP blocks in SIMD configuration and the remaining two-stages are implemented in CLBs utilizing LUTs and carry-chains. The adder tree result is then accumulated and compared against the threshold parameter t, which was trained together with the weights. The neuron core fires an output spike, i.e. $\textit{so}=1$, if the membrane potential exceeds t, and remains silent otherwise. The entire operation of the core is pipelined with an accompanying enable signal pipe_en, which streams through the core together with the data. That translates into a signal so_en at the output, which indicates a valid spike value.

As we adopt the data-flow architecture from FINN [42], the feature buffer (FBF) is important to hold and transfer spike features seamlessly from one layer to another. In DF2, two-stage buffering of the feature maps is proposed as shown in Figure 1 (a), (c) and (f).

An intermediate buffer (c) uses the first stage for feature column storage. It will gather all the output spikes for a particular column from the previous layer and pass them to the FIFO (first in, first out) buffer in the second stage. Each FIFO holds three feature columns for 3$\times$3 convolution. In the specific case of (c), the feature map has five rows. Hence, three FIFOs are instantiated for the three overlapping kernel windows in vertical direction (depicted are two FIFOs). The spiking features are sent to the neuron core bus si. At the end of each convolution, the FIFOs will discard one column and, thus, shift the convolution windows in horizontal direction along the rows. Figure 1 (a) and (f) show special cases of the FBF, as they precede the transduction and fully-connected layers, respectively.

The transduction layer is the first layer in the DF2 network where the raw image data are transformed into spiking signals (Figure 1 (b)). It closely follows the neuron core design described above. But since both weights w and activations are 8-bit operands, a multiplier is used instead of the AND operation. DSP slices are used to implement the multiplier. This is followed by the adder tree, accumulator and comparator. The decision of whether an output spike is fired or not, depends again on the membrane potential after accumulation and the trained threshold t. The binary spikes are queued in a buffer (1:8) to convert the data width from 1-bit spike values to 8-bit column features. The column features are forwarded to the next FBF.

There is a controller at every SNN layer. It coordinates the kernel operation of the layer as well as performs handshake with the layer before and after. The controller at layer $n$ will only start the kernel operation upon meeting both of the following conditions.

1. 1.

   The second stage of the FBF in layer $n$ is loaded with the features required for the current convolutional window.

2. 2.

   The first stage of the FBF in layer $n+1$ is empty that output column features can be stored when the kernel operation starts.

This condition ensures that there is no data overflow in the FBFs. When the first state of $\text{FBF}_{(n+1)}$ is full, it will naturally create a backpressure all the way back to the transduction layer until it is cleared. The backpressure scenario can be avoided by manipulating the number of the weight units $\omega$ in each layer (more details in Section 4.1).

To characterize the DeepFire2 architecture for neural network of different sizes, we implemented models for a variety of datasets, as shown in Table I. They closely follow VGG-9 and VGG-11 architectures with a customized neuron count for each layer. The neuron count is chosen to efficiently utilize the fixed-size BRAM and URAM blocks in the device. For example, mapping a kernel with 112 neurons to 4kB large memory results in instantiation of 8 BRAM blocks, each containing the weights of 14 neurons. This leads to 88.8% utilization for each 4kB-BRAM. 128 neurons, as per original VGG configuration, results in 16 BRAMs instantiation with 8 neurons per BRAM block (50.8% utilization). A number between 8 and 16 is not possible because of the byte write operation to the FBF. In summary, the number of weight units W for weight parameters in each kernel must satisfy the following condition.

where $i$ is the integer greater or equal to zero. The maximum pooling layer in the conventional VGG architecture is replaced by a 2$\times$2 convolution with a stride of 2 and valid padding. Hence, DF2 pooling layers carry trainable weights. Table I also presents the cascading parameters for BRAM and URAM for each layer.

The neural network architecture dictates the allocation of memory blocks. The default BRAM and URAM modules in the FPGA can be cascaded to form larger memory. As layers become wider, more parameters are used and, hence, deeper memory is required to store these parameters. Each URAM block has a capacity 8$\times$ as large as a BRAM block. Moreover, URAM has a deep internal pipeline for data readout and thus, it is capable of cascading multiple URAM blocks without sacrificing its maximum clock frequency Fmax [43]. Whereas for BRAM, its Fmax performance drops when blocks are cascaded. Therefore, the usage of URAM is preferred for wider layers later in the model. For our larger models, however, up to 16 BRAM blocks had to be cascaded due to a lack of URAM resources, which lead to a slight drop in frequency Fmax.

The memory depth influences parameters like the number of weight units $\omega$. Making memory blocks shallower, but wider, increases parallelism and resources. The initial layers, where high-resolution feature maps get processed, are instantiated with shallow memory depth and many neuron cores. The deeper layers with lower feature resolution are allocated deeper memory and a lower number of cores. In an optimal setting, the data flow from the first layer to the last layer will be seamless without any backpressure in the pipeline. Through this setting, the DF2 IP can be adapted to various FPGA platforms and it is able to maximize the utilization of compute resources for maximum performance and low power consumption.

The first version of DeepFire [14] demonstrated its scalability by layer-wise SLR mapping. This allows it to scale for a deeper network. The limitation of this strategy is that layers can no longer be mapped into a single SLR, if they have too many parameters. As an example, the deeper layers Fc-1200 and Fc-1056 from Table I require more memory resources than a single SLR can offer and it would not be possible to map these layers using the original DF1. DF2 introduces the split-kernel mapping for those heavily parameterized layers. Therefore, DF2 can support not only deeper but also wider networks.

The implemented VGG-11 mappings of Tiny-ImageNet and ImageNet are shown in Figure 4 with respect to their dataflow. A few initial layers with less weight parameters adopt layer-wise mapping and the following heavily parameterized layers adapt split-kernel mapping. DF2 provides the flexibility to carry out optimizations on a higher level of abstraction, which is not within the scope of this paper. While we mapped layers manually by estimating their required numbers of SLRs, other more optimal mapping strategies could certainly be applied. The ultimate goal of this mapping is

• •

  to enable efficient distribution of FPGA resources across multiple SLRs and avoid bottlenecks caused by lack of memory or compute resources,

• •

  to maximize the use of available resources in line with overall performance goals.

The results of the Vivado post-implementation on five different SNN architectures are shown in the Figure 6. Each unique color represents a layer in the network. As the network grows from Figure 6 (a) to (e), DF2 scales from one SLR to three SLRs. In theory, it can scale to an unlimited number of SLRs but in practice, due to the limited scale of production, only up to four SLRs per package are available commercially [33].

Figure 5 shows the distribution of DF2 resources with different SNN architectures. When an SLR is assigned with the compute intensive initial layers of the network, it will consume a lot of DSP resources but yet, a lot of on-chip memory will be left unused. On the other hand, the deeper layers will deprive the SLR of all the on-chip memory resources and leave many of the DSP resources untouched. Regardless of the number of assigned SLRs, the flexible DF2 mapping is capable of distributing the resource utilization for both memory and DSP evenly across multiple SLRs. That prevents a single type of resource from capping the performance. An additional advantage is a good power distribution throughout the large silicon substrates. Hence, the system is less prone to local drops in supply voltage, making it more robust. Since all the assigned SLRs share the resources almost equally, any localized routing congestion is also minimized. Figure 5 (a) shows the resource utilization of Cifar-10 and due to its small memory footprint, its implementation is confined to SLR1. Cifar-100 is implemented in SLR1 and SLR2 and (b.1) shows the equal distribution of resources among the two SLRs when both layer-wise and split-kernel mappings are applied. The resources are unbalanced in two SLRs of (b.2) despite putting the best effort in layer-wise only mapping. In that case, the URAM resources in SLR1 are exhausted, limiting further performance improvements. DF2 is also able to map the larger Tiny-ImageNet and ImageNet network in two and three SLRs efficiently, achieving almost 80% utilization in both memory and compute resources ((c) and (d)).

The DF2 subsystem consists of PCIe DMA, DDR memory, AXI DMA, AXI interconnect, and a clock generator. PCIe DMA is mainly used for the host PC to send images and network parameters (weights and thresholds) to the FPGA DDR, and then reads back the classification results from the same DDR. AXI DMA is responsible for sending images from DDR to DF2 and writing classification results from DF2 to DDR. AXI interconnects provide routing for all the AXI-related IPs and enable clock domain crossings. The clock generator uses the reference clock from PCIe to generate a higher frequency clock for DF2 IPs. The post-placement result of the subsystem is highlighted in Figure 6 (a).

The DF2 compilation flow (Figure 7) starts with the user defined config.json file. It contains SNN network information for TensorFlow training such as kernel size, channel and stride information for each layer, etc. The SNN training is performed just like CNN training using standard back-propagation with batch normalization at every layer. This version of DeepFire supports padding which positively affects the classification accuracy. We start training with a leaky ReLU activation to avoid overfitting. Later in the training, we switch to tanh activation. We also make use of data augmentation. The outputs of the activation function are binarized and a straight-through estimator is applied for back-propagation. Finetuning the model with spike activations helps to maintain a high accuracy while using fewer neurons.

FPGA-specific mapping information, such as kernel split and memory depth for each layer, are used to extract the weights and threshold parameters from the TensorFlow graph in accordance with the data sequence the FPGA is expecting to receive. During the extraction, 8-bit quantization is performed. The threshold value of each neuron is computed by folding the activation with the batch normalization parameters and quantizing them with the respective scale factor from the weight quantization. The same config.json is also used to generate the RTL wrapper and FPGA-specific constraints for the SLR bridges and the floor planning. We created a tcl script in the Vivado tool to build the DF2 subsystem from the DeepFire IPs. Once the subsystem is built, it will automatically trigger the synthesis and finally, produce an FPGA bitstream.

A detailed performance comparison between DeepFire2 and previous SNN hardware implementations is provided in Table II. The table rows are grouped by dataset. While there have been plenty implementations for smaller datasets, like MNIST and Cifar-10, only DF2 is able to handle Tiny-ImageNet and ImageNet. In MNIST image classification, DF2 produces 10.26 kFPS/W which is only 20% shy of TrueNorth’s ASIC efficiency (see also Figure 8 (a)). At the same time, DF2 is at least one order of magnitude more energy efficient than other FPGA implementations [35, 44, 34, 36, 14], and three and five orders of magnitude superior to GPU and CPU, respectively [35, 34]. The maximum throughput of DF2 is 79 kFPS which almost doubles the original DF1 design [14]. This is due to the improved pipelining at the neuron core level and the higher DF2 clock rate. DF2 throughput for MNIST can be much higher if we reduce the depth of the weight units in the network and deploy more neuron cores. DF2 Fmax is 600 MHz which is at least 3$\times$ higher than non-DF FPGA implementations. ASICs such as TrueNorth can achieve a relatively high throughput at 1 kFPS. However, it falls short of our FPGA throughput. GPUs acceleration is much superior to CPU throughput in general, but their energy efficiency is limited by the dataflow being bound to external DDR memory.

In addition to the frames-per-second, we report the performance of the listed neuromorphic architectures in GOPS (giga-operations per second) as a method of normalizing the throughput metric, thus making it independent of the processing required by a specific SNN model. Our GOPS measurements are calculated as the product of the ANN-equivalent number of operations per inference and the number of inferences per second. Similarly, we evaluate the power efficiency in GOPS per Watt.

Cost-effectiveness is an important consideration when it comes to large-scale deployments, for example in data centers. It is determined by normalizing the cost of the board to the throughput in FPS. How effectively the resources of a given device are used, i.e. the choice of compute architecture and various optimization strategies, greatly influences cost-effectiveness. In that regard, CPU is the least cost-effective solution and FPGA-based DF1 [14] and DF2 are the most cost-effective implementations across all hardware platforms. Figure 8 (b) also shows examples of FPGA implementations being far worse than the GPU counterpart. The cost-effectiveness of TrueNorth and Loihi cannot be measured because they are not commercially available.

When it comes to Cifar-10 classification, DF2 is still the most energy efficient SNN on an FPGA. Since we trade off the compute resources for better compute efficiency, our throughput is slightly less than [14]. Better throughput and energy efficiency is expected in DF2 than that of [36] since we store all the weight and activation values on-chip. Furthermore, DF2 spike trains are shorter, while not sacrificing or even improving the classification accuracy. TrueNorth is 23% better than DF2 in terms of FPS/W despite having to bridge four chips to map the large SNN. For the Cifar-100 dataset, TrueNorth has to bridge eight chips to accommodate a larger SNN and it is almost on par with DF2’s energy efficiency, which bridges only two SLRs.

DF2 demonstrates the mapping capability of a larger SNN for Tiny-ImageNet. DF2 achieves 46.7% accuracy while its mapping is limited to two SLRs. Despite using over 90% of on-chip memory and DSPs across two SLRs, DF2 can still clock at 450 MHz delivering 10.3 kFPS. DF2 also shows its scalability across multiple SLRs without compromising its Fmax for an ImageNet SNN, where it utilizes all three SLRs depleting almost all the on-chip DSPs and the memory resources (Figure 5 (d)). Even when operating at the limits of the VU9P FPGA, it can maintain a high performance. But approaching those limits does not allow the storage of all of the VGG-11 parameters, and the ImageNet accuracy on DF2 is limited to 40%. This shows the general trade-off between accuracy and performance when neuromorphic hardware is compared to software implementations. But avoiding DDR accesses for weight transfer has significant advantages in terms of power consumption. The DF2 ImageNet performance is 21 TOPS (tera-operations per second) which about 78% of the total DSP performance and it delivers approximately 86 images per TOPS.

Overall, DF2 is one of the most energy efficient and most scalable SNN implementations on FPGA, and it can provide the highest throughput among peers.