On the RTL Implementation of FINN Matrix Vector Compute Unit

Deep convolutional neural networks, referred to simply as CNNs, have shown a tremendous growth in the past many years with networks now having millions of parameters. E.g., AlexNet, VGGNet and ResNet-152 have $62M$, $138M$ and $60.3M$ parameters, respectively [1, 2, 3]. The computational cost of a CNN is primarily derived from convolution layers. These convolution operations are typically performed between input matrices, which can be as large as $224\times 224$ with multiple channels as in the popular ImageNet category[4], and filter kernels, which are typically small on the order of $5\times 5$ [5] or $3\times 3$ [2]. The implied convolutional compute amounts to many dot products between the filter kernels of all output channels and correspondingly sized, overlapping tiles of the input across the full depth of input channels. Many networks conclude the convolutions with a small number of fully connected layers. These layers, indeed, compute one large dot product for each output channel but without any stencil tiling. In consequence, convolutional layers dominate the total execution time of modern CNNs. In total, these compute requirements are challenging in CNN deployments especially on resource-constrained devices, say, for enabling Internet-of-Things (IoT) use cases both embedded and at the edge. The deployment of such networks on edge devices is critical to enable technologies like smart cities, driver-less cars and other autonomous systems [6] as it is neither energy-efficient to submit data from an edge device for remote cloud processing nor conducive for the real-time processing required by such state-of-the-art applications.

The challenging demands of the desired CNNs must be mitigated on embedded devices with limited processing and memory capacity. It is important to reduce the number of layers or parameters, and the memory required to store them. There are various ways in which these aspects of a network can be reduced. One of the ways is referred to as pruning. It involves replacing insignificant parameters in weight tensors by zero. Pruning can be performed at different levels of granularity ranging from pruning individual weights to complete channels [7, 8].

Another orthogonal measure of reducing the memory footprint of such networks is quantization. It means that the data inside a network (i.e., input and output activations, weights and biases) are represented using shorter word lengths. Admissible word lengths have been explored thoroughly for fixed- and floating-point number systems. Capotondi et al. [9] propose a CNN inference library, called CMix-NN, for integer-only mixed low-precision quantization of weights and activations at $(8,4,2)$-bits. The library is optimized for the ARM instruction set with vector arithmetic extensions and tested for the deployment on microcontroller targets like the STM32H7. Garofalo et al. [10] also propose a library of kernels for quantized neural network (QNN) inference, called PULP-NN, using a homogeneous quantization scheme with $(8,4,2,1)$-bits. They optimize for a cluster of low-power RISC-V processors by exploiting $4\times 8$-bit SIMD MAC instructions and bitwise extension operations in order to benefit from precisions below 8 bits. Bruschi et al. [11] extend PULP-NN by targeting the acceleration of mixed-precision DNNs. Hubara et al. [12] present the quantization with various low-bit representations as small as $1$ bit per weight and activation to allow for the use of bitwise operations during the forward pass. A further contribution in the field of these binary neural networks (BNNs) is reported by Rastegari et al. [13]. They present two approximations of DNNs by firstly representing all weights with binary values and secondly approximating both weights and inputs to the convolutional and fully connected layers by binary values. Their interpretation of the two values as $\pm 1$ motivates their term XNOR network as inspired by the resulting elementary multiplication.

Field-programmable gate arrays (FPGAs) are a very suitable platform to implement fixed-point systems due to the presence of fast multipliers and adders. Their ability to customize datapaths down to the bit level has made them an attractive target and the technology of choice for very-low-precision QNNs and, of course, BNNs. One framework exploiting this technological match is FINN, which is able to build FPGA-based accelerators for QNNs. While initially limited to BNNs [14], FINN was later extended to non-binary word lengths [15]. The prototypes designed using this framework were demonstrated to improve performance measures, such as latency, throughput, Top-1, Top-5 and mean average precision (mAP) accuracies, with respect to the state of the art. The FINN framework uses C++-based high-level synthesis (HLS). HLS allows software engineers and scientists, not well versed with hardware description languages (HDLs) like Verilog and VHDL to describe digital systems. They can, thus, target hardware platforms like FPGAs and ASICs using familiar languages like C++ or OpenCL [16, 17, 18]. Designs described in these high-level languages are translated to traditional HDLs using dedicated compilers. The time required to develop and design a digital system using HLS is considerably less than what is needed for a genuine HDL implementation. The general reduction of code complexity is a benefit for design space exploration, optimization, and maintenance. Also, the debugging of a design can ideally be performed on a higher functional abstraction level, which is more accessible to the application engineer and considerably faster than a simulation on the register-transfer level (RTL). Finally, HLS offers more flexibility in code exploration, which is powerful for example when evaluating the many options for ordering different dimensions in tensors, convolutional layers or different degrees of parallelism.

On the other side, adding an extra layer of abstraction and translation also incurs costs in terms of design implementation times, hardware resource cost and the predictability of design quality. In this work, we analyse HLS- and RTL-based implementations of the core compute unit of the accelerator designed for the FINN framework to evaluate the implied trade-offs. This core unit performs matrix-vector multiplications using different types of compute engines and uses AXI-Stream-based interfaces for communication. We use Xilinx’s Vivado and Vivado HLS for RTL and HLS implementations, respectively.

The main contributions of this work are:

• •

  RTL implementation of the key FINN compute component, the MVU, available in open source at https://github.com/asadalam/FINN_MatrixVector_RTL.

• •

  Systematic measurement and analysis of resource utilization, critical path delay and synthesis time for designs realized using HLS and RTL. We show that:

  • –

    HLS uses more FPGA resources for smaller designs and suffers from complex multiplexer structures when processing large input streams using a limited number of compute units.

  • –

    The LUT usage converges between HLS and RTL as the core compute unit increases in size but HLS consistently uses more flip-flops and block RAMs (BRAMs).

  • –

    The RTL implementation results in a significantly reduced critical path delay across different parameters and for the three different types of compute units as compared to HLS

  • –

    HLS takes at least $10\times$ more synthesis time as compared to RTL synthesis and is the limiting factor towards synthesizing and analyzing designs with large numbers of compute units.

• •

  Implementation and analysis of a multi-layer perceptron (MLP) network, consisting of 4 fully connected layers, for network intrusion detection with the UNSW-NB15 dataset [19].

This paper is organized as follows: Section 2 highlights the differences between HLS and RTL and the implied design and workflow trade-offs before Section 3 provides a literature review of related work. Section 4 briefly describes the FINN platform for generating HLS architectures for QNN inference. The architecture of the matrix vector compute unit and its RTL refinement are detailed in Section 4.1.1. The results of our HLS vs. RTL comparison are finally presented in Section 6 before Section 7 concludes the paper.

The design of digital microelectronic systems has evolved tremendously over the years. Starting with the manual placement of transistors, it eventually embraced hardware description languages (HDLs) as a major milestone boosting reuse and productivity. HDLs can describe a digital design at various abstraction levels, commonly identified as the behavioral, register-transfer, gate and switch levels. Typically, a combination of behavioral and register-transfer level is used for design entry with few instances of gate level design where detailed control is necessary. Conventionally, the term RTL design is used as an umbrella subsuming the adjacent abstraction levels as used for typical design entry.

RTL design entry was an important but by far not conclusive step towards a functional, software-like specification of hardware systems. Behavioral control structures, like if- and case-statements, and high-level logic and arithmetic operators raise the abstraction level significantly. However, designers must still schedule their computation into clock cycles explicitly and deal with low-level control details, such as the synchronicity of clock domains and the polarity of reset signals. Listing 1 gives a small impression of this style of design entry modelling a D-Flip Flop with a synchronous active-low reset and an active-high enable.

Three HDLs are in widespread use today: VHDL, Verilog, and SystemVerilog. SystemVerilog is the youngest among them and is closely related to Verilog. All of them require special skills in RTL design and are exclusively used by hardware engineers to implement designs on hardware platforms like application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs) and to build testbenches for these designs.

In order to enable software engineers, research scientists and others to deploy and accelerate their applications on FPGAs without the need for writing RTL, high-level synthesis (HLS) tools were envisioned and created [20]. These efforts yielded, for example, Maxeler’s OpenSPL [21], Intel’s OpenCL [22] and Xilinx’ Vitis unified software platform [23]. While Intel’s HLS [24] uses OpenCL, the Xilinx Vitis platform uses C++ as its design entry language. Pragma annotations, comparable to those used by OpenMP, guide the RTL synthesis by the HLS compiler. These pragma recipes and language restrictions, particularly the lack of dynamically allocated memory and late virtual function dispatch, still pose a learning and implementation challenge. This challenge can, however, be approached within familiar C++ terrain.

HLS technology has, indeed, demonstrated its ability to make FPGA acceleration accessible to a wider audience. It has triggered a number of contributions using HLS in diverse fields, e.g., scientific high-performance computing (HPC) [25, 26] as well as machine learning applications like routing congestion prediction [27] and deep neural networks [28]. FINN [14, 15] is also an example for realizing accelerators for neural networks on FPGAs using C++ HLS.

The design entry in HLS comes with significant benefits. The time and effort to get an initial design up and running is significantly lower. Its functional validation can be conducted efficiently in software. The higher-level description is much more flexible and, hence, more amenable for a thorough design space exploration. Re-arranging loop nests and tuning the degree of parallelism are small changes that can be evaluated quickly. Finally, the quality of the hardware designs generated from HLS has been improving significantly and constantly over the recent years which led to meanwhile widespread adoption of this new form in descign entry.

Designing on a higher functional abstraction level does, however, also incur costs. First of all, HLS produces some wrapping overhead for standardizing all control and communication interfaces of a design. Internally, HLS tools can brilliantly apply formalized knowledge about transforming and optimizing standard code structures. However, they have no intuition about exploitable application constraints and the consequences of custom control structures. They do not automatically push into optimizing congested parts of the design and regularly fail in scaling a design up or in meeting the expected timing. When a critical generated kernel violates physical design constraints, the available means and levers are much less clear. Cutting one or two LUT levels from a critical path that did not meet timing typically turns into a massive investigative endeavor, which cannot even be expected to be approached by the high-level audience of HLS users. Last but not least, current HLS compilers perform an additional compilation step with synthesis times that clearly grow superlinearly. The compilation of designs that somewhat fill modern high-end devices may take days. This may squander the gains of faster turnaround times in earlier design phases. In the end, the system implementation cost of an HLS application may suddenly surge when operating the tools close to the limits of the target platform. Particularly for such challenging and for settled use cases, RTL still poses as a worthy option.

As HLS design entry has become main stream, a number of research contributions for its analysis and evaluation have been made. Nane et al. [29] present a survey and evaluation of FPGA HLS tools. In addition to evaluating various tools, they also describe different optimizations and problems being addressed by researchers. They identified benchmark-specific optimization and constraints that can enable HLS tools to significantly improve performance while also highlighting the differences between optimizations needed for hardware designs as compared to software designs. Furthermore, they also present an analysis of academic HLS tools against commercial ones.

An overview of HLS techniques and tools has also been compiled by Coussy et al. [30] who outline the steps involved in an HLS design flow and compare it against an RTL design flow. A similar overview of a much larger set of tools was contributed by Meeus et al. [16]. They outline a number of challenges faced by HLS, such as application-specific tool flows, more optimization options for improving HLS designs, and design entry standardization asking for additional training for designers to adapt their applications for hardware platforms. Martin et al. [17] divide HLS tools into generations highlighting both their key features along with their shortcomings. According to the authors, the HLS of the early to late 2000’s establishes the $3^{rd}$ generation comprising tools like Xilinx’s AccelDSP, Synopsys Synplicity Synplify DSP among others. Although not documented, surely we are seeing a new generation of tools like Xilinx’s VivadoHLS and VitisHLS and Intel’s OpenCL-based HLS tool.

In addition to these contributions towards the analysis of HLS, specific algorithms have been realized in HLS and compared against RTL counterparts. Homsirikamil et al. [31] present such an analysis for various cryptography algorithms. They use Xilinx’s Vivado HLS and benchmark the algorithms on throughput and throughput/area metrics. Another relevant work is presented by Winterstein et al. [32]. They analyze HLS for two types of implementations of the K-means algorithm, a dataflow and a recursive tree implementation. They demonstrate that dataflow architectures described using HLS achieve near RTL performance but recursive architectures do not.

It is to be noted that FINN [14, 15], the base for the HLS vs. RTL comparison in this paper, in fact generates HLS dataflow architectures. This work performs an analysis on a large scale sweeping through the design space of a critical component of FINN networks. We implement the main compute unit of a neural network layer with varying design parameters, such as the input feature map size, kernel dimension, output feature map size, input and kernel word length, both from FINN’s native HLS output and from a drop-in RTL description. We determine the relationship between the chosen design parameters and achieved quality metrics including throughput, resource utilization, end-to-end delay, and tool execution time.

Xilinx provides an open-source framework called FINN [14, 15] to generate highly specialized accelerators on FPGAs leveraging reduced-precision datatypes and streaming dataflow (DF) architectures. FINN customizes the hardware architectures to the specifics of a DNN topology and the exact datatypes used. Each layer is instantiated with its designated compute units in hardware. On-chip data streams interconnect the compute units to form the desired network topology. The small and compact size of reduced-precision DNNs allows to store all parameters on the chip - off-chip memory access and with that potential memory bottlenecks are avoided. FINN produces synthesizable C++-based HLS code for describing the generated QNN accelerators. HLS was initially chosen for the hardware description of these highly customized architectures, as C++ templates can be used to parameterize canned designs for different datatypes, different layer types and different degrees of parallelism. Furthermore, it enables rapid implementation in comparison with handwritten RTL. In the following, we discuss the hardware architecture and the tool flow of FINN in separate sections.

In this section, we evaluate the RTL and HLS implementation alternatives of the MVU with respect to different performance metrics. In Section 6.2, we look at the resource utilization in terms of look-up table (LUT) and flip-flop (FF) counts and in terms of block RAM (BRAM) usage. We also report the total number of execution cycles, which are a measure of how long each implementation takes to process the same number of inputs. In Section 6.3, we analyze the critical path delay while in Section 6.4, we report the impact on synthesis time. Finally, in Section 6.5, we look at a complete application of network intrusion detection. It uses a multi-layer perceptron (MLP) network composed of four fully connected layers [41]. We start with describing the experimental setup.

High-level synthesis (HLS) has become a popular alternative to the design of digital systems on the register-transfer level (RTL) using hardware description languages (HDL). HLS provides flexibility to designers, especially to software engineers, and allows them to describe their designs in a high-level language like C++. It reduces the design time and requires less specialized training as compared to RTL design entry. Over the years, HLS has improved significantly, and the performance gap between HLS and RTL has reduced.

HLS has been used successfully to describe designs of various applications like high-performance computing and neural networks. The FINN framework is one such example. It generates DNN accelerators targeting FPGAs by producing synthesizable C++ HLS code. These accelerators are constructed as streaming dataflow (DF) architectures and leverage reduced-precision data types. The main compute units in FINN accelerators are instances of the matrix-vector unit (MVU). They multiply input feature vectors with the kernel weight matrices of a neural network layer. This work presented an alternative RTL description of this MVU and evaluated the attained design trade-offs in comparison to the baseline HLS implementation.

The RTL description is tailored to the requirements of the MVU and is designed to achieve an initiation interval (II) of one. By implementing an AXI-stream input/output interface, it is a drop-in replacement for the generated HLS cores. The design comparison, which was conducted for a Zynq-7000 FPGA target platform, analyzed the resource utilization, the critical path delay and the needed synthesis time. The resource utilization was further detailed into look-up table (LUT), flip-flop (FF) and block RAM (BRAM) usage.

The analysis was carried out by sweeping through a number of network and design parameters, namely, the number of input feature map (IFM) and output feature map (OFM) channels, the dimensions of the IFM and the kernel as well as the numbers of processing elements (PEs) and SIMD slices per PE. It was shown that HLS requires significantly more FPGA resources as compared to RTL for smaller designs due to a large base implementation to cater for I/O protocols, buffers and overall architecture. Increasing the design complexity, i.e. the number of PEs and SIMDs, lets the resource utilization figures of HLS and RTL designs, particularly the LUT usage, converge. However, HLS consistently consumes orders of magnitude more flip-flops and at least $2\times$ more BRAMs than the RTL design. This is caused by the HLS synthesis aggressively pipelining the generated RTL code as a proactive measure for reducing the risk of later timing violations. Finally, it was demonstrated that the resource consumption by the HLS design is particularly sensitive to the number of IFM channels. The increase of this parameter mandates larger input buffers, which the HLS design accesses through an unfavorably growing multiplexer structure.

The critical path of the MVU is in the control logic for RTL and in the SIMD elements or the adder tree for HLS for small number of PEs and SIMDs. As the design becomes larger, the critical path of the RTL designs is also in the SIMD elements or the adder tree and the delay is directly proportional to the number of PEs and SIMDs. This results in similar delay through the circuit as the number of IFM and OFM channels are changed while keeping the number of PEs and SIMDs constant. As the design becomes larger with an increase in the number of PEs and/or SIMDs, the critical path delay increases for both RTL and HLS designs. In all cases, however, The RTL produces faster designs, between $45\%$ – $80\%$, for all types of networks as compared to HLS.

HLS also suffers from significantly longer synthesis times compared to RTL. This was a limiting factor on the size of designs that can be synthesized. It took at least $10\times$ more time to synthesize an HLS design as compared to RTL. This has the potential to undo any gains achieved in the reduction of the original design time when the design space exploration of debugging demand additional complete design cycles.

To conclude, the RTL abstraction is an attractive alternative for code-generated hardware designs of frameworks such as FINN and hls4ml, given the gains achieved in synthesis time together with some potential resource benefits depending on the design size.

This material is based upon work supported, in part, by Science Foundation Ireland under Grant No. 13/RC/2094_P2 and, in part, by the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement and Grant No. 754489. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the Science Foundation Ireland and European Union’s Horizon 2020 programme.