A Cycle-Accurate Soft Error Vulnerability Analysis Framework for FPGA-based Designs

An FPGA usually contains a set of configurable logic blocks (CLBs) and programmable interconnect resources, making it possible to implement flexible digital systems in many areas such as aerospace, automotive, machine learning, and datacenters [1]. In order to have high density and fast configuration speed, the majority of FPGAs use SRAM (Static RAM) technology to store the configuration bitstream. However, SRAM-based FPGAs are vulnerable to radiation induced soft errors [2, 3]. Soft errors may occur with a particle strike to a sensitive region of a transistor in a SRAM cell resulting in an inversion of the logical value, which is called bit-flip. The bit-flips occurring in the FPGA configuration memory remain unchanged during the following cycles. Such bit-flips are permanent and can cause failure in an FPGA-based design. Therefore, it is very important to analyze such bit-flips in FPGA-based designs and provide corresponding protection and/or correction, especially for highly-reliable systems used in aerospace and automotive.

Modern FPGAs often use error correction methods, such as scrubbing, to correct configuration memory bit-flips [4, 1]. As shown in Fig. 1, memory bit-flips are usually corrected using coding mechanisms, or golden models, together with partial reconfiguration [5, 6]. As a result, single event upsets (SEUs) need not be permanent and their consequences can be classified as a transient fault that lasts until a scrubbing occurs. Moreover, if a bit-flip effect is further masked in the circuit design, this error will not be propagated to the circuit outputs. And thus the effect of that bit-flip will be completely transparent to the system. Therefore, to implement a highly-reliable design that still achieves high performance, it is essential to precisely analyze which parts of the FPGA-based design are most vulnerable to soft errors (bit-flips) and thus provide lightweight error protection and/or correction.

As presented in Section II, there are many methods developed to analyze the reliability of FPGA-based designs [9-48]. However, there is still the need for a fast and accurate toolflow, that is open source, to perform early-stage analysis of soft error vulnerability for FPGA-based applications in the design phase.

In this paper, we present a framework that enables fast and cycle-accurate soft error vulnerability analysis of an FPGA-based design using the post-synthesis netlist as input. First, given an initial FPGA design or partial design in the early design stage, our framework extracts the post-synthesis netlist. Based on the extracted netlist, it automatically inserts the bit-flip configuration faults. Then it seamlessly feeds the golden copy and fault copies of the netlist into the open source simulator Verilator [7] to perform cycle-accurate simulation. According to the simulation results, it generates a histogram of vulnerability scores for each component in the original design. To demonstrate the usefulness of our framework, we use it to analyze the reliability of the widely used ITC’99 benchmarks [8] with injected bit-flip faults and show which components are more vulnerable to bit-flip errors. Compared to the Xilinx Vivado fault simulation, our framework runs up to 53x faster while achieving cycle-level accuracy.

To analyze the vulnerability of soft errors for an FPGA-based design, a common technique (called fault injection) is to intentionally inject faults to disturb some parts of the design and analyze the possibility of changes of its normal behavior [9]. There are many different means to insert artificial faults into an FPGA-based design, and we classify them as follows.

1. 1.

   Using radiation techniques to induce upsets in actual physical FPGA design [2, 10, 11]: These methods are fast and realistic. But they need a prototype version of the design and cannot be applied in early stages of the design flow. The lack of controllability of fault locations is another weakness. In addition, these methods destroy the equipment and are quite expensive.

2. 2.

   Modifying configuration memory to force upsets [12, 13, 14, 15, 16, 17]: Some studies try to modify an FPGA’s configuration bitstream file directly, where others modify an FPGA’s configuration memory at run-time using dynamic partial reconfiguration. According to the different configuration ports, these methods can be divided into two categories: external and internal fault injections. Note that they need the prototype version of the design.

3. 3.

   Emulating faults in FPGAs [18, 19, 20, 21, 22]: These studies add some extra hardware (saboteurs) to the original design and run the modified version on the board to emulate the behaviour of bit-flips. Existing techniques in this subject manipulate the design’s description in RTL (Register-Transfer Level) or post-implementation netlist. These methods need complex internal or external components for controlling the fault injection and inference processes. Therefore, they either need dedicated external hardware platforms or impose considerable area overhead. In addition, they need the prototype version of the design.

4. 4.

   Analytical modelling that applies probabilistic and/or statistical analysis to model the behavior of fault masking in an FPGA design [23, 24, 25, 26, 27]: These methods often use worst case analysis. Hence they usually lead to inaccurate results.

5. 5.

   Fault simulation that uses a software simulator to simulate the behaviour of memory upsets for a given design descried in hardware description languages (HDLs) [28, 29, 30, 31, 19, 32, 33, 34, 35, 36, 37, 38]: These methods use either code modification to add mutants and saboteurs or simulator commands to change variable and signal values. Fault simulation offers important advantages over other techniques. They have both high observability and controllability of the faults, and can be applied soon at various level of design abstractions. Hence they are very useful for reliability aware CAD (computer-aided design) tool development. However, fault simulation methods suffer from long run-time.

We summarize the fault injection techniques in Table I and compare their controllability, run-time, accuracy, flexibility, design flow stage, and cost. Considering the pros and cons of fault injection methods, fault simulation is a good choice to analyze the reliability of an FPGA design accurately at early design stages. On one hand, high-level fault simulation suffers from high inaccuracy compared to low-level fault simulation [33]. On the other hand, low-level fault simulation tools [38, 39] use event-driven simulation, and thus, suffer from long execution time and large memory consumption.

In this paper, we explore a cycle-accurate fault simulation framework to enable accurate yet fast soft-error vulnerability analysis for FPGA-based designs. We decide to explore the fault simulation at the post-synthesis netlist level, which can be considered precise because it provides accurate fault injection locations in an already optimized netlist.

The overall structure of our proposed framework is presented in Figure 2. Given an input FPGA circuit described in VHDL or Verilog, it can automatically extract the post-synthesis netlist and insert bit-flips at the basic element (BEL) level, and seamlessly feed the golden and fault copies to the open source simulator Verilator [7] for fast and cycle-accurate simulation. Finally, it will generate a histogram of vulnerability scores for the input circuit. Next, we present more details of each component of our framework.

To demonstrate the performance gain of cycle-accurate simulation using Verilator in our framework, we compare it to the Xilinx Vivado ISIM simulator. Both simulators are evaluated with the widely used ITC’99 benchmark subset circuits [8].

Both simulators are parallelized using the make command with the -j option and use 12 cores. Ideally, the each circuit should be simulated with all possible combinations of inputs. But as this number grows exponentially with the product of the number of input bits and the number of intended simulation cycles, we decided to use 1,024 pseudo-random input combinations due to the long time run. It already takes several hours to run the 1,024 input stimulus for the biggest circuit. Table II presents the simulation performance comparison for the Verilator and Vivado simulators. As shown in column 5, the cycle-accurate Verilator simulator is from 9.7 up to 53.8 times faster than the Vivado ISIM simulator. This is because the Verilator simulator compiles the Verilog code into a much faster optimized model and does not model intra-cycle events.

To demonstrate the usefulness of our framework, we conduct two case studies.

The proposed fault simulation is not limited to small circuits, but it can be challenging to perform the fault simulation on large designs due to the computational complexity involved. Large FPGA designs can have thousands or even millions of BLEs, making it impractical to simulate every possible fault (even random faults) in the circuit.

To address this issue, partitioning techniques can be used to divide the large design into smaller, more manageable parts that can be simulated individually via the proposed accurate method. The design can be partitioned based on various criteria, such as functional blocks, critical parts, or input/output interfaces. Each partition can then be simulated separately using the proposed cycle-accurate fault simulation techniques, and the results can be combined to obtain the overall reliability of the design. In addition, it can also help identify potential problems in specific parts (i.e. critical parts) of a large design, allowing for targeted optimization and testing.

In this paper, we have presented a framework that enables fast and cycle-accurate analysis of soft error vulnerability for FPGA-based circuits in the early design stage. The fault model is at the basic element (BEL) level in an already optimized post-synthesis netlist. In our framework, a post synthesis netlist is first extracted for a given FPGA design or partial design. Then, bit-flip errors are inserted at BEL level. Both the golden copy and faulty copies run through the cycle-accurate simulator Verilator to identify the effects of faulty LUTs. By applying user-defined weights to each output bit, a histogram of vulnerability scores will be generated to guide designers to identify and rank the more vulnerable LUTs in the design. Two case studies have been conducted on the widely used ITC’99 benchmark circuits to demonstrate the usefulness of our framework: one on cycle-level error possibility analysis for each LUT output, and the other on histogram of vulnerability scores of all LUTs. In general, our framework is up to 53x faster than the Xilinx Vivado simulation, while achieving cycle-level accuracy.