CIMR-V: An End-to-End SRAM-based CIM Accelerator with RISC-V for AI Edge Device

Fig. 2 depicts the comprehensive architecture of CIMR-V, which is composed of a 512Kb SRAM-based CIM unit for energy-efficient computing, a modified RISC-V core for controlling the CIM and conducting high-precision computing, an instruction memory, a 256Kb feature map SRAM for layer fusion, and a 512Kb weight SRAM for weight fusion. CIMR-V is implemented on the PULPissimo platform, featuring the ibex 32-bit 2-stage RISC-V architecture [6]. In order to reduce routing complexity and power consumption, the CIM input buffer is designed with a 32-bit shift.

This design is based on the SRAM-based CIM [7], which supports two distinct modes:

1. 1.

   X-mode, which accommodates high input data with 1024 wordlines (WLs), 512 bitlines (BLs), and 256 sense amplifiers (SAs).

2. 2.

   Y-mode, which accommodates high output data with 512 WLs, 1024 BLs, and 512 SAs.

To minimize power consumption and quantization error, the output of 512 / 1024 MAC operations is sensed by the SA on the corresponding long BL, and the activation function (ReLU) is executed simultaneously. Additionally, to enhance the robustness and accuracy of CNNs, we apply the symmetry weight mapping method to mitigate nonlinearity (NL) and cell variation in binary or ternary weights.

Fig. 3 illustrates the modified RISC-V core, which incorporates not only the native Arithmetic Logic Unit (ALU) for basic arithmetic operations but also energy-efficient components, specifically the CIM read/write and CIM convolution modules.

To reduce pipeline overhead, we employ a two-stage CPU architecture. In the first stage, an instruction is fetched from memory using a prefetch buffer. In the second stage, the fetched instruction is decoded by the decoder unit, and the controller adjusts the program counter (PC) address and the control and status register (CSR) as needed. Simultaneously, the Load Store Unit (LSU) unit simultaneously loads data from data memory into the register file. Finally, in the execute block, the input data is computed and the output is stored in the register file.

To achieve optimal efficiency, all CIM instructions, such as the CIM convolution, write, and read instructions, are executed atomically within a single cycle, thereby minimizing latency and storage overhead. Additionally, to reduce feature map data movement, the CIM instructions utilize data from the feature map SRAM or weight SRAM instead of the register file.

Fig. 4 illustrates the energy-efficient CIM-type instruction (opcode 1111110) employed in CIMR-V, which includes three main instructions: CIM convolution, CIM read, and CIM write. To improve system throughput, the CIM convolution instruction is executed within a single cycle by loading FM SRAM data from address register 1 (rs1) with the immediate source offset ($imm_{s}$) into the CIM input buffer, performing 512/1024 Multiply-Accumulate (MAC) operations, and storing the resulting CIM output into FM SRAM at rs2 with the added offset $imm_{d}$.

Similarly, the CIM read instruction reads the CIM macro weights from rs1 + $imm_{s}$ and stores them in SRAM at rs2 + $imm_{d}$. Additionally, the CIM write instruction supports the writing of 32-bit data in a single operation, further enhancing writing performance. Overall, this instruction set contributes to the energy efficiency and high performance of CIMR-V.

The overall dataflow for the CIM convolution instruction is depicted in Fig. 5. This involves shifting the input feature map into the input buffer, flattening the CNN weights into macro BLs by output channel, and executing the convolution to obtain output data through SA.

To minimize power consumption and feature map data movement, the CIM convolution instruction supports CIM layer fusion, achieved by dividing the input feature map into blocks and storing overlapping data through the CIM input buffer to reduce layer fusion overhead. (Fig. 6) Furthermore, CIMR-V supports a pipeline with CIM convolution and max pooling to reduce latency and storage space. (Fig. 7)

In general, row-wise convolution dataflow is crucial for enhancing throughput by utilizing CIM layer fusion and convolution/maxpool pipeline technology. Moreover, the dataflow is highly scalable to row-wise 2-D convolution.

Fig. 8 depicts the proposed weight fusion method for CIM, which significantly reduces the weight loading latency from DRAM. This is achieved by pipelining the CIM convolution and weight loading into the weight SRAM, resulting in faster weight data access.

Furthermore, to minimize CPU loading, we employ uDAM for the parallel loading of weight data. uDAM is a technique that allows for direct data transfer between devices without the intervention of the CPU. This approach enables efficient weight data transfer and reduces CPU overhead.

Finally, the updated weight data from the weight SRAM is written into the CIM macro using the CIM write instruction. The proposed method offers a more efficient and streamlined weight loading process for CIM convolution, enhancing the overall performance of the system. (Fig. 9)

CIMR-V offers a full stack flow for model deployment, distinguishing it from conventional AI accelerators. The workflow involves converting the model’s Python code, trained on TensorFlow or PyTorch, to C/C++ for deployment on CIMR-V using the GCC compiler and RISC-V architecture.

Furthermore, the full stack flow allows for seamless and efficient model deployment on CIMR-V, where CIM-type instructions effectively execute computation. This feature enables CIMR-V to deliver high performance with minimal latency, making it an excellent choice for AI workloads.

Fig. 10 presents the CIMR-V end-to-end flow for model inference through the RISC-V mode, the CIM mode, and the weight fusion mode. The pre-processing and post-processing steps that need high precision will be executed on RISC-V. The convolutional and pooling layers are executed through the CIM mode but with pipeline to reduce latency. Furthermore, the full model can be executed on chip without feature map I/O to DRAM through layer fusion processing. The corresponding weight update can be directly loaded from internal weight SRAM through weight fusion to avoid DRAM latency.

To showcase its efficacy, we adopt the keyword spotting task as an illustrative example. The architecture of the keyword spotting model, as illustrated in Fig. II, comprises pre-processing, 1-D binary convolution, max pooling, and post-processing. The model achieves an accuracy of 94.02% in the Google Speech Commands Dataset (GSCD)[10] with 12 classes.

Simulated latency optimization has included DRAM access latency based on DDR4 DRAM [11]. By means of layer fusion, the latency of the convolution execution can be effectively reduced by 33.16% due to the decreased movement of the feature map between the DRAM and the chip. Moreover, through weight fusion, an additional 62.94% of latency can be saved by minimizing weight transfer between the DRAM and the chip. Subsequently, by employing a pipeline for convolution and max pooling operations, another 40.00% of latency can be reduced by mitigating idle cycles of CIM macros. In summary, the proposed optimizations result in a significant 85.14% reduction in the end-to-end inference latency.

The proposed digital design has been implemented with Verilog and synthesized with the Synopsys Design Compiler. Through the integration of the proposed design with the CIM macro, CIMR-V achieves 26.21 TOPS and 3707.84 TOPS/W with the TSMC 28nm CMOS process, when operating at a clock rate of 50 MHz.

Table I presents a summary of performance metrics and a comparative analysis with other SRAM-based CIM designs. In  [4], a dedicated dataflow is proposed for specific algorithms, which cannot be reconfigured for alternative algorithms. Both  [9] and  [5] did not consider weight fusion, resulting in increased latency in transferring weights between DRAM and chip. Although  [5] adopted multiple small CIM macros, which reduced energy efficiency due to overhead from peripheral circuits, the proposed design achieves higher energy efficiency and supports end-to-end model inference compared to other designs.