Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis

Based on the modality of inputs and outputs, language models for software engineering can be categorized into several downstream code tasks (Zhang et al., 2023) such as text-to-code (code generation/synthesis (Ling et al., 2016)), code-to-text (code summarization (Iyer et al., 2016)), and code-to-pattern processing (defect detection (Ray et al., 2016)). This paper focuses on code generation that aims at producing code from natural language descriptions/prompts. To facilitate the development of code generation with language models, various datasets, approaches, and pre-trained models have been introduced over the past decade.

Due to the lack of model capability, the early-stage methods (Ling et al., 2016; Iyer et al., 2018) of code generation mainly focus on a few programming languages such as Python or Java. Subsequently, with the increasing computational power, larger datasets are introduced to train models for multiple general-purpose programming languages. CodeXGLUE (Lu et al., 2021) presents a comprehensive code dataset consisting of different code tasks such as clone detection and code repair. HumanEval (Chen et al., 2021) dataset together with the code model CodeX marks as a milestone by using pre-trained LLMs for code generation. The promising capability shown by CodeX sparks significant academic and industrial interests in developing LLM-assisted code generation. Larger code datasets, such as StarCoder (Li et al., 2023a) and CodeParrot (Tunstall et al., 2022), and LLMs, including Code-LLaMA (Roziere et al., 2023) and CodeFuse (Di et al., 2023), are open-sourced in this community. However, most of these recent efforts focus on software programming languages.

Building on the success of LLM-assisted software programming, recent studies have explored using language models for automated hardware generation. Since the data is the key to training LLMs for hardware code generation, multiple HDL datasets have been introduced recently. Thakur et al. present Verigen (Thakur et al., 2023) dataset that contains $17$ hardware designs with $0.3$K lines of HDL code. To increase the diversity of hardware designs for training and evaluation, Lu et al. open-source a larger benchmark consisting of $30$ designs with more than $2.5$K lines. Sourced from HDLBits^‡‡‡https://hdlbits.01xz.net/wiki/Main_Page, Verilogeval (Liu et al., 2023a) introduces larger datasets with $156$ problems. These open-sourced datasets provide public benchmarks for text-to-HDL generation.

The evaluation metrics of LLM-assisted hardware code generation focus on three aspects: i) syntax, ii) functionality, and iii) quality. In previous literature (Chen et al., 2021; Liu et al., 2023a), syntax correctness and functionality are measured using pass$@k$ metric which represents whether any of $k$ generated code samples can pass the syntax check of synthesis tools or functional unit tests. The quality usually is defined as power, performance, and area of the generated hardware, collectively reflect the capability of the code generation methods.

Aiming at improving these metrics, existing approaches (Thakur et al., 2023; Lu et al., 2024; Liu et al., 2023a) fine-tune pre-trained LLMs on the downstream task with optimized sampling schemes. These LLMs are mainly pre-trained using software programming languages. RTLFixer (Tsai et al., 2023) introduces an automated framework that adopts retrieval-augmented generation (Lewis et al., 2020) and ReAct prompting (Yao et al., 2022) to enable LLM-assisted debugging for RTL designs. LLM-VeriPPA (Authors, 2024) enhances the code generation of RTL using a two-stage refinement process to progressively improve syntax, functionality, and hardware performance. However, these approaches focus on low-level hardware languages instead of HLS. In this work, we take the first step to investigate the HLS code generation with LLM. Since HLS shares similar semantics and syntax with programming languages commonly used during LLM pre-training, this paper explores whether HLS is better than low-level hardware languages for automated hardware design generation. Although code generation with feedback and CoT prompting is not new in the literature of coding language models, our experimental results, insights, benchmark, and evaluation infrastructure specific to LLM-assisted HLS design offer valuable contributions to the future development of automated hardware generation.

Following the practices of Python benchmark HumanEval (Chen et al., 2021) and Verilog dataset VerilogEval (Liu et al., 2023a), each design point has three components: i) user instruction prompts, ii) design descriptions and iii) reference designs. Figure 3 shows the standardized format template, with each data point stored as a JSONL following the Alpaca format. The default user instruction prompt is Generate HLS code with the following instructions:, which can be enhanced using the chain-of-thought (COT) prompting technique as detailed in  Section 4.2.

We collect $52$ HLS-based designs from open-source repositories, including HLSyn (Bai et al., 2023)^§§§https://github.com/UCLA-DM/HLSyn and ML4Accel^¶¶¶https://github.com/UT-LCA/ML4Accel-Dataset. These designs are split into training and testing sets at a 4:1 ratio and fall into five categories:

• •

  Matrix and Linear Algebra Operations: Includes sparse matrix-vector multiplications, dense matrix-matrix multiplication, array transformation and stencil computations.

• •

  Scientific Simulations: Methods for solving physical and mathematical problems such as heat distribution and electromagnetic simulations.

• •

  Statistical Computations: Calculations of statistical metrics from datasets.

• •

  Iterative Methods: Techniques for solving equations using iterative approaches.

• •

  Other Computational Kernels: Specialized computational tasks like molecular dynamics and interactions, encryption algorithms, and optical flow.

Each of these $52$ designs is associated with different combinations of programs such as PIPELINE, PARALLEL and TILE. We filter out the HLS programs that are invalid, resulting in a collection of over $42,000$ HLS programs. The whole dataset is split into training and test sets for supervised fine-tuning and evaluation, respectively.

Given that the dataset encompasses over 42,000 HLS programs, manually generating design descriptions for each program is both labor-intensive and time-consuming. Inspired by both HumanEval (Chen et al., 2021) and Verilog, we utilize ChatGPT (version $3.5$ and $4$) to automate the creation of design descriptions for the datasets. We append each HLS program with this base prompt when utilizing ChatGPT to generate the corresponding design descriptions. Both the reference design and its generated description are stored in JSON format, adhering to the structure outlined in Section 3.1. This method ensures streamlined and consistent documentation of design descriptions across the dataset. Following the practice of (Liu et al., 2023a), we provide two versions of prompts for each HLS program in the test set: MachineGen and HumanRefine. The MachineGen version comprises prompts and instructions generated by GPT-based models without human modifications. In contrast, the HumanRefine includes manually refined prompts to ensure more concise and human-like instructions.

We provide evaluation infrastructure for both syntax and functionality. For syntax verification, we use the GCC compiler with the ”-fsyntax-only” option. It allows us to verify the syntax without the overhead of compiling the code, thereby enhancing time and space efficiency. Regarding functionality correctness, we design unit tests tailored for each example in the test dataset. Each test case is associated with its corresponding ‘source_file‘. To achieve this goal, we modified the original test JSONL file to add another attribute ‘source_file‘. The testing process involves executing both the generated code and the original source code to compare their outputs. For instance, if the outputs from both codes consist of matrices, we conduct a targeted comparison. This is done by selecting specific positions within the matrices from both outputs at random and verifying if they match.

An overview of our proposed framework is depicted in Figure 4. The framework comprises two main stages: i) model fine-tuning and ii) iterative code generation. The final output is an HLS-based program that can be synthesized into the corresponding hardware design. While focused on Vivado-HLS, our framework is adaptable to any HLS language with appropriate datasets.

In the model fine-tuning stage, our framework initiates by retrieving coding LLMs from open-source repositories, such as Code-Llama^∥∥∥https://huggingface.co/codellama and Start-Coder^******https://huggingface.co/blog/starcoder. Then, supervised fine-tuning is conducted on these pre-trained models using the HLS training data (Section 3.2). We adopt axolotl^††††††https://github.com/OpenAccess-AI-Collective/axolotl to perform the fine-tuning, allowing for customization of models and training parameters, such as learning rate, batch size, and epoch number to fit specific scenarios and available resources.

In the second stage, we employ the fine-tuned LLM for iterative code generation. The process begins with initial inputs consisting of user instruction prompts and design descriptions. To enhance the quality of the generated HLS designs, we incorporate a chain-of-thought optimization technique (Section 4.2) into the instruction prompts. The code generation then proceeds with a feedback loop (Section 4.3) that iteratively improves the correctness of the HLS designs. This iterative process continues until a refined HLS program is generated as the final output. Users can specify the number of iterations, providing the flexibility to navigate this trade-off according to their specific needs, with more iterations typically yielding higher quality at increased runtime expense.

Previous studies indicate that the quality of generated content is significantly influenced by the instructional prompt (Zhao et al., 2021). The Chain-of-Thought (CoT) (Wei et al., 2022) technique has proven simple and effective for enhancing the performance of LLMs across a wide range of tasks, including arithmetic, commonsense, and symbolic reasoning (Wei et al., 2022). Although initially, COT yielded a modest $0.82$ point increase in the pass@1 metric in code generation, this improvement was substantially augmented through structured prompting (Li et al., 2023b). In this paper, we investigate the effect of CoT in generating HLS-based hardware designs.

Figure 5 illustrates the CoT prompt structured for HLS code generation. The prompt guides a systematic approach through several targeted steps: understanding FPGA characteristics, defining program structure, developing logic, selecting data types and interfaces, before finalizing the code. This structured process ensures thorough consideration of each key aspect to optimize the HLS code generation.

Code generation with feedback loop has shown promising results in previous literature (Liu et al., 2023b; Shojaee et al., 2023; Wang et al., 2022; Authors, 2024). This paper investigates its impact on HLS code generation using a two-step feedback loop tailored for automated hardware generation, focusing on HLS-related feedback. At each iteration, the framework evaluates the syntax and functional correctness of the generated HLS program. The located syntax error and functional defects are then fed back into the input prompts as additional information for subsequent code generation.

In the first step, syntax feedback is provided using the GCC compiler with the ‘-fsyntax-only‘ option, as specified in 3.4. It captures the syntax errors without fully compiling the code. It allows rapid identification including the types and locations, and precise error mapping for targeted corrections. If the syntax check passes, our framework proceeds to the second step by executing predefined unit tests to compare the outputs of the generated and the original source code. Functional defects are recorded and added to the prompts for the next iteration. This two-step feedback loop continues for user-specified iterations, providing flexibility to balance the trade-off between design quality and runtime cost.

In our evaluation, we adopt Code-Llama-7B as the pre-trained model for fine-tuning, employing the low-rank-adaption (QLoRA) (Hu et al., 2021; Dettmers et al., 2024) technique for faster training and lower memory consumption. Key configurations include loading the model in 8-bit, a sequence length of 4096, sample packing, and padding to sequence length. We set the warmup steps to 100, with a gradient accumulation of 4 steps, a micro-batch size of 4, and an inference batch size of 2. For both syntax and functionality checks, we measure pass@3 accuracy as metrics. In the ablation study from Section 5.2 to Section 5.6, we adopt MachineGen for evaluation.

Experiments are conducted on a server with four NVIDIA L20 GPUs (48 GB each), an 80 vCPU Intel® Xeon® Platinum 8457C, and 100GB of RAM. This setup ensures sufficient computational power and memory to handle the intensive demands of fine-tuning and inference efficiently, especially for long data sequences in the feedback loop experiment.

Our first ablation study investigates the effect of the model fine-tuning. We evaluated the performance based on both syntax and functionality checks. As shown in Figure 6(a), the results demonstrate that the finetuning dramatically increases syntax correctness from $54.85$% to $88.44$%. More importantly, the impact of finetuning is even more pronounced in the functionality evaluation, where the non-finetuned model failed to achieve any correct functionality test, but the accuracy is improved to $53.20$% in the finetuned model. These enhancements highlight the critical role of finetuning in producing not only syntactically correct but also functionally viable codes, which demonstrates the benefits of finetuning LLMs for hardware design in the HLS code generation task.

To assess the effect of the chain-of-thought (CoT) technique, we perform both syntax and functionality evaluation on the fine-tuned model with and without the use of CoT. As indicated in Figure 6(b), incorporating CoT leads to a noticeable improvement in both metrics. Specifically, syntax correctness increases from $88.44$% to $94.33$%, and functionality score rises from $53.20$% to $61.45$%. The result demonstrates the effectiveness of CoT in enhancing the reasoning capability, thereby improving its overall performance.

Our two-step feedback loop provides both syntax and functionality feedback. We evaluate the impact of these feedback loops with different numbers of iterations, ranging from 0 to 2.The results, shown in Figure  Figure 7 and  Figure 8, indicate that both syntax and functionality feedback loops significantly improve model performance, especially when combined with COT prompting. The initial feedback loop yields substantial accuracy improvements in both syntax correctness and functionality evaluation, though the second loop shows diminishing returns.Syntax feedback loops enhance both syntax correctness and functionality performance, suggesting that iterative refinement is particularly effective for complex tasks. Similarly, functionality feedback loops not only improve functionality checks but also boost syntax accuracy, indicating that enhancements in functional understanding contribute to better syntactic performance.

Figure 9 shows the time cost for generating 120 data entries under different conditions, measuring the impact of CoT and feedback loops. Without a feedback loop, CoT significantly reduces the time. Adding a syntax feedback loop increases the time, but CoT continues to notably decrease the duration. The functionality feedback loop is the most time-consuming, though CoT still provides a notable reduction, albeit less dramatic. This demonstrates CoT’s effectiveness in reducing operational times across varying complexities.

For the test set, we evaluate the latency and resource consumption of the generated HLS designs using a Xilinx VCU118 as our target FPGA, with a clock frequency of $200$MHz and Xilinx Vivado 2020.1 for synthesis. As shown in Table 1, all HLS designs demonstrate reasonable performance, with BRAM usage consistently remained at zero due to the design scale.

We analyze the effects of code complexity on the performance of fine-tuning our language model with CoT prompting and tested without the use of any feedback loops during inference. We categorize MachineGen into three classes according to their code complexity: easy, medium, and difficult. The results shown in the Table 2 indicates a clear trend: as the complexity of the generated code increases, both syntax and functionality correctness rates decline. This outcome could be attributed to several factors. First, more complex code inherently presents more challenges in maintaining syntactic integrity and functional accuracy. Second, the absence of feedback loops in the inference phase may have limited the model’s ability to self-correct emerging errors in more complicated code generations.

As shown in Table 3, this section compares the performance of our model on MachineGen and HumanRefine test sets. Our findings reveal that the performance on the HumanRefine is significantly lower than on the MachineGen. This disparity suggests that the model is more adept at handling machine-generated prompts. The primary reasons for this are: the model’s training data bias towards machine-generated prompts, the increased complexity and nuanced nature of human-generated prompts, and the conciseness and clarity of human-generated prompts that often omit repetitive or explicit details found in machine-generated prompts, making it harder for the model to generate syntactically and functionally correct code.