Extending the Frontier of ChatGPT: Code Generation and Debugging

In order to ascertain the practical applicability of ChatGPT in addressing coding problems, a dataset was meticulously curated comprising coding challenges sourced from Leetcode, a renowned platform known for its extensive collection of coding problems. Leetcode provides users with a comprehensive array of coding questions, complete with visualizations and an integrated development environment (IDE) that facilitates coding solutions. The dataset construction process commenced with the extraction of coding questions from diverse domains that represent the most prominent areas of coding expertise, including but not limited to Hash-tables, Divide and Conquer, Greedy approach, and Graph. To ensure a comprehensive evaluation, the incorporation of coding challenges of varying difficulty levels, ranging from rudimentary problems to advanced and intricate ones was ensured. Figures 1 and 2 provide a visual depiction of the domains of problems covered in the dataset along with the difficulty variation. It is imperative to acknowledge that the data collection occurred prior to May 2023, and the outcomes presented in this study are contingent upon the specific dataset utilized. It is essential to recognize that these results may be subject to modifications if more recent data is gathered.

The dataset construction process meticulously considered the need for diversity across various problem domains, while also accounting for the difficulty levels of the challenges and their solution acceptance rates on the Leetcode platform. The resulting dataset comprises carefully chosen problems that reflect a wide range of genres, ensuring comprehensive coverage of the coding landscape. Table I presents a detailed overview of the dataset, presenting the genres of problems alongside the number of problems from each difficulty level and their respective solution acceptance rates. The dataset was thoughtfully crafted to include precisely 15 problems from each problem category. Within each difficulty level (Easy, Medium, and Hard), we ensured the inclusion of five problems, following a specific acceptance rate criterion. Two problems were selected with an acceptance rate below 30%, one problem with an acceptance rate between 30% and 70%, and the remaining two problems with an acceptance rate exceeding 70%. It is essential to acknowledge that due to the scarcity of problems meeting these specified criteria in certain categories, the total number of problems in the dataset amounts to 128. Nevertheless, the careful selection process and adherence to acceptance rate distributions ensure a balanced representation of difficulty levels and problem acceptance rates within the dataset which can be noticed in the figures 1 and 2.

After constructing the dataset, each individual problem, along with its corresponding code structure available in Leetcode, was presented as input to the ChatGPT model. The objective was to leverage the power of the large language model (LLM) to generate code solutions based on the provided code structure. The ChatGPT model, prompted with the problem description and code structure, responded by generating a code solution, which was then utilized as the suggested solution within Leetcode’s integrated development environment (IDE).
The generated solution provided by ChatGPT after running in the IDE of Leetcode is submitted for evaluation which results in one of the following:

1. 1.

   Upon successful execution of the generated code solution, the Leetcode platform displays vital performance metrics and the solution’s comparative efficiency in terms of outperforming other submitted solutions. In that case, the following steps are taken:

   1. a.

      The solution provided by ChatGPT is listed as a “Passed Instance”.

   2. b.

      The runtime of the generated solution in millisecond (ms) and the memory consumption of the executed solution in megabytes (MB) is noted.

   3. c.

      The percentage of other submitted solutions for that problem in Leetcode that this solution beats in runtime is noted.

   4. d.

      The percentage of other submitted solutions for that problem in Leetcode that this solution beats in memory consumption is noted.

2. 2.

   In the event that the generated solution is not accepted by the Leetcode platform, it can be attributed to one of the following scenarios:

   1. a.

      The occurrence of a runtime error (RTE), indicating that the program encountered an error during the execution of the provided test cases.

   2. b.

      A time limit exceeded error (TLE), signifying that the program surpassed the allotted execution time designated by Leetcode.

   3. c.

      A memory limit exceed error (MLE), denoting that the program surpassed the allocated memory usage threshold specified by Leetcode.

   4. d.

      Failure to pass all the test cases provided by Leetcode, indicating that the solution does not attain complete correctness and accuracy.

3. 3.

   In the event of a failed solution, the error messages generated by the Leetcode platform or the failed test cases are utilized as feedback to the ChatGPT model. The model is then prompted to rectify and enhance the provided solution, effectively evaluating the model’s debugging capabilities. The resulting modified solution is subsequently re-submitted to the Leetcode IDE for evaluation.

   1. a.

      If the modified solution successfully passes all the test cases, the process proceeds to Step 1 for further analysis and assessment.

   2. b.

      Conversely, if the modified solution fails to meet the desired requirements and triggers any of the error messages encountered in Step 2, the corresponding problem is deemed as a failed attempt by ChatGPT.

ChatGPT exhibited an overall success rate of 71.875% across the entire dataset. This indicates that out of 128 problems, ChatGPT successfully generated solutions for 92 of them. Notably, among these successful cases, 84 problems were solved in the initial attempt, rest 8 were solved after prompting the ChatGPT to debug the previously provided solution with the feedback received from Leetcode.
There were 36 problems within the dataset for which ChatGPT did not produce satisfactory solutions, even after revisiting the problems with feedback from Leetcode.

Out of the 36 problems for which ChatGPT initially failed to produce any correct solution, the Leetcode platform provided error messages and feedback as guidance for improvement. This feedback, accompanied by a few of the test cases that the solution failed to pass, was utilized to prompt ChatGPT to rectify its errors and generate corrected solutions. Surprisingly, despite the availability of this feedback, ChatGPT failed to produce correct solutions in the majority of cases.
In fact, when ChatGPT attempted to fix its errors using the provided feedback, the new solutions exhibited a downgrade in performance. It can be seen from Figure 3 that approximately 63% of the time, these revised solutions failed to pass test cases that were previously passed, indicating a decrease in solution correctness. Only around 36% of the instances did the new solutions perform better, passing more test cases than before but yet not completely accurate.
The inability of ChatGPT to generate correct solutions even after receiving feedback highlights its limitations in effectively incorporating debugging information from Leetcode. This indicates a weakness in ChatGPT’s current debugging capabilities, impeding its ability to learn from and rectify errors based on provided feedback. Consequently, the model’s performance in improving solution correctness falls short, as demonstrated by the downgrade in performance observed in the revised solutions.

Analyzing the success rate of ChatGPT at solving problems across different domains reveals intriguing findings, as depicted in Figure 4. The model showcases the highest success rates on problems stemming from “Tree” and “Divide and Conquer” domains. On the contrary, the model showed subpar performance while solving problems from “Greedy” and “Dynamic Programming (DP)” domains.
This observation suggests that ChatGPT has a higher success rate at solving problems that adhere to well-defined rules and structured patterns as they are easier for the model to understand and generate solutions for. Therefore, intuitively, the model finds it more challenging to solve problems that require deeper analyses and do not conform to specific rules for generating solutions that eventually lead to lower success rates. For instance, in a tree problem, it is possible for the model to easily identify the root node and then recursively solve the problem for each child node. Similarly, in a divide and conquer problem, the model can easily divide the problem into smaller subproblems, solve each subproblem, and then combine the solutions to solve the original problem.
In contrast, the fundamental nature of Greedy and Dynamic Programming is much different. Greedy problems start with local optimal decisions with the goal to reach an optimal global decision via the steps it takes. Similarly, in a Dynamic Programming problem, the solution comprises a sequence of decisions where it breaks the original problem into overlapping subproblems and then reuses the subproblems’ solutions to solve the original problem. Such categories of problems require advanced reasoning and exploration of a wide-ranging set of possibilities that are difficult for ChatGPT to accommodate during the generation of solutions, eventually resulting in a lower comparative success rate. These findings suggest that ChatGPT excels in generating solutions for problems that follow a well-structured methodology or adhere to established patterns. However, the model faces challenges in generating solutions for problems that require more complex decision-making processes and the analysis of various test cases. In such cases, ChatGPT’s performance falls short, indicating the limitations of the model in handling problems that demand nuanced reasoning and consideration of diverse scenarios.

A similar problem trend can be observed from Figure 6 and Figure 5, which provide insights into ChatGPT’s success rate in solving problems of varying difficulties and acceptance rates. Notably, the model encountered challenges when tackling problems labeled as “Hard”, achieving a success rate of 55% in generating accurate solutions. Conversely, when confronted with problems labeled as “Easy”, ChatGPT demonstrated a notably high success rate of 90%. This observation reveals a downward trend in the percentage of successfully solved problems as the difficulty level increases. This phenomenon can be attributed to the fact that “Easy” problems often involve well-established techniques and require less analytical prowess, while more challenging problems necessitate deeper analysis and novel problem-solving approaches, which may exceed ChatGPT’s current capabilities.
Additionally, ChatGPT’s performance is influenced by the acceptance rate of the problems. The acceptance rate reflects the success rate of all submitted solutions for a particular problem on the Leetcode platform. It is noteworthy that the dataset used in this study does not encompass problems with acceptance rates below 10% or above 90%. When analyzing problems with lower acceptance rates, particularly below 30%, ChatGPT struggled to produce accurate solutions, resulting in success rates ranging from 30% to 40%. Conversely, as the acceptance rate increased, ChatGPT exhibited higher success rates, achieving an impressive 95.65% success rate when the acceptance rate exceeded 80%.
These observations that have been made from our experiments indicate that ChatGPT thrives in generating solutions for problems that conform to well-structured methodologies or established patterns, like Tree and Divide and Conquer problems. However, when confronted with challenges on domains whose solutions are not structured or do not have a blatant pattern for approaching the problem and require intricate test case analysis and consideration of input correlations, such as Dynamic Programming and Greedy problems, the model encounters difficulties in generating successful solutions.

Let us shift our focus from the broader dataset to a more specific subset comprising instances where ChatGPT successfully solved problems. Figure 7 provides a visual representation of the runtime distribution of these successful instances, categorized by the difficulty level of the problems and the acceptance rate of the solutions. The runtime ranking metric indicates the percentage of submitted solutions on Leetcode that ChatGPT outperformed in terms of runtime efficiency. Similarly, Figure 8 illustrates the percentage of submitted solutions on Leetcode that ChatGPT’s solution surpassed in terms of memory efficiency. These analyses offer valuable insights into ChatGPT’s ability to optimize runtime and memory utilization while generating solutions for the presented problems.
Upon analyzing Figure 7 and 8, several notable trends emerge, providing insightful observations regarding the efficiency of ChatGPT in terms of solution run-time and memory efficiency. Firstly, there is a discernible correlation between the difficulty level of “Easy” problems and their higher acceptance rates, resulting in higher efficiency in terms of run-time and memory usage. This means that ChatGPT was able to develop solutions that are efficient in terms of running time and memory utilization for these problems.
However, no distinct trend is apparent when examining problems classified as “Medium” difficulty level. This suggests that the relationship between problem difficulty and run-time or memory efficiency may not be as straightforward in this situation.
Remarkably, an intriguing phenomenon is observed among problems categorized as “Hard” with lower acceptance rates. Contrary to initial expectations, these problems exhibit higher efficiency in terms of run-time as well as memory consumption for ChatGPT’s solutions. These findings run counter to intuition, as problems marked as “Hard” with lower acceptance rates are anticipated to pose greater challenges for the model.
The efficiency patterns of ChatGPT in terms of run-time and memory usage reveal complex relationships between problem difficulty, acceptance rate, and the model’s performance.