DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production

Large language models (LLMs) are powerful pre-trained models designed to process and generate natural language. These models undergo two key training phases: first, they are pre-trained on vast amounts of text in an unsupervised manner, learning general linguistic patterns and structures; then, they are fine-tuned for specific tasks to optimize their performance on particular applications.

LLMs used for code-related tasks can generally be grouped into three categories based on their architecture: encoder-only models, decoder-only models, and encoder-decoder models [10]. These models, often built on the transformer architecture, are known for their superior ability to learn from large datasets and scale effectively to handle complex tasks.

• •

  Encoder-only models, such as CodeBERT [11], rely on a bidirectional transformer encoder and attention mechanisms to learn vectorized embeddings of input code sequences. These models are particularly suited for tasks that require understanding and representation of code, such as code classification, code clone detection, and code search, where generation is not the primary objective.

• •

  Decoder-only models, like CodeGen [12], InCoder [13], and Codex [14], utilize an autoregressive transformer decoder to generate code sequences. These models excel at open-ended tasks such as code generation, where the goal is to produce new code from a given input prompt, making them ideal for scenarios like autocomplete or writing code from natural language descriptions.

• •

  Encoder-decoder models, such as LEAM [10], combine both an encoder and a decoder, making them versatile for tasks that involve both understanding and generating code. These models are effective in a range of applications, including code completion, summarization, and generation, allowing them to handle more complex, multi-stage tasks in development workflows.

Regardless of their architectural differences, most LLMs can be fine-tuned with task-specific datasets to improve performance on particular tasks. As a result, the need for high-quality training data has become more critical than ever [9, 15, 16]. In this paper, we focus on supervised fine-tuning (SFT) on query-response pairs derived from real-world developer interactions, which is promising to enhance the performance of LLMs in code-related tasks. However, obtaining this data remains a significant challenge.

There have been two main methods of collecting such training data: human annotation and automated data generation [17, 18, 19, 20, 21, 22]. While human annotation provides high-quality, contextually relevant data, it is a time-consuming and costly process that limits scalability. On the other hand, existing automated approaches often fall short of capturing the diversity and complexity of real developer interactions, resulting in synthetic datasets that do not fully represent real-world behaviors in integrated development environments (IDEs). These limitations hinder the effectiveness of LLMs when applied to practical coding scenarios, where diversity in query types, code contexts, and problem-solving approaches is critical for model success.

IDE users frequently interact with code completion, explanation, and repair tools embedded within the environment. These interactions often involve multi-step, context-driven queries that span across different programming languages, code files, and even past interactions in the same session. Therefore, capturing this rich context, is paramount for collecting training data, representing diverse real-world scenarios.

The process of QA-DBA is shown in Figure 2. To ensure the generated data is diverse and reflects real online interactions, we first conduct a thorough analysis of developer behaviors using our QA-DBA module. This agent performs a comprehensive review of Q&A interactions within IDEs, capturing 10 key behavioral dimensions across more than 40 categories as shown in Table I, of which the first 7 are classified by the Rule Matcher, and the last 3 are classified by the LLM classifier with the prompt in Figure 3. This analysis informs a data production plan, tuned to reflect online developer behavior patterns, as shown in Figure 4. The QA-DBA framework ensures that the generated training data mirrors real-world developer interactions, maintaining consistency and diversity across different query types, code contexts, and development tasks. The LLM classifier and planner are both based on the Deepseek-Coder-V2-Instruct [24] model.

The chat configuration generator takes as input the interaction context based on developer behavior patterns analyzed by QA-DBA. These configurations define the repository setup (e.g., cursor position, selected code) and trigger mechanisms (e.g., query type, dialog turn) within the IDE (as illustrated in Figure 7). Based on the generated configuration with Cursor Behavior and Programming Language definitions, an open-source repository from BigCode  [25] randomly selected to craft queries. A sample query generation prompt is illustrated in Figure 5, which incorporates various attributes such as Query Locale, Intent, and Difficulty Level. A sample of the generated query is in Table II. In the final phase, we harness the prompt illustrated in Figure 6 and pass it through Deepseek-Coder-V2-Instruct [24] that filters out the queries based on their quality. These configurations ensure that the generated data reflects diverse development scenarios, resulting in high-fidelity training data that closely mirrors real-world developer interactions.

Due to the complexity of replicating real IDE environments and the lack of a low-cost solution to operate the IDE via APIs, we employ a UI automation tool to simulate user interactions within the IDE. This tool executes predictable tasks such as selecting files, positioning cursors, and triggering the Q&A interface, based on the chat configurations. The Q&A triggering process is shown in Figure 7. According to the input chat configurations, the UI automation tool will first pull down the repository and open the target file, the cursor position will first determine whether the code is selected or not, and then determine whether to move the cursor to the target position. In the next step, the tool will trigger the online Q&A process by clicking on the icon to get to the Q&A interface and entering the query in the dialog box. It captures the resulting Q&A interactions, which are stored and iteratively refined for further use in model training. The use of UI automation not only ensures data diversity but also dramatically reduces the manual effort required for generating large-scale datasets. Details of the UI automation process are shown in pseudo code in Algorithm 1.

After generating diverse and complex queries, high-quality responses are crucial for producing a useful training dataset. To achieve this efficiently, we employ a multi-model response generation pipeline using models from the DeepSeek series [26, 24]. The pipeline comprises two phases: response generation and response judgment.

We first de-identified real developer Q&A data to ensure compliance with privacy standards. For evaluation, we balanced the dataset according to user intent and programming languages, selecting a random sample of 300 entries. The evaluation process was carried out by developers proficient in various programming stacks who manually scored the model’s responses.

The scoring process is shown in Figure 10. Each model’s final score is the average of two rounds of inference, with each round going through three stages: initial scoring, quality inspection, and final score confirmation. A team of 21 annotators handled the scoring process: 13 for initial scoring, 6 for secondary quality inspection, and 2 for final score confirmation. We used a penalty points system (Table IV) for scoring, with a maximum score of 5.

We established two key evaluation metrics:

• •

  Usability Rate (UR): The proportion of responses deemed usable ($FinalScore\geq 4$).

• •

  Perfect Score Rate (PSR): The proportion of responses judged as perfect ($FinalScore=5$).

We use Deepseek-coder-33B-Instruct as the seed model and employ high-quality Q&A data produced by DialogAgent for supervised finetuning (SFT) , and each SFT model iterates for 3 epochs and follows the experimental setup of [23]. During inference, we generated two response variations per prompt using a temperature of 0.3 and top-p of 0.95. The results were averaged over two rounds of inference.

In this section, we assess the impact of DialogAgent on six key code Q&A tasks: code generation, code editing, code explanation, comment generation, code repair, and general Q&A. The comparison results between the seed model (DeepSeek-33B-Instuct) and the fine-tuned model (DeepSeek-33B-Instruct-SFT) are shown in Tables V,  VI and  VII.

We observe significant improvements across all tasks and metrics for the fine-tuned model. For code generation tasks such as code repair, code generation, and code editing, UR increased by 11%, 19%, and 50%, respectively. For code understanding tasks such as code explanation, general Q&A, and comment generation, UR improved by 15%, 2%, and 55%, respectively.

To study the efficiency compared with human annotators, we asked the annotation team to construct code Q&A pairs with the help of online search and other intelligent tools. However, manual construction of code Q&A pairs is time-consuming, yielding only 30 pairs per day per annotator. In contrast, DialogAgent can generate 1,440 pairs per day per instance — 4.8 times the productivity of human annotators. By scaling computational resources, multiple automation instances can further increase production capacity. Additionally, human inspection also reported that synthetic data generated by DialogAgent demonstrates superior variability and consistency with real-world distributions compared to manually generated data.

Ww designed 3 groups of ablation experiments to verify the effectiveness of its online consistency, data type increment, and data increment, respectively, and the overall experimental results are compared as in Table VIII, the results of the code generation intention as in Table IX, and the results of the code understanding intention as in Table X.

Synthetic data’s consistency with online reality. We generated a batch of data randomly, incorporating elements such as arbitrary programming languages, selected code fragments, and queries, etc. By employing identical training and inference parameters, we fine-tune the Deepseek-coder-33B-Instruct based on the two types of synthetic data, separately. The comparison result is DS-33B-Inst-SFT(Random) VS DS-33B-Inst-SFT(Ours). Evidently, the data generated by DialogAgent overall performs better, an absolute increase of ${5\%}$ in ${PSR}$ and ${6\%}$ in ${UR}$. It’s noteworthy that in code editing and comment generation, the absolute values of ${UR}$ have improved by more than ${10\%}$.

Data type increment of synthetic data. Multi-turn dialogue capability of a conversational model is important for the performance of IDE code Q&A. However, in the complex task of generating high-quality multi-turn dialogue data, our DialogAgent provides robust support. We compared the effectiveness of fine-tuned models using synthetic data, considering the presence or absence of multi-turn dialogue data (abbreviated as MTD), i.e., DS-33B-Inst-SFT(Ours) VS DS-33B-Inst-SFT(w/o MTD) and DS-33B-Inst-SFT(subset) VS DS-33B-Inst-SFT(subset w/o MTD). The results indicate that the multi-turn dialogue data generated by DialogAgent can further enhance the performance of the SFT model, with an absolute increase of ${3\%}$ in the ${PSR}$ and ${6\%}$ in the ${UR}$. For intents with a stronger historical dialogue dependency (code editing and code generation), there is at least a ${7\%}$ improvement in ${UR}$.

Increment of synthetic data. DS-33B-Inst-SFT(subset) is a subset of the final version dataset DS-33B-Inst-SFT(Ours) with half the data volume. The comparison results indicate that the model with the increased SFT data from DialogAgent can sustain stable effects on the old period evaluation set.

In the same time, the final version of DS-33B-Inst-SFT(Ours) also includes a new batch of data generated based on online data from another time period, and we sampled online data from this time period to serve as a new evaluation data. The comparison results in the new evaluation data are presented in Table XI,  XII and  XIII. Notably, the model with the increased SFT data generated in the new period performs significantly better on the new period’s online data, with ${PSR}$ achieving a ${4\%}$ increase, ${UR}$ a ${1\%}$ increase.

Given that DialogAgent can autonomously generate data closely resembling online user behavior based on developers’ online activity, we can infer that an SFT model based on DialogAgent can achieve continuous automatic optimization.

To ensure the quality of the generated responses, we evaluate the overall quality and optimization benefits of the response generator, and also assessed the consistency between the automated judgment system and human scoring.

Our annotation team manually score 430 data samples with responses generated by each LLM in our response generator pool. Table XV shows the overall $PSR$ of the LLM pool and the $PSR$ of each LLM in the pool. Although the PSR of each model is not high, when combined together, it reaches 79.76%. Since each model excels in different types of questions, the approach of using the LLM pool as a response generator combines the strengths of the various LLMs, enabling 79.76% of the data produced in one round has the quality to be included in the training set. Despite this, some prompts still lack perfect answers and will be added to the next production chain until a perfect answer appears. This has to some extent affects the efficiency of our data production. In order to increase the proportion of data that can be used as a training set in each round of data production, we also iteratively optimize the LLM pool through self-improvement method. Table XIV illustrates that after fine-tuning all the models in the LLM pool with the training set from the last production round, the ${PSR}$ for each intent significantly increases and the average ${PSR}$ has an increase of 8.65%. The ${PSR}$ for the Comment Generation intent, particularly, increased by 19.32%.

To evaluate the accuracy of our automated judgment system, our annotation team manually scores 600 responses with distinct instructions. Since the responses we ultimately select for the training set are only 5-point responses, the accuracy rate of scoring 5-point responses is used as a metric to assess our judement system:

$Accuracy_{5}$ represents the proportion of 5-point responses scored by our judgement system that align with the scores given by human annotators. It measures the degree of agreement between the judgement system’s scoring of 5-point responses and the human evaluations. Also, ${pred_{5}\textunderscore pos}$ indicates instances where both judgement system and human annotation assigned a score of 5, and ${pred_{5}\textunderscore neg}$ represents instances where the judgement system assigned a score of 5, but the human annotation did not.

Considered human scoring as the ground truth, the $Accuracy_{5}$ for the judgement system is 88.47%. Among the ${pred_{5}\textunderscore neg}$ instances, 96.00% are usable responses, scored more than 3-point by human annotators. 81.62% of the responses manually scored as 5-point are recalled by the system. This indicates that our judgment system has a relatively high consistency with human scoring, and most of the inconsistently scored 5-point responses are still usable.