Code Like Humans: A Multi-Agent Solution for Medical Coding

The proposed CLH framework decomposes the medical coding task into four steps, corresponding to the Analyze-Locate-Assign-Verify approach implemented by the UK National Health Service NHS England (2023). We first describe the framework abstractly in terms of the interface and functionality of each component. In section 4.2, we present our implementation, which reflects one of many possible realizations of CLH and is not intended as a definitive standard.

We implement the CLH framework as an agentic system (see Figure 1). We define agents as modular components that can be implemented in various ways, such as standalone language models and retrieval-augmented generation (RAG).

In our implementation, each agent is instantiated as a distinct inference step using the same backbone model, but with different instructions (via prompts) tailored to their role (see Appendix C). The current implementation relies on models with ‘thinking-enabled’ mode to enable test time computation Snell et al. (2024); Muennighoff et al. (2025).

We use the ‘reasoning’ models for improved performance; theoretically the ‘reasoning traces’ could also provide transparency into the model decision process, but at least for the current ‘reasoning’ models this process is not faithful Chen et al. (2025); Kambhampati et al. (2025); Shojaee et al. (2025); Zhao et al. (2025). Specifically, we experiment with three open-weight models of different sizes: small (DeepSeek-R1-0528-Qwen3-8B), base (DeepSeek-R1-Distill-LLaMA-70B), and large (Qwen3-235B-A22B). Additionally, we evaluate two closed OpenAI models, o3-mini and o4-mini, under a HIPAA-compliant use setup. Further implementation details are provided in Appendix B.

The evidence extractor (step 1) operates solely on clinical notes, extracting verbatim text snippets expected to justify coding decisions. For each snippet, we retrieve the top-10 alphabetical terms by embedding snippets and terms into the same semantic space (see appendix D.1 for details). Next, the index navigator (step 2) processes each snippet’s set of terms in parallel, selecting the most appropriate terms, which in turn yield a set of candidate codes. These candidate codes, often grouped by ICD chapter, are then passed in parallel to the tabular validator (step 3) along with chapter-level guidelines,⁸⁸8The guidelines span 100+ pages; each chapter averages about three pages. Parallel processing lets us input only the chapters relevant to the candidate codes, typically one or two. receiving input triplets of {clinical note, chapter guidelines, candidate codes} to yield tentative assigned codes. Finally, outputs from parallel processing steps are merged, and the code reconciler (step 4) receives input triplets of {clinical note, instructional notes, tentative codes} to verify and finalize the coding assignments, where instructional notes are retrieved by looking up codes in the ICD hierarchy.

We focus on ICD-10-CM, as (i) ICD-10 is the most widely adopted coding system globally Teng et al. (2023), (ii) its introduction made the coding task much more laborious and time-consuming,⁹⁹9It takes a professional inpatient human coder approximately 30 minutes per case on average Chen et al. (2021). as the number of codes increased more than sevenfold (rising from  9,000 to  70,000), and (iii) it is well-maintained with high-quality open-access resources to guide coding decisions.

Like previous studies, we report F1 scores (micro and macro), exact match ratio (EMR), and recall for each agent. We pay particular attention to macro F1, which assigns equal weight to every label. Edin et al. (2023) shows that F1 macro is shaped by the long tail of low-frequency labels, where per-label F1 increases with log frequency until roughly 100 training examples and then plateaus. We call this long-tail subset “rare codes”; gains on rare codes, therefore, translate directly into higher F1 macro.

We evaluated our implementation of the CLH framework end-to-end, comparing it directly to baseline models on MDACE. Table 1 summarizes the results. With the largest base model, the CLH approach achieves comparable performance with PLM-ICD, while handling a 70-fold larger label space.

Notably, while PLM-ICD achieves higher performance on frequent codes (as indicated by its superior F1 micro score), this advantage primarily stems from its supervised training regime, which utilizes data from the same intensive care unit (ICU) as the test set. This shared setting creates strong distributional priors, allowing PLM-ICD to excel specifically on frequent codes within that clinical practice. Conversely, CLH does not benefit from such distributional priors, which explains its superior performance on rare codes.

To our knowledge, no prior work has considered the full label space. Hence, when considering the realistic setting of human coders, our results can be regarded as the state-of-the-art. In the future, it may be helpful and fairer to compare medical coding systems in the full and label-constrained settings.

As discussed in subsection 4.1, the rationale for introducing the CLH framework is the 4-step process used by medical coders at the NHS. To consider whether this process was indeed beneficial to CLH, Table 2 reports the results for each component in the filtered setting (i.e., the errors made at the previous step of the pipeline are discarded).

We observe that the index navigator (step 2) achieves high recall with low precision, suggesting broad coverage of candidate codes. However, the tabular validator (step 3) and the code reconciler (step 4) show much higher precision while maintaining high recall.

As with any pipeline approach, CLH can propagate early mistakes to later agents. A promising direction for mitigating this in future work is self-refinement Madaan et al. (2023) on the same clinical note. After pass $t$, the code reconciler (step 4) outputs a set of codes with rationales. We could append this output to the note as a lightweight scratchpad, then re-invoke all steps once again. The process then repeats for $t{+}1$.

Falis et al. (2024) observed that GPT-3.5 works well for coding only when code descriptions are verbatim. That setting is rare in practice, where cues embed abbreviations, medical jargon, and implied context. We analyzed the MDACE expert-annotated evidence spans with the text snippets extracted by the index navigator (step 2). Then, we measured recall at the level of ICD-10 classification chapters to identify which cues the agent overlooks, and which the retriever fails to link to alphabetical-index terms.

Figure 2 shows the retrieval recall@$25$ for different ICD-10 chapters. On the x-axis, we report recall based on alphabetical terms retrieved using the evidence extractor (step 1), while the y-axis reports recall based on expert-annotated text snippets. In each case, we calculate recall for the top 25 retrieved ICD-10 terms. High recall for J00–J99 (respiratory) and N00–N99 (genitourinary) shows that both the agent and expert annotations reliably retrieve index terms for body-system diseases. When the expert annotations recall is limited, as for C00–D49 (neoplasms), S00–T88 (injury or poisoning), and V00–Y99 (external causes), the agent cannot compensate, indicating a retrieval bottleneck. In contrast, we find that for the chapters F01–F99 (mental and behavioral disorders), H00–H59 (eye and adnexa), and Z00–Z99 (factors influencing health and contact with services), the 1. evidence extractor agent underperforms significantly.

To further investigate this shortcoming, we inspected the word clouds for the evidence spans linked to false negatives (see Figure 8 in the subsection D.2). Frequent misses involve abbreviations (Hx of CVA), social or historical qualifiers (quit smoking, tobacco use), administrative directives (DNR), and medication names (warfarin, coumadin). These cues are not active disease mentions but modifiers that require abbreviation resolution, temporal inference, and context classification. Psychological terms such as depression and anxiety are likewise overlooked, explaining the agent’s low performance on F01–F99.

Given that much note content is noise Liu et al. (2022), we speculate that CLH could operate on entity+assertion spans rather than full notes, a choice recently shown to maintain accuracy and provide clearer entity-level evidence (Douglas et al., 2025).

In this section, we study the effect of the candidate space in a controlled setting that does not rely on CHL. Concretely, at inference time we construct a per-note candidate set $C$ by combining the ground-truth codes $P=\{y_{i}:y_{i}=1\}$ with $K$ hard negatives per positive: $C=P\cup R_{K}$ with $|R_{K}|=K|P|$. We identify hard negatives by embedding the short descriptions of codes using our retriever (S-PubMedBert-MS-MARCO, see subsection D.1) and sampling the $K$ nearest neighbors that are not in $P$. This allows us to examine how the size of the candidate set affects model behavior. We then run the tabular validator (step 3) and the code reconciler (step 4) only on the candidate set $C$. The tabular validator (step 3) receives mutually exclusive choices, which simplifies selection, while the code reconciler (step 4) operates in a multi-label setting and must also decide how many codes to output.

Figure 3 shows the F1 micro scores of the tabular validator (step 3) and code reconciler (step 4) when scaling the number of negative codes per positive code. As the candidate set grows, both agents exhibit a decline in performance, confirming the challenges of reasoning over larger solution spaces in long, context-rich sequences Li et al. (2024). However, the tabular validator (step 3) proves more robust, benefiting from a mutually exclusive candidate set that simplifies the selection task. In contrast, the code reconciler (step 4) must also determine the number of correct codes to output, compounding the difficulty in a multi-label setting. These findings validate our framework’s modular design, where the tabular validator (step 3) predicts tentative codes in parallel, and the code reconciler (step 4) audits them in a final step.

Though the context window of some LLMs exceeds millions of tokens Gemini Team et al. (2024), they are known to have an effective context window of an arbitrary smaller size Lee et al. (2024). We conducted an ablation study to establish how the capacity of code reconciler (step 4) saturates with adding more data. We incrementally enriched the input with code-specific information to examine how the model leverages the added context. We started with (1) alphanumeric code identifiers, adding (2) short code descriptions, and then (3) chapter-specific guidelines. We follow the same setup as in subsection 6.2 to construct the candidate set.

Figure 4 shows that enriching the context input with structured guideline information affects the performance of tabular validator (step 3). Starting with only alphanumeric code identifiers, we observe that adding short descriptions leads to clear improvements. Including chapter-specific coding guidelines yields further gains, particularly as the number of candidate codes increases. These results highlight two observations: the model benefits from in-context reasoning grounded in guidelines, and the ability to process a larger candidate set improves with access to richer contextual information about codes. However, as the trend is unlikely to continue infinitely, in practice, the effective size of the context should be identified for a given model.

Test-time compute has recently improved accuracy on multiple-choice QA and math benchmarks Snell et al. (2024); Muennighoff et al. (2025). We analyzed the performance impact of ‘thinking-enabled’ output generation employing the 3. tabular validator. This setup begins with an unconstrained ‘reasoning’ phase, enclosed within <think>...</think> tags, followed by a constrained output phase within <answer>...</answer> tags. We compared this to structured decoding, where outputs are constrained using a regex pattern to a predefined set of candidate codes Willard and Louf (2023), and code prediction begins from the outset of the generation.

Figure 5 compares F1 micro score as the number of negative codes per positive code increases for the 3. tabular validator with and without ‘thinking-enabled’ generation. ‘Thinking-enabled’ generation consistently outperforms structured decoding. Both settings achieve near-perfect accuracy at zero negatives, yet the gap widens as the candidate set and context length grow. The gentler slope suggests the agent can better explore and exploit in-context information throughout the generation process. In contrast, the structured decoding approach initiates code prediction from the outset, limiting the benefit from additional context. It only improves performance marginally when the alphanumeric codes are extended with semantic descriptions.

In the scope of this work, we consider primarily the performance of the ‘thinking’ mode, and do not investigate its interpretability or faithfulness.

Despite the growing interest in applying auto-regressive LLMs to clinical text, to our knowledge, no study has fine-tuned them for medical coding. We choose Llama-3.2-1B as our backbone model, and fine-tuned it using LoRA adapters on all the projection matrices in the attention layer and feedforward network. We did not apply any adapters to the language-modeling head. We use supervised fine-tuning, disregarding the order of the codes generated by the model. Due to the limitation of the input size of this model, we can only provide 50 code options in the model prompt. We train and test on MIMIC-III-50, as this allows us to provide descriptions for all 50 codes in the prompt. For efficiency reasons, we subset the training, validation, and test sets to 7500, 2000, and 2000 examples. We train with an effective batch size of 32 for five epochs. We pick the model associated with the best validation F1 macro to produce our test-set metrics (see Appendix E for more details). Table 3 shows that fine-tuning a Llama-3.2-1B, a generative model, for medical coding severely underperforms PLM-ICD (a fine-tuned model of BERT generation).

We speculate that much of PLM-ICD’s advantage over predecessor models is due to label-wise attention. We repeat our previous experiment to investigate this empirically, but replace the causal language modeling head with label-wise attention. We follow Edin et al. (2024), implementing it as cross-attention between all tokens and a learned representation per code. We fully train this in addition to the LoRA updates described above. This changes the task from auto-regressive generation to predicting probabilities over the entire label space simultaneously. We tune the decision boundary based on the highest F1 macro score. We use the boundary associated with the best validation set performance for our test scores. All other fine-tuning details are identical to the auto-regressive setting.

Table 3 shows that the performance gap is closed when label-wise attention is added to the backbone model, and the performance matches PLM-ICD. It is interesting that even with the label-wise attention, the Llama-3.2-1B model only manages to recover the original performance of MIMIC-III-50 from Edin et al. (2023), despite being 9 times larger.

Discriminative classifiers, such as PLM-ICD, retain knowledge about every code in their weights, whereas our CLH framework dynamically retrieves ICD codes from the alphabetic index, enabling true open-set coding at inference. CLH does not yet outperform the PLM-ICD baseline on frequent codes, but it achieves comparable performance, and has several practical advantages. CLH is much easier to update when coding conventions change: one would only need to substitute external resources in the prompt. It also suffers less from distributional priors and theoretically can handle even extremely rare codes such as W59.22 “Struck by a turtle” Edin et al. (2023).

Nevertheless, both discriminative classifiers and LLMs have apparent limitations that would make both approaches unusable for the real-world ICD-10 end-to-end classification. The former inherit strong distributional priors and only support a toy version of the real  70K label task. The latter still struggles with precision. Given the current state of the technology, we argue that a more realistic goal for NLP for medical coding is to develop assistive tools that keep human coders in the loop, rather than aim for full end-to-end classification. In that approach, CLH has a key advantage, as it already mimics the workflow of human coders.

Human-computer interaction. CLH framework provides an exciting opportunity to reimagine the task of medical coding as computer-assisted coding, where the goal is not to automate the task but to provide the human coder with the best information to base their decisions on  Murray (2024). This could involve research into user interfaces, identification of the pain points where the coders actually want assistance, the threshold of information overload, potential sources of automation bias, and strategies for mitigating them.

Improving CLH components. CLH allows for more tailored instruction fine-tuning strategies for each component of the pipeline. For example, the evidence extractor (step 1) can be tuned to predict span-level clinical entities with assertion status and key modifiers (e.g., laterality, acuity) Douglas et al. (2025), and to pass only asserted, coding-eligible evidence forward, which reduces noise while preserving code-relevant detail. The index navigator (step 2) can then align entity representations to alphabetical index terms using a label-wise cross-attention objective that improves handling of synonyms and eponyms. The tabular validator (step 3) could be fine-tuned using clinical notes paired with correct codes and closely related incorrect codes, enabling the model to leverage external resources to identify the most appropriate code. Finally, the code reconciler (step 4) can be fine-tuned using clinical notes paired with candidate code lists, incorporating hard negatives to train the model in distinguishing relevant from irrelevant codes with greater precision.