LaTeX Author Guidelines for CVPR Proceedings

Medical diagnosis of histopathology through the examination of whole slide images (WSIs) is a cornerstone of modern pathology. WSIs are high-resolution, digitally scanned histopathology cases, providing an extensive view of tissue architecture and cellular detail. Pathologists navigate these gigapixel-scale images to identify morphological features and spatial relationships critical for accurate diagnoses. They start with a low magnification to identify suspicious regions and then zoom into image patches for detailed examination [ghezloo2022analysis, liu2024semantics]. They gather evidence across patches and accumulate them together to make a final holistic diagnosis. This process is the gold standard. However, it is labor-intensive and requires significant expertise to interpret complex visual information effectively. It is becoming increasingly unsustainable due to the rising number of cancer cases globally.

The shift towards more efficient diagnostic methods in medical imaging is essential, yet must maintain accuracy. Recent advancements in deep learning report achieving expert-level performance, promising such a scalable approach [topol2019high]. However, current methods typically divide WSIs into smaller patches for independent analysis, making diagnoses without the holistic context [li2021dual, yang2024foundation, zhou2024pathm3, seyfioglu2024quilt, sun2024pathgen, ahmed2024pathalign, xu2024whole, ikezogwo2023quilt, gu2023augmenting]. Transformer-based models attempt to capture both local and global patterns but are not scalable with the high-resolution demands of WSI [shao2021transmil, wu2021scale, guo2023higt, zheng2022graph, chen2022scaling].

In contrast, we propose PathFinder, a multi-modal and multi-agent system designed to mimic the decision-making process of expert pathologists by integrating four AI agents: Triage, Navigation, Description, and Diagnosis. The system begins with the Triage Agent, classifying the WSI as benign or risky; if risky, the Navigation and Description Agents iteratively examine patches, generating natural language descriptions and refining their focus with each cycle. Finally, these detailed insights are integrated by the Diagnosis Agent to produce an accurate and holistic diagnostic classification. Figure LABEL:fig:nav-pipeline-simple demonstrates an overview of PathFinder’s pipeline.

Our experiments demonstrate that the proposed agentic system significantly outperforms prior state-of-the-art (SOTA) methods on WSI skin melanoma grading, achieving an accuracy of 74% on the M-Path Skin Biopsy dataset [elmore2017pathologists]. This marks an 8% improvement over the best baseline with accuracy of 66% and a 9% improvement over the 65% average performance of pathologists [elmore2017pathologists]. Our proposed system is also fully explainable from the patches visited to the description of the patches and the final diagnoses, which takes into consideration all the patch-wise information. To the best of our knowledge, PathFinder is the first AI-based system capable of surpassing the average performance of pathologists on this challenging melanoma classification task.

Multi-modal Histopathology Models. There have been a series of studies in histopathology that leverage WSI-level and pacth-level images to train unimodal classifiers based on multiple instance modeling leveraging pretrained feature extractors [shao2021transmil, scatnet, hatnet]. More recently unimodal foundational models trained on varying self-supervised objectives have achieved significant improvements on performance downstream [xu2024whole, virchow, ikezogwo2022multi, uni]. With the introduction of large-scale multi-modal datasets in histopathology, we have seen significant advancements, with the emergence of large language models and vision-language models for histopathology. For instance, studies like Quilt-1M [ikezogwo2023quilt] and PathGen-1.6M [sun2024pathgen] curate large histopathology image-text paired dataset and train CLIP-based models to learn joint vision-language representations, significantly enhancing clinical histology downstream tasks on patch-level. On the WSI-level, PathAlign [ahmed2024pathalign] aligns diagnostic texts from pathology reports with corresponding WSIs, facilitating applications such as automatic report generation and case/patient-level visual question answering, moving towards a more clinically integrated and holistic diagnostic process. While many other studies like Quilt-LLaVA [seyfioglu2024quilt], SlideChat [chen2024slidechat], and PathChat [pathchat] train histopathology Multi-modal Large Language models (MLLM) and improve on diagnostic reasoning tasks, none of these models effectively automatically navigate the giga-pixel scale WSIs towards a diagnosis.