Toward Universal Embeddings in Digital Pathology through Multimodal LLMs

We propose PMEB, a comprehensive benchmark designed to evaluate pathology multimodal embeddings. PMEB include cancer tissue classification [2, 4, 22], Gleason pattern grading [36], and cell identification [9], among other tasks. As shown in Table 1, PMEB comprises 15 original tasks from 14 datasets, organized into three meta-tasks: retrieval, classification, and composed retrieval. All tasks are reformulated to assess multimodal embeddings, where the model receives an instructional prompt and a query, which may consist of text, images, or both, and selects the correct target from a set of options.

In the retrieval task, the query can be either an image or text, with the target being the corresponding paired text or image. For the classification task, queries are images with targets as text labels representing various classes. The composed retrieval task involves queries that combine an image with text formatted as a question, with the target being the expected answer. This composed retrieval task, specifically designed for multimodal embedding evaluation, incorporates integrated visual and language inputs and has not been applied in previous methods.

Due to the large size of some datasets, we standardize their sizes. For datasets containing more than 1,000 samples (except Osteo [3]), we choose a random sample of 1,000, ensuring the original category distribution is preserved. For the composed retrieval task, we select only open-ended questions from the data source, where answers are more complex than a simple “yes” or “no”. Additionally, since there is an overlap between questions and answers in the VQA data source, we utilize ChatGPT [1] for data cleaning to ensure clarity and accuracy in task evaluation. Further details of each dataset within PMEB are provided in Supplementary Materials.

We first introduce how our proposed MLLM4PUE leverages MLLMs for universal multimodal embedding learning. Unlike CLIP, which directly generates embeddings for multimodal inputs, MLLMs are inherently generative models and require a different approach to produce embeddings. We uses a prompt-based strategy specifically designed to guide MLLMs in embedding multimodal data.

Given an MLLM $f_{\varphi}$ and a query $q$, we use the prompt, <$q$> \n Summarize the above <> in one word:, to generate $q_{input}$. In this setup, <$q$> serves as a placeholder for the query, while <> indicates the type of the query’s content. For instance, if query $q$ is an image, the prompt would be <image>\n Summarize the above H&E image in one word:; if query $q$ is a sentence, then the prompt would be <text>\n Summarize the above sentence in one word:.

The prompt design incorporates two components: first, the word Summarize directs the MLLM to distill the information of the multimodal input. The second part, in one word:, instructs the MLLM to compress this distilled information into a single token, thereby facilitating a unified multimodal embedding. Once the extended input $q_{input}$ is constructed, it is processed by the MLLM $f_{\varphi}$ to generate embeddings: $h=f_{\varphi}(q_{input})$, where $h$ is the embedding of last token obtained from the MLLM output, as illustrated in Fig. 3. This approach tailors the generation capabilities of MLLMs to effectively produce and utilize multimodal embeddings, expanding their application potential beyond traditional generative outputs.

Next, we describe the approach for projecting pathological data into a latent embedding space using conservative learning. As illustrated in Fig. 2, given a batch of $N$ paired image and text samples $\{(v_{n},t_{n})\}_{n=1,\dots,N}$ and MLLM $f_{\varphi}$, for each sample $(v_{i},t_{i})$, we feed them into $f_{\varphi}$ to get the image and text embeddings $(h_{vi},h_{ti})$ as described in Sec. 3.2.

The objective is to build a latent space for pathology image-text embeddings that maximizes the similarity between embeddings of the paired samples in the batch, while minimizing the similarity between embeddings of the $N-1$ incorrect pairs. We optimize an infoNCE contrastive Loss $\mathcal{L}$ to train our model.

$$\mathcal{L}=-(\log\frac{e^{cos(h_{vi},h_{ti})/\tau}}{\sum_{j=1}^{n}e^{cos(h_{vi},h_{tj})/\tau}}+\log\frac{e^{cos(h_{ti},h_{vi})/\tau}}{\sum_{j=1}^{n}e^{cos(h_{ti},h_{vj})/\tau}}),$$

(1)

where $\tau$ is a temperature parameter and $cos(h_{vi},h_{ti})$ and $cos(h_{ti},h_{vi})$ represent the cosine similarity in both directions in contrastive learning.

Our model, trained primarily on retrieval tasks focused on image-text pairs, requires adaptation to handle zero-shot classification and composed retrieval tasks. The details of how we modify the model for these different zero-shot downstream tasks are introduced as follows.

For zero-shot classification, we adopt a prompt-based method inspired by the CLIP model [34]. In this approach, each class name is expanded into a sentence using a specific template. For instance, the class name “Colon adenocarcinoma” is converted into the sentence “An H&E image of Colon adenocarcinoma” using the template “An H&E image of { }.” We apply this method to create sentences for all class names. Our model then computes embeddings for these sentences and the test images, extracting the last token of the MLLM output as described in Section 3.2. The similarity between these embeddings is calculated as outlined in Section 3.3, and test image labels are assigned based on the highest similarity scores.

For the composed retrieval task, where the query consists of both an image and a question, we need to convert this multimodal information into a unified embedding. Thus, we use the prompt <image>\n <question>\n Summarize the above H&E image and question in one word: to integrate the image and the question. Our model then generates embeddings for this combined input, as explained in Section 3.2, which are used to retrieve the embeddings of the corresponding answers.

We first evaluate our methods on the retrieval task on Arch-PubMed [12] and Arch-book datasets [12], where the query is an image or text, with the target being the corresponding paired text or image. Recall@k metric is employed to assess the model’s ability to retrieve relevant pathology image-text pairs among top k results. As shown in Table 2, our models consistently outperform the baseline models, especially at higher recall values. Notably, in the image-to-text (i2t) task on Arch-book dataset [12], our MLLM4PUE trained on Openpath dataset [15] achieves a Recall@5 of 0.185, indicating an approximate 9% improvement over PLIP [15]. Moreover, our PathCap-pretrained model surpasses the performance of PathCLIP [37] on both benchmark datasets. A similar enhancement is observed in the text-to-image (t2i) retrieval task. For instance, on the Arch-PubMed dataset [12], our model achieves a Recall@10 of 0.193, marking a 10% increase in performance. These results collectively demonstrate that our framework excels at capturing multimodal embeddings more effectively than CLIP-based models with the same training data. Furthermore, it is important to note that the E5-V model [18] yields significantly lower recall scores across all tasks and datasets due to the lack of fine-tuning with pathology-specific data.

We then evaluate the performance of zero-shot classification on patch-level pathology images across 13 classification tasks from our PMEB. In this downstream task, each query is an image, and the target is an extended sentence of the class name as described in Section 3.4. Table 3 illustrates that our MLLM4PUE consistently surpasses baseline models on most datasets. Notably, with the same training data, MLLM4PUE-O outperforms PLIP [15] across all datasets. Similarly, MLLM4PUE-Q achieves the best performance compared to QuiltNet [16], demonstrating the effectiveness of our pretraining strategy on the same training dataset. While CONCH [31] attains the highest score on SkinCancer [22] and Wsss4luad [14], its superior performance stems from utilizing a much larger private dataset (1.16M). In contrast, our model, MLLM4PUE-P, outperforms all baseline models across seven datasets with only 0.2 million training samples, and is only 0.02 less than CONCH [31] on the Wsss4luad dataset [14]. This underscores the advantages of leveraging vision-language alignment in pathology representation learning. These findings collectively highlight the effectiveness of our approach, especially considering that our models were trained on fewer samples, further demonstrating their data efficiency in pathology-specific zero-shot classification. The accuracy metric shows a similar trend, where MLLM4PUE-P outperforms the baselines on most datasets.

We further evaluate the performance of composed retrieval on two datasets, Quilt-VQA [35] and Quilt-VQA-RED [35], where the task involves using an image and a question as the query to retrieve the correct answer. For our model and E5-V [18], we apply the prompt <image>\n <question>\n Summarize above H&E image and question in one word: to integrate the image and the question. For the other three CLIP-based baseline models, we add the image and question embedding directly since these models handle image and text separately. The Recall@k metric is employed to assess how well the model can identify the correct answer within the top k candidates. As shown in Table 3, our proposed MLLM4PUE achieves better results compared to baseline methods. Specifically, on the Quilt-VQA dataset [35], MLLM4PUE achieves an increase in Recall@5 by approximately 4%, 3%, and 30% over PLIP [15], PathCLIP [37], and QuiltNet [16], respectively. Similarly, on the Quilt-RED-VQA dataset [35], MLLM4PUE pre-trained on Quilt1M [16], has achieved a Recall@5 of 0.675, which represents the best performance. Interestingly, the E5-V model [18] exhibits superior performance compared to all CLIP-based models, despite lacking fine-tuning on pathology-specific data. This demonstrates the robust capacity of our method in multimodal fusion, highlighting its proficiency in handling diverse data inputs effectively.

To illustrate the modality gap, we employed t-SNE visualization to compare the embeddings of our MLLM4PUE-P model with those of PathCLIP [37], both trained on the Patchcap dataset [37]. As shown in Fig 4, the qualitative analysis clearly shows that the embeddings produced by MLLM4PUE-P form tighter clusters in the low-dimensional space, whereas those from PathCLIP [37] are more dispersed. This difference reflects a significant improvement in cross-modal alignment of our method.

To further quantify the modality gap, we adopted the metric $||\Delta||_{gap}$ as proposed in Liang et al.  [26]. Our experiments reveal that, on the Arch-PubMed dataset [35], our method reduces the gap from 0.9284 to 0.2587, and on the Arch-Book dataset [35], from 1.0408 to 0.3739. These results provide strong evidence that our model can significantly narrow the gap between image and text embeddings.

Robustness of PMEB. Since some datasets in our PMEB are super large, we choose a random sample of 1,000. To assess the robustness of our benchmark, we conduct an ablation study by computing the wf1 scores using 1,000, 3,000, 5,000, and all available samples for each dataset. As shown in Figure 5, the performance change between 1,000 and the full dataset remains within a small margin across all datasets, confirming that using only 1,000 samples provides a reliable approximation of the complete dataset’s performance. Given that the performance difference is typically within 0.01 to 0.04 across different sample sizes, using 1,000 samples significantly reduces computational costs while maintaining reliable performance. This makes PMEB highly efficient for large-scale experiments without sacrificing meaningful evaluation. We also report the result of accuracy metric and the results in CONCH [31] and PathCLIP [37] model in the Supplementary Materials, which shows the same trend.

Second, the results reveal that QuiltNet [16], a CLIP-based model, shows little difference between using only the question text and integrating image and text embeddings. This indicates that CLIP’s method lacks robust modality integration, as the inclusion of visual information fails to significantly enhance its effectiveness. Conversely, our model MLLM4PUE demonstrates substantial improvements by effectively fusing image and text information. The true strength of our approach becomes evident when employing a prompt-based method, leading to significantly higher recall scores. Specifically, it achieves Recall@5 values of 0.475 on Quilt-VQA and 0.675 on Quilt-VQA-RED, representing approximately an 8% and 12% improvement over the simple addition method, respectively, and an 18% and 25% improvement over using the question text alone. These findings clearly indicate that our model’s ability to integrate both visual and textual modalities results in significant enhancements in retrieval accuracy. This surpasses both mere modality addition and the question-only configuration, highlighting the effectiveness of our modality fusion technique for composed retrieval tasks. Our approach provides advantages that cannot be achieved through separate handling or limited integration of text and image data.