MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining

All image-text pairs in the CFP and FFA modalities were sourced from four fundus diagram books. As depicted in the upper left of Fig. 1, a semi-automated three-stage pipeline was implemented to construct the dataset. First, we manually captured all image-text pairs from the diagram books, ensuring each pair was recorded in a single screenshot with a resolution of no less than 800×800 pixels for each image. These screenshots were then processed using a custom program that integrates Adobe tools for image extraction and OCR technology for text extraction. Then, given the challenge of aligning extracted images and texts—especially in cases where multiple sub-figures are associated with a single caption—we applied regular expressions to achieve precise caption separation. Finally, the images were classified into CFP and FFA modalities using a color histogram-based method. We manually corrected OCR recognition errors and translated the text into English and Chinese reports to ensure language consistency. To further expand our dataset, we refined the DEN dataset[14] through a systematic filtering and cleaning process. This involved manually removing images that did not belong to the CFP, and FFA modalities, as well as excluding collages comprising multiple images. Subsequently, a color histogram-based classification method was used to automatically classify two modalities, and the label files were reorganized according to their respective modalities.

Due to the scarcity of electronic OCT diagram books with rich image-text paired data, we collaborated with a key provincial hospital to establish the OCT modality part of MM-Retinal V2. We initially collected 5,587 OCT images from 3,403 patients. To control data quality, images that could not be diagnosed solely based on OCT were excluded. The ophthalmologist provided detailed definitions for the full range of abnormalities that can be observed from OCT images. Then, each image was independently assessed by two ophthalmologists. Two senior ophthalmologists reviewed all captions and provided final verification and decisions, ensuring the correctness and consistency of the dataset. At last, each OCT image was accompanied by a detailed textual description, capturing both the diagnosed diseases and the pathological features observed in the images.

When diagnosing ocular diseases, ophthalmologists usually rely not only on ophthalmic images but also on their knowledge of ophthalmology and other medical specialties. To support this, we also constructed an MM-Retinal-Text subset, which primarily integrates knowledge from the ophthalmic field. The textual data were sourced from three main origins: (1) four fundus diagram books, (2) three ophthalmology textbooks, and (3) twelve public datasets for medical LLM pretraining. Texts from books were digitized using OCR, with irrelevant elements such as names and figure sequence numbers removed. For public datasets used in medical LLM pretraining, we filtered them using ophthalmology-related keywords derived from the contents of diagram books, as these datasets were originally designed for comprehensive medical areas. Although this filtering approach ensures a focus on ophthalmic knowledge, it may also include content from other medical specialties. We opted not to clean this data further, as a certain proportion of knowledge from other fields would improve the model’s capability to generalize on common medical terms. The detailed experimental results are presented in Table VII. More details on these datasets can be found in our project page.

MM-Retinal V2 is the first high-quality dataset to simultaneously encompass image-text pairs of CFP, FFA, and OCT modalities with ophthalmic text data. In the diagnostic process, different imaging modalities offer unique perspectives. CFP highlights the structure of the fundus, FFA captures vascular changes, while OCT reveals details of the retinal layers. By integrating these modalities with high-quality textual descriptions, MM-Retinal V2 provides a solid base for the development of advanced retinal foundational models.

As the MM-Retinal V2 dataset is derived from comprehensive ocular diagram books and clinical sources, it comprises over 96 fundus abnormalities and disease categories, including both common and rare diseases, such as retinal vascular diseases, macular diseases, vitreous diseases, optic nerve diseases, congenital anomalies, and inflammatory diseases, among others. Fig. 2 (a) shows the words that appear most frequently. More detailed retinal categories are presented in the supplementary material, providing a broader perspective on the extensive coverage of the dataset.

Fig. 2 (b) shows the distribution of caption lengths in MM-Retinal V2. In the CFP modality, 46.8% and 35.0% of captions range from 1 to 25 words and 26 to 50 words. In the FFA modality, 57.8% caption length is over 25 words. In the OCT modality, 91.3% captions range from 1 to 50 words. Moreover, a small percentage of texts exceed 50 words, reaching up to nearly 101 words. This demonstrates that the textual captions in MM-Retinal V2 are detailed and comprehensive, providing accurate and rich descriptions of the images and effectively conveying the information appearing.

MM-Retinal V2 captions encompass disease diagnoses, detailed lesion attributes (such as color, shape, and appearance), clinical symptoms, and post-treatment efficacy, utilizing a rich vocabulary for comprehensive descriptions. Fig. 2(c) compares the vocabulary size and lexical diversity across various public datasets and languages. Notably, since MM-Retinal V2 does not generate captions by merely expanding category names with fixed templates like [33], its vocabulary is exceptionally rich and diverse, exhibiting high lexical diversity.

By comparing the images from two data sources, namely public datasets and our MM-Retinal V2 dataset, we observe that MM-Retinal V2 exhibits a high degree of similarity with the public ones and almost covers all the common retinal disease categories. However, the textual content of the public datasets contains simple expansions of category labels based on fixed text templates, while the texts in MM-Retinal V2 are extensive and lexically diverse. Textual discrepancies reveal where knowledge dissemination is most needed. Consequently, to address the first obstacle, the image-guided texts of the MM-Retinal V2 will be regarded as the source for knowledge extraction, and the textual content of public datasets as the destination where knowledge is injected. This allows the knowledge from the MM-Retinal V2 to act as a spark, spreading and enriching the public datasets, and jointly contribute to the pretraining of KeepFIT V2.

We next investigate how to extract the expert knowledge from MM-Retinal V2. As mentioned above, MM-Retinal V2 shows minimal domain differences with public datasets and encompasses their fundus disease categories, making it ideal for knowledge extraction. Hence, we propose a semantics-oriented expert knowledge extraction module based on multi-head cross attention[35] to equip the model with high-level semantic retrieval ability. In particular, we perform visual matching on the images from two data sources and weight the corresponding text from the MM-Retinal V2 as expert knowledge based on the matching score.

Given the image feature $V_{p}$, $V_{m}$ from image encoder $E_{v}$ and text feature $T_{p}$, $T_{m}$ from text encoder $E_{t}$, we input the image features of public datasets as $query$ of cross attention, the image features of the MM-Retinal V2 as $key$, and the text features of the MM-Retinal V2 as $value$. Subsequently, matrix multiplication followed by the softmax function is applied to compute the attention weights $\Psi^{h}$ for head $h\in[H]$, which represent similarity scores between images:

where $p$ symbolizes public datasets that only have category labels and $m$ indicates MM-Retinal V2 dataset, and $W_{Q}^{h}$, $W_{K}^{h}$ are learnable projection matrices.

Next, similarity scores are used to reweight the text features of MM-Retinal V2, assigning different levels of attention to the text features based on the semantic similarity between image features. The semantics-level expert knowledge $EK_{s}$ from MM-Retinal V2 can be formulated as:

where $W_{V}^{h}$ and $W_{O}$ are learnable projection matrices.

When images from the public datasets and the MM-Retinal V2 demonstrate high similarity, their corresponding texts are expected to align more closely. As a result, higher similarity scores lead to greater text retention from MM-Retinal V2, thereby enriching the public datasets with relevant and complementary textual knowledge in a semantic visual matching way.

In addition to high-level semantic visual matching, we further explore matching the visual features between public datasets and MM-Retinal V2 in a low-level appearance way. Detailed appearance features can be derived from image generative learning. Therefore, a combination of semantic representation from contrastive learning and appearance representation from generative learning is a complementary way to achieve this goal. Specifically, we employ a vector quantization (VQ) approach to converting semantic visual features into discrete tokens for appearance-oriented knowledge extraction.

Fully representing continuous retinal features from the image encoder of KeepFIT V2 by quantized discrete tokens is challenging, as these retinal image features are highly semantic and contain limited low-level visual information like detailed lesion appearance. To address this, image tokenization in appearance-oriented knowledge extraction is inspired by index backpropagation quantization (IBQ) [32], which leverages a large-scale, high-dimensional codebook with efficient utilization. Moreover, IBQ allows the joint optimization of all codebook embeddings, effectively preventing codebook collapse. In this work, the codebook is trained in advance, following [32], using the same training data as KeepFIT V2, which includes images from MM-Retinal V2 and public retinal datasets.

As illustrated in the lower right of Fig. 3, after obtaining the image features $V$ from the image encoder, these image features comprise two parts, with the first part being the flattened features processed through the projector $P_{v}$ and the second part being unflattened feature maps bypassing the projector. The visual feature maps are first projected by a convolutional layer to achieve dimensional consistency. Then a quantization process is performed to tokenize the contiguous visual feature maps into discrete tokens using a fixed codebook $C\in\mathbb{R}^{K\times D}$, where $K$ is the codebook size and $D$ is the code dimension. First, the dot product between the visual feature and all code embeddings $C_{k}$ is calculated and followed by the softmax function and one-hot function to obtain probabilities:

Afterward, the gradients of soft one-hot distribution are transferred to hard one-hot index:

where $sg[\cdot]$ means stop-gradient operation.

After acquiring the index, the discrete code $\mathcal{Q}$ obtained by IBQ is:

Similarly, multi-head cross-attention is leveraged for knowledge extraction in an appearance-oriented way. The quantized vector $\mathcal{Q}_{p}$ from the public datasets is used as the $query$, $\mathcal{Q}_{m}$ from the MM-Retinal V2 as the $key$, and the text feature $T_{m}$ from the MM-Retinal V2 as the $value$. Consequently, Eqs. (5) and (6) are changed into:

The final extracted appearance-level expert knowledge $EK_{a}$ is computed by:

where $\sigma_{\tau}$ denotes the softmax function with temperature $\tau$.

The hybrid expert knowledge extraction module empowers KeepFIT V2 with both global semantic representation ability from contrastive learning and local appearance representation ability from generative learning, which is one of the major improvements compared to KeepFIT V1. Through this module, the cross-modality alignment capability and representation capability of KeepFIT V2 are significantly enhanced, resulting in more precise visual matching and knowledge injection.

The last significant problem is how to inject the obtained knowledge into the text of the public datasets to assist vision-language pretraining. Considering that the primary distinction between the public datasets and MM-Retinal V2 lies in the depth and granularity of their texts, we propose expert knowledge refinement loss $L_{EK_{s}}$ for semantics-oriented knowledge refinement and $L_{EK_{a}}$ for appearance-oriented knowledge refinement. These losses encourage the text of the public datasets to closely resemble the text extracted from MM-Retinal V2 that corresponds to their image features. To achieve this, the mean squared error (MSE) Loss is utilized as the basis:

The above formulas use the knowledge extracted at two levels to refine the text of the public datasets, creating a complementary and synergistic effect that makes the text refinement more comprehensive.

As depicted in the Fig. 3, the pretrained codebook is frozen, and we optimize the parameters of the image encoder, text encoder, and hybrid knowledge extraction modules, simultaneously. Finally, the overall training objective is defined as:

where $\lambda_{1}$ and $\lambda_{2}$ are hyperparameters, which are set to 100 and $1\times 10^{4}$ in our implementation, achieving the best performance.

KeepFIT V2 in CFP modality is trained on the proposed image-text MM-Retinal V2 and the categorical public retinal datasets from flair[33] which consist of over 190K images across 96 categories. Besides, we additionally collect over 80K data samples from public categorical retinal datasets. Using all 270K public datasets along with MM-Retinal V2, we trained KeepFIT V2_L. For evaluation, we utilize REFUGE[24], ODIR200$\times$3[34], iChallenge-AMD[9], Retina[1], FIVES[16] and APTOS[2] to perform various downstream classification tasks. These evaluation datasets encompass several common fundus diseases, including glaucoma, pathologic myopia, cataract, diabetic retinopathy, age-related macular degeneration, and other retinal disorders.

The pretraining for FFA modality is conducted on MM-Retinal V2 and FFA-IR[21], an image-text paired dataset comprising 10,790 reports and 1,048,584 FFA images, spanning 46 retinal lesion categories. We extensively evaluate KeepFIT V2 using two FFA public datasets. MPOS[36] includes 600 images across four fundus disease categories. AngioReport (APTOS2023)[44] contains a total of over 50K images, covering 24 distinct categories.

For pretraining, besides MM-Retinal V2, we also collect eleven OCT public datasets with only category labels for KeepFIT V2 pretraining, totaling over 181K images. In addition, OCTID[11] and OCTDL[17] datasets are used for evaluation, consisting of 9 common fundus diseases.

Table I reports the zero-shot experimental results on four unseen downstream datasets in terms of ACC, AUC, and AUPR. In addition, ODIR200$\times$3 includes two unseen categories during training, which are pathologic myopia and cataract. From an overall perspective, KeepFIT V2 and KeepFIT V2_L surpass the KeepFIT V1 on all the downstream datasets, achieving an average score improvement of up to 3.4%, 2.9%, 3.7%, and 4.2% on REFUGE, ODIR200$\times$3, Retina, and iChallenge-AMD, respectively. With the expansion of training data volume, though KeepFIT V2_L shows a slight performance decline on REFUGE and Retina, it achieves an improvement of 2.9% on ODIR200$\times$3 and 2.1% on iChallenge-AMD when compared to KeepFIT V2, indicating that a larger volume of training data can lead to performance gains.

It is worth noting that ViLRef, RET-CLIP, and RetiZero were trained on private datasets of 200K to 450K image-text pairs, which have not been publicly released. This may explain the performance gap between VLP models trained on large-scale and small-scale image-text paired data. Nonetheless, the KeepFIT V2 model consistently outperforms all comparison models on the REFUGE dataset, achieving improvements of 22.6%, 4.3%, and 24.0%, respectively. These results highlight that, even with a modest high-quality image-text dataset, KeepFIT V2 could effectively incorporate expert knowledge from MM-Retinal V2 into foundational vision-language pertaining, by spreading such an elite spark on public retinal datasets, demonstrating its exceptional generalization capabilities.

Next, we assess the performance of the proposed KeepFIT V2 in low-data regimes by conducting few-shot classification experiments. These experiments are derived by varying the number of shots (images per category) used for adaptation with the utility of Clip-Adapter [10] and Tip-Adapter [42]. From Table II, KeepFIT V2 and KeepFIT V2_L consistently outperform KeepFIT V1, improving the average score of ACC, AUC and AUPR metrics with a maximum improvement of 3.7% on REFUGE, 1.1% on ODIR 200$\times$3, 1.2% on Retina, and 2.2% on iChallenge-AMD when compared to KeepFIT V1.

Our KeepFIT V2 and KeepFIT V2_L achieve top-2 performance on the REFUGE and iChallenge-AMD datasets, as well as top-3 performance on the ODIR200$\times$3 dataset. Notably, they not only outperform FLAIR and KeepFIT V1 but also surpass models that are trained on large-scale image-text paired datasets. These results point to the superior representation and vision-language alignment capability of KeepFIT V2, underscoring its generalizability under limited image-text data resource.

To further evaluate the effectiveness and transferability of the proposed KeepFIT V2, we conduct linear probing experiments. Concretely, we use the frozen image encoder from KeepFIT V2 as image feature extractor and incorporate an additional linear layer as the classifier, whose parameters are fine-tuned on downstream datasets.

The experiment results are presented in Fig. 4. Compared to VLPs trained on small-scale elite image-text paired data / public categorical data (i.e. FLAIR, RETFound, and KeepFIT V1), our KeepFIT V2 and KeepFIT V2_L achieve significant performance improvement across all metrics. Furthermore, when compared to VLPs trained on large-scale image-text paired data (i.e., ViLRef, RET-CLIP, and RetiZero), KeepFIT V2 and KeepFIT V2_L still maintain comparable or even superior performance. From the above experiments in CFP modality, we can conclude that no foundation models can perform the best across all datasets in every evaluation setting. This indicates that building a retinal foundation model that performs well across diverse scenarios and various disease types is highly challenging and calls for further exploration.