Automated Detection of Myopic Maculopathy in MMAC 2023: Achievements in Classification, Segmentation, and Spherical Equivalent Prediction

The MMAC dataset is an extensive collection of color fundus images dedicated to research on myopic maculopathy. The dataset comprises fundus images gathered from various patients diagnosed with and without myopic maculopathy.
The sizes of each split of each task are summarized in Table 1. To maintain the challenge’s integrity and fairness, both the validation and test datasets are securely held and not released to participants. The validation and testing phases are executed on the organizer’s side to ensure unbiased evaluation.

The first Task (Classification of Myopic Maculopathy) is focused on a five-category image classification. The categories are as follows:

• •

  Category 0: No macular lesions

• •

  Category 1: Tessellated fundus

• •

  Category 2: Diffuse chorioretinal atrophy

• •

  Category 3: Patchy chorioretinal atrophy

• •

  Category 4: Macular atrophy

The annotations were meticulously generated and reviewed manually by professional ophthalmologists. Two ophthalmologists annotated each image independently. In cases of discrepancies in labeling, a third senior ophthalmologist provided the final label. This rigorous process ensures the high quality and reliability of the dataset annotations.

The second Task (Segmentation of Myopic Maculopathy Plus Lesions) aims to segment three types of lesions:

• •

  Lacquer Cracks (LC)

• •

  Choroidal Neovascularization (CNV)

• •

  Fuchs Spot (FS)

An ophthalmologist first performed the lesion annotations. A second ophthalmologist refined these annotations in consultation with the first. Both these ophthalmologists have over five years of experience. A senior ophthalmologist with a decade of experience in ophthalmology reviewed and finalized the annotations.

For the third Task (Prediction of Spherical Equivalent), the true value of the spherical equivalent (SE) was ascertained using the corneal curvature computer refractometer TOPCON KR-8900. The SE was computed as:

where $S$ and $C$ are the spherical and cylinder diopter, respectively; these values were acquired through the computer refractometer.

Several models were trialed for the classification task. These models included Resnet (18 and 50) [9], ViT [5], and Swin [22], among others. However, optimal results were achieved using a pipeline based on contrastive learning. This approach has recently gained traction for its ability to learn expressive representations from unlabeled data [33][32]. Our implementation can be detailed as follows:

Segmentation of myopic maculopathy plus lesions in MMAC2023 is intended to detect pixel-level lesions, including LC, CNV, and FS. Table 1 illustrates that the datasets required for different lesion segmentation tasks are different. As a result, it is difficult to obtain a unified segmentation model through multi-task learning in order to segment different lesions. Three independent segmentation models were used for this purpose. For the purpose of achieving optimal lesion segmentation, a data augmentation strategy has been proposed for the training of models on small datasets while backbone selection is performed. The TTA strategy was also incorporated to enhance the model’s robustness.

The prediction of spherical equivalent can assist in diagnosing the risk of myopic maculopathy associated with increased degrees of myopia [30]. Considering that there is a limited number of images in the training set and a limited number of submissions in the validation phase, we proposed the workflow shown in Figure 2. The following steps were tested for the prediction of spherical equivalent:

1. (1)

   Backbone Selection
   

   To determine which backbone is most effective for the prediction task, we used five-fold cross-validation on the training set to assess various backbones’ overall performance. In order to ensure a balanced data distribution, we sorted the data according to spherical equivalent values from smallest to largest. The training set of the challenge was split into three sets using five-fold cross-validation: an internal training set (3 folds), an internal validation set (1 fold), and an internal test set (1 fold).
   

   Testing was performed on VGG [27], ResNet [10], DenseNet [12], EfficientNet [28], and EfficientNetV2 [29] architectures. The internal training set was used for training the model, the checkpoint was selected based on the R-Square value of the internal validation set, and the mean of the R-Square on the internal test of Split(0-4) was calculated to represent its overall performance. The list of data augmentation strategies is described in Table 2.
   

2. (2)

   Model Re-training & Ensemble
   

   In order to maximize the use of the training set data, we resplit the training set after the backbone testing. In this step, we used one fold of the internal test set from the previous step as the internal validation set and the remaining four folds as the new internal training set. The backbone that performs well is trained using the internal training set, while the checkpoints selected from the internal validation set are used to test the results on the challenge validation set. We got different models by training with different splits, and we used model ensemble [18] to further boost the performance by averaging their outputs. The performance of TTA during inference was also tested.

The challenge comprised three diverse tasks, each with a set of unique requirements and evaluation parameters. Due to the inherent differences in the nature of these tasks, specialized evaluation metrics were formulated and employed for each. For task 1, the adopted evaluation indicators are Quadratic-weighted Kappa (QWK), F1 score, and Specificity. For task 2, the Dice Similarity Coefficient (DSC) is used to indicate the degree of coincidence of lesion segmentation. Finally, for the third task, the coefficient of determination R-squared, and the Mean Absolute Error (MAE) are employed to evaluate label regression’s degree of correlation and distance.

In terms of computational infrastructure, the models and algorithms were implemented on a high-performance machine boasting 196 GB of RAM. The Nvidia GPU Tesla V100s with 32GB memory and NVIDIA A6000 with 48GB memory were employed to facilitate the intense calculations demanded by deep learning tasks. The software ecosystem was primarily built around PyTorch, a leading deep learning framework. Additional libraries like Timm (known for its efficient training routines and pre-trained models), Segmentation Models PyTorch (SMP) for the segmentation task, and Lightly for contrastive learning were incorporated to provide a robust and efficient system for the challenge’s requirements.

Table 4 provides a brief description of our operators and detailed parameters for training. Unless otherwise specified, all experiments are conducted using reported configurations and parameters.

Among the backbones we tested for classifying myopic maculopathy, ResNet50 delivered the most promising results, as presented in Table6. This superior performance prompted us to delve deeper into optimizing ResNet50 further. During this optimization, we integrated SimCLR as a Pretext task. The added value of SimCLR to our pipeline was evident, as demonstrated by the improved results compared to Resnet50 without a pretext task. This suggests that SimCLR effectively captures enhanced representations from unlabeled data, thus enriching our model’s features and subsequently elevating its performance. Following this, the model was fine-tuned on the Task 1 dataset.
Further extending the model’s capabilities, we experimented with several ensemble strategies:

• •

  Mean: The strategy involved selecting the classifier that showcased the best mean performance across the three metrics based on the validation set during training.

• •

  All: This method calculated the mean of the logits outputs of each model before the Argmax operation, capturing a holistic insight from all models.

• •

  Majority: A majority voting approach, this strategy collated predictions based on the predominant class predicted by all classifiers.

Upon integrating ensemble strategies with ResNet50 optimized using SimCLR, we observed significant improvements in performance. The ’Majority’ ensemble strategy combined with SimCLR achieved the highest Macro_F1 score of 0.8176 and an overall score of 0.8881. Interestingly, the ’All’ strategy demonstrated the peak QWK (0.9080) and Specificity (0.9427), emphasizing its capacity to capture comprehensive insights from the models. The ensemble method ’Mean’ also displayed commendable results, with an overall score reaching 0.8781 when combined with SimCLR. It’s evident from the data that the addition of ensemble strategies, particularly in tandem with SimCLR, significantly boosts the model’s performance across various metrics.

As requested by the organizers, we submitted our four best-performing versions of our solutions for the final test, with these models being evaluated on a new, unseen dataset. Referring to Table6, we notice a slight decline in scores from validation to test phases, a common phenomenon due to the nuances of real-world data that may not be entirely captured in the validation set. The highest test scores were achieved using ResNet50 with the Majority ensemble method and TTA. This consistency from validation to test suggests that our methods are robust and not merely overfitting to the validation set.

During the validation phase of the MMAC Challenge, we tested the performance of different backbones and encoders on the validation set, as shown in Table 7. Based on the UNet++ structure, we compared the performance of ResNet and EfficientNet encoders. The ResNet34 encoder performed the best among them, and we used it to test the performance of other architectures. The segmentation of LC lesions using $U^{2}$-Net was found to be the most accurate, followed by segmentation using MAnet. MAnet demonstrated the best performance for segmenting both CNV lesions and FS lesions.

Based on the results of the backbone tests, we selected the four best-performing versions as the solutions submitted in the testing phase of the challenge, as shown in Table 8. The Ver. (1) combines the excellent performance of $U^{2}$-Net and MAnet on different segmentation tasks, thus performing well during the validation phase. Unfortunately, $U^{2}$-Net suffers from the overfitting problem due to the limited number of patients in datasets and therefore, performs poorly in the testing phase. Additionally, the tests indicate that the TTA approach significantly improves the robustness of the model and boosts the segmentation performance of the different submission versions in the testing phase. As a result, we found that the MAnet-based model with TTA strategy submission (Ver. (4)) performed optimally, with dice of 0.6838 for LC lesions segmentation, 0.6557 for CNV lesions segmentation, and 0.8348 for FS lesions segmentation on the test set. Figure 3 illustrates the performance of our MAnet-based model on different lesion segmentation tasks. Our model demonstrates proficiency in segmenting smaller lesions, as depicted in Figure 3(a), but it shows limitations in accurately segmenting larger or more complex lesions, as observed in Figure 3(b).

In order to evaluate the overall performance of different backbones on the internal test set, we first tested our proposed five-fold cross-validation method as shown in Table 9. In light of the mean values of R-Squared across different Split internal test sets, we selected three backbones that provided a better performance: tf_efficientnetv2_l, tf_efficientnetv2_xl, and tf_efficientnet_b8. As the organizer’s Python Packages currently do not support the current implementation of tf_efficientnetv2_xl, we chose to use tf_efficientnetv2_l and tf_efficientnet_b8 as backbones.

The dataset was resplit and then tf_efficientnetv2_l and tf_efficientnet_b8 were trained. As shown in Table 10, some models that performed well on the new internal validation set were evaluated on the validation set of the competition. Based on the results, it can be seen that the tf_efficientnet_b8 model obtained by Split’1 training and the tf_efficientnetv2_l model obtained by Split’2 / Split’3 / Split’4 perform well on the validation set. These models are then ensembled. Our model ensemble methods performed well in both the validation and testing phases, improving prediction accuracy without overfitting. As a result of ensemble tf_efficientnetv2_l (Split’ 3) and tf_efficientnetv2_l (Split’ 4), the solution Ver. (3) obtained an R2 value of 0.8735 and an MAE value of 0.7080 on the test set.