GCS-M3VLT: Guided Context Self-Attention based Multi-modal Medical Vision Language Transformer for Retinal Image Captioning

The Vision Encoder converts pre-processed retinal images into visual features using a Convolutional base and a Guided Context Attention mechanism. The Convolutional base, employing EfficientNetV2B0 [21], extracts initial spatial features from images. Given a retinal scan image $X_{R}$ of dimensions $(356\times 356\times 3)$, Convolutional base produces visual features $F_{R}$ of dimensions $(12\times 12\times 1280)$.

The language encoder transforms processed diagnostic keywords into attention embeddings. It employs the Embedding layer to convert the processed diagnostic keyword sequence into embeddings, denoted as $KE$, encapsulating individual keyword embeddings $ke_{1},ke_{2},...,ke_{n}$. The encoder captures the semantic meaning of input keywords, but the generated embeddings initially lack explicit relationships between keywords, making them context-free. To overcome this limitation, the embeddings undergo further processing by a Multi-Head Attention. This layer employs a scaled-dot product attention mechanism, wherein three matrices are computed: Queries $(Q=KE)$, Keys $(K=KE)$, and Values $(V=KE)$. These matrices are subsequently processed to obtain context-enriched keyword embeddings $(KE_{c})$ that encapsulate meaningful relationships between the keywords .

Equations 4 to 6 show the computation of attention for each head in the MHA mechanism, with 5 detailing the Self-Attention scores and 6 defining the concatenation of attention outputs from all heads. In these operations, $KE$ represents the input embeddings; $W_{q_{i}}$, $W_{k_{i}}$, and $W_{v_{i}}$ are the weight matrices associated with the keyword embeddings for $Q$, $K$, and $V$ in the $i$-th head, respectively; $\sqrt{d_{k}}$ is the length of $K$; $W_{o}$ is the output weight matrix and, $h$ denotes the number of heads. The final embeddings are derived from the fusion of the input embedding and context embedding. This fusion is achieved by element-wise addition and normalization, represented by the equation 7 where $\oplus$ denotes the element-wise addition operation.

The Vision and Language encoders generate intermediate features within their domains, and the TransFusion Encoder (TFE) attentively fuses keyword context embeddings $(KE_{final})$ with visual representations $(F_{gca})$ for comprehensive clinical understanding. This fusion enables the cross-modality interactions necessary for accurate caption generation. Within the TFE, we implement a Vision-Language Cross-Attention (VLA) mechanism using Multi-Head Attention. The VLA selectively incorporates features from both domains, ensuring that critical clinical context is preserved and emphasized. This mechanism dynamically adjusts the contributions of visual and textual features, assigning higher weights to informative elements while suppressing irrelevant or redundant information, thus enhancing the relevance of generated captions. The VLA mechanism is formalized in Equation 8:

Subsequent operations, including Residual Addition, Layer Normalization (LN), and a Feed Forward Neural Network (FFNN) involving ReLU $(\Gamma)$, with parameters $W_{h}$ and $W_{r}$, refine the attention representations, resulting in $F^{{}^{\prime}}$.

The final component of GCSM3VLT is the Language Generation Decoder, which translates the fused multi-modal features into clinically meaningful textual descriptions. This decoder is based on a Transformer (GPT-2) [23, 24] that incorporates both the original visual representations and the integrated Vision-Language features generated by the TransFusion Encoder. By leveraging a sequence of multi-head attention layers, the decoder constructs a contextualized representation of the input modalities, which is then used to predict the next word in the sequence until the entire caption is generated.

To evaluate GCSM3VLT, we use the DeepEyeNet dataset [5] comprising 15,710 retinal images across various modalities: Fluorescein Angiography, Fundus Photography, Optical Coherence Tomography, and multi-modality grids. Annotated by expert ophthalmologists with 609 unique diagnostic keywords and clinical descriptions averaging 5-10 words. Covering 265 retinal diseases, from common to rare cases, the dataset is split into training (60%), validation (20%), and testing (20%) subsets, offering a robust foundation for high-precision diagnostic model development. All Images were resized to $(356\times 356)$ pixels with three channels, and text was cleaned and standardized, with keywords limited to 5-50 words, captions to 50 words, and rare words replaced by $<$UNK$>$. The vocabulary was capped at 5000 tokens, with 1024-dimensional embeddings. The model was trained for 100 epochs (loss: cross_entropy, optimizer: Adam, batch size: 64, learning rate: 0.0001) on an NVIDIA P100 GPU. Performance was evaluated using BLEU, CIDEr, and ROUGE metrics.

Table I highlights the impressive performance of our GCS-M3VLT, which surpasses state-of-the-art Vision-Language Models (VLMs) across all evaluation metrics. Our model achieves the highest BLEU scores, outshining leading models such as DeepOpht, Contextualized Keywords, and VisionGPT. It also excels in CIDEr and ROUGE, showcasing its superior ability to generate precise and contextually relevant medical captions compared to VLMs like Gated Contextual Attention Net and M3 Transformer. Additionally, the GCS-M3VLT’s lightweight design, with fewer parameters, contrasts favorably with heavier models like Expert Transformer, LlaVA-Med, and VisionGPT. Its Guided Context Self-Attention mechanism proves to be more effective than Non-local and Gated Contextual Attention, emphasizing its enhanced vision-language integration.

In Figure 4, we illustrate the input images, heatmaps, and overlays of attention maps, visualized through GradCAM [26], when the input images consist of a grid of multiple scans with the same or different modalities. For instance, we present scenarios where the input includes OCT scans, fundus images, and AF images, either individually or in combination. Our model robustly captures clinical context and lesion-specific information, even when the input comprises a grid of multiple images.

Figure 3 illustrates that our GCS-M3VLT model provides accurate and contextually rich medical captions for retinal images and keywords. For various retinal conditions, including suprachoroidal hemorrhage, subretinal neovascularization (SRNV), and Goldmann-Favre syndrome, our model’s predictions closely match actual clinical descriptions, effectively capturing key diagnostic details and clinical context. Compared to existing methods [14, 11], GCS-M3VLT consistently demonstrates superior accuracy in reflecting relevant retinal pathologies and patient details, highlighting its potential for enhancing clinical image interpretation and diagnosis.