Transformer-based Personalized Attention Mechanism for Medical Images with Clinical Records

Pathological diagnosis plays an important role in medicine. Various computer-aided diagnosis methods for pathological images have been developed for various problems, such as classification [13, 21, 9], tumor region identification [4, 5, 1], segmentation [34, 22, 31, 29], survival prediction [38, 33, 14], and similar image retrieval [12, 16, 10]. In digital pathology, a digital scan of the entire pathology specimen, called a WSI, is used as the target image. Because a WSI is usually huge (e.g., 100,000$\times$100,000 pixels), it cannot be directly fed into a model. Therefore, image patches extracted from the WSI are often used as the inputs to a model. In pathological diagnosis based on WSIs, it is important to note that WSIs contain both tumor cells and normal cells. Therefore, if there is no annotation of the tumor cell region, it is necessary to first identify the tumor cell region and then make a pathological diagnosis. This problem is a weakly supervised learning problem in the sense that only the WSI is labeled and the tumor cell region is not.

MIL is a weakly supervised learning problem in which labels are not given for instances but for a group of instances called a bag. In an MIL formulation of a binary classification problem, it is assumed that a positive bag contains at least one positive instance, whereas a negative bag contains only negative instances. By considering a WSI as a bag and an image patch as an instance, the subtype classification problem can be interpreted as an MIL problem in which class-specific image patches (e.g., tumor patches) are considered positive instances. Several MIL approaches have been developed for digital pathology tasks [15, 25, 18]. Among them, attention-based MIL [15, 9, 17] is particularly useful because the identified attention regions can be interpreted as class-specific image patches.

Although the development of deep learning techniques has dramatically improved the accuracies of many medical image analysis tasks, it is critically important for medical practice to develop techniques that provide explanation of the results. Various visualization methods, such as Grad-CAM [26], have been proposed to interpret and explain the rationale for classification results. Singla et al. [28] proposed a method that visualizes the medical image regions that serve as the basis for the classification of diseases for each concept derived from clinical report analysis. However, most visualization methods visualize the regions that contribute to the classification results after the classifier is applied to given images (rather than using clinical reports for finding and visualizing the image regions that contribute to the classification results, as in our proposed method). The attention-based MIL described above can also be considered as a method to provide explanation of the results because attention can be visualized as an informative region for making decisions. In this study, we employ the Transformer architecture [32] as a basis for estimating the attention region in medical images. Although the Transformer was originally developed for natural language processing (NLP) tasks, it has been demonstrated to be effective for general computer vision tasks [6], including medical image analysis [24, 27, 20, 7]. In particular, the Transformer has been effectively used to aggregate bag features in an MIL setting [27]. The Transformer architecture can encode the relationship among a pair of components in input data, e.g., between two words in the case of NLP tasks, and between two image patches in the case of computer vision tasks. This study was inspired by the use of the Transformer in the context of vision and language [37, 3], where it has been demonstrated that a change in the language token can change the attention of the image token. The Transformer has a mechanism to quantify the relevance of multimodal information in the form of attentions. The encoding mechanism for the relationship of input data can be expanded into multimodal input including image patches and table data, which enables computing attention from an image patch to table data, or attention from table data to an image patch. We expect that encoding the relationship between image patches and clinical factors can provide more appropriate personalized attentions based on the clinical record.

In clinical practice, doctors obtain additional information from patients’ clinical records and image diagnoses are performed by considering such clinical factors as prior information. In multimodal analysis, clinical records that include the basic information of patients and some examination results can often be used in addition to digital images [35, 36, 30, 23]. Yala et al. [35] used a combination of mammography images and patient data (basic information, medical history, etc.) and demonstrated that the performance of breast cancer risk prediction could be improved. Multimodal analysis of dermoscopic images and patient data (age, gender, and body location) has also been studied to enhance the accuracy of skin lesions classification and melanoma detection [36].

Multimodal analyses of medical images and clinical records have primarily been performed on radiology images, but recent works have reported that pathological images can also be combined with clinical records to improve the task performance. Li et al. [18] combined tabular clinical data with histological images in an MIL setting, where eighteen attributes, including age, genes, and tumor location, were used as inputs with multiscale histological images. Additional clinical factors have also been used in a mixture-of-experts model as inputs of a gating network [25]. Chen et al. [2] proposed a Transformer-based multimodal model for survival prediction using images and genetic data. Their method could visualize co-attentions between images and genetic information. However, these previous works primarily focused on performance improvement by using multimodal inputs, and detailed effects on attention regions by additional clinical factors have not been reported. The advantage of the proposed PersAM method is that it can provide personalized attention regions according to the clinical records, which mimics the actual pathological practice of human expert pathologists.

Our database of malignant lymphoma was composed of $N=842$ clinical cases with three subtypes: 277 DLBCL, 270 FL, and 295 reactive lymphoid hyperplasia (Reactive). Figure 5 shows sample image patches for typical DLBCL, FL, and Reactive cases. DLBCL has large tumor cells over a wide region in the tissue specimen, and FL has follicular structures which have tumor cells. In contrast, Reactive is classified as non-lymphoma, which has diverse cell structures but no tumor cells. All the patient data were clinically diagnosed by expert hematopathologists and a WSI of a hematoxylin-and-eosin (H&E)-stained tissue specimen and a clinical record were given for each case. A gigapixel digitized WSI of the entire H&E-stained tissue slide, was used as an input image $\mathbb{X}_{n}$. All the glass slides were digitized using a WSI scanner (Aperio GT 450; Leica Biosystems, Germany) at 40x magnification (0.26 µm/pixel), where the maximum image size was approximately 100,000$\times$100,000 pixels. The OpenSlide [8] software was used for handling WSIs and extracting image patches from $\mathbb{X}_{n}$. An original clinical record includes the definitive subtype $\mathbb{Y}_{n}$ and clinical factors $\mathbb{T}_{n}$. Note that we cannot use patch-level annotations, and the class label $\mathbb{Y}_{n}$ is given only to a WSI $\mathbb{X}_{n}$, not image patches. The clinical factors $\mathbb{T}_{n}$ consist of 28 elements that are summarized by $M=2$ clinical factors: an 18-dimensional vector with patient basic information and interview results and a 10-dimensional vector with blood test results. The details of items included in each clinical factor are listed in tables 1 and 2.

In the experiment, $224\times 224$-pixel image patches were randomly extracted from an entire WSI and 100 image patches were used as a bag due to the amount of computation and memory. The corresponding label $\mathbb{Y}_{n}$ was assigned to a bag generated from a WSI $\mathbb{X}_{n}$, e.g., a bag generated from image patches of a WSI of DLBCL was labeled as DLBCL. A maximum of 30 bags were generated from a single WSI in our experiment. The length of the feature vector was set to $R=512$ inspired by TransMIL [27]. To obtain feature vector of image patches $\bm{h}_{\ell}^{\mathrm{p}}$, $f$ employed an ResNet50 [11] pre-trained with ImageNet and a two-layer NN that had 1024 hidden units, 512 output units, and ReLU as its activation function, where a 2048-dimensional vector after global average pooling layer in ResNet50 was converted into the 512-dimensional feature vector. Clinical factors were mapped to $\bm{h}_{m}^{\mathrm{t}}$ by a two-layer NN that had 256 hidden units, 512 output units, and ReLU as its activation function, where both 18-dimensional and 10-dimensional clinical factors were converted into 512-dimensional feature vectors. Class tokens $\{\bm{h}_{c}^{\rm cls}\}_{c\in[C]}$ were designed as three 512-dimensional vectors, and then we could obtain input data for the Transformer architecture $\tilde{\bm{H}}\in\mathbb{R}^{512\times 105}$. In (15), to prevent underflow computing, $1-a_{\ell,c}^{\prime}$ was normalized to $[0.95,1.0]$. The dataset was divided into training, validation, and testing data in the ratio of 3:1:1, and the models were evaluated via five-fold cross-validation where the model that had the smallest validation loss after third epoch was used for testing.

For the setting on Transformer, the number of layers and heads were set to two and eight, respectively, where dropout rate was set to 0.1. The classifier $g_{\textrm{clf}}$ was an NN that had a hidden layer with 256 units and an output layer with three units to compute the class probability from a 512-dimensional aggregated feature vector $\bm{z}$.

For stability in the optimization, label smoothing was applied as a regularization technique in calculating the loss function $\mathcal{L}_{\textrm{bce}}(\mathbb{Y}_{n,c},\pi_{n,c})$, where the label for a correct class was set to $\mathbb{Y}_{n,c}=0.95$ and the labels for incorrect classes were set to $\mathbb{Y}_{n,c}=0.05$. As an optimization method, momentumSGD (nesterov, weight decay=$10^{-4}$) was employed and the training of the model was performed in nine epochs. Learning rates were determined as $10^{-4}$ for optimizing the parameter $\theta_{\mathrm{f}}$, $2\times 10^{-4}$ for the parameters $\theta_{\rm enc}$, $\theta_{\rm agg}$ and $\theta_{\rm clf}$, and $4\times 10^{-6}$ for the parameter $\theta_{\mathrm{g}}$ in which the learning rate was multiplied by 0.1 every three epochs. Random horizontal flip and random rotations ($0^{\circ},90^{\circ},180^{\circ},270^{\circ}$) were applied to the input image patches as the data augmentation. All the parameters of the model were simultaneously optimized in the above setting. It took about 20 hours to perform five-fold cross-validation by a computer with eight Quadro RTX 5000 (NVIDIA, U.S.). Our source code is available from https://github.com/PersAM-MIL/PersAM.

The proposed PersAM model was compared with the following baseline models:

1. 1.

   MLP using clinical factors as input (clinical MLP)
   Clinical MLP employs a three-layer NN that has hidden layers with 256 and 512 units and uses a 28-dimensional vector indicating clinical factors as input data. The training of clinical MLP was performed in 500 epochs, where the learning rate was set to $10^{-3}$ without scheduling.

2. 2.

   Attention-based MIL using images [15] (img MIL)
   Img MIL employs an attention-based MIL that aggregates 2048-dimensional feature vectors in a bag and predicts the class label from an aggregated feature vector using the classifier $g_{\textrm{clf}}$ with the hidden layer having 1024 units.

3. 3.

   Attention-based MIL using images and clinical factors (img-clinical MIL)
   In img-clinical MIL, in addition to a 512-dimensional aggregated feature vector computed from image patches by the attention-based MIL, a 28-dimensional clinical factor is also used as an input for computing the 512-dimensional feature. By concatenating the aggregated feature vector for images in a bag and the computed feature vector for clinical factors, the classifier $g_{\textrm{clf}}$ predicts the class using the 1024-dimensional concatenated feature vector through the hidden layer with 512 units.

4. 4.

   Transformer-based MIL using images (img Transformer)
   In img Transformer, only one class token was concatenated to feature vectors for $L$ image patches. The classifier $g_{\textrm{clf}}$ predicts the class using the encoded class token (it is a common technique in the Transformer-based classification model).

5. 5.

   Transformer-based MIL using images and clinical factors (img-clinical Transformer)
   Similar to img Transformer, img-clinical Transformer uses only one class token, and it is concatenated to feature vectors for $L$ image patches and $M$ clinical factors. Img-clinical Transformer predicts the class using the encoded class token as an input for the classifier $g_{\textrm{clf}}$.

The setting for an optimization method and learning rates are the same as above except for clinical MLP.

The classification results are shown in Table 3, where each row shows the mean accuracy and standard error in three-class classification by five-fold cross-validation. The results show that the proposed method achieved the highest accuracy compared to all the baseline methods. In particular, whereas the baseline methods using image and clinical factors showed low accuracy, our proposed method classified the subtype more accurately by properly aggregating image and clinical features through the multimodal aggregator.

We also performed visualization experiments to demonstrate that the proposed personalized attentions could be adaptively changed according to input clinical records. In the visualization results, attention weights ranging from zero to one were assigned in the range blue to red. Figs. 6 and 7 show the visualization results of class-wise attentions, exploratory attentions, and explanatory attentions, where the images on the right are thumbnails of the original WSIs. In the matrices on the left, the columns show the class-wise attentions $\{a_{\ell,c}\}_{(\ell,c)\in[L]\times[C]}$, the rows show exploratory attentions $\{\psi_{\ell}\}_{\ell\in[L]}$, and each element shows explanatory attentions $\{a^{\prime}_{\ell,c}\}_{(\ell,c)\in[L]\times[C]}$, where clinical records sampled from other cases of three different subtypes were input with the WSI of a patient instead of the original clinical factor of that patient. Fake clinical records were used to confirm that exploratory and explanatory attentions changed when different clinical records were input with the same WSI.

Fig. 6 shows the results for an FL case. It is known that the follicular structure, a subtype-specific region for FL, is important in the diagnosis of FL cases. This case has large follicular structures in the tissue, and we can confirm that exploratory attention $\{\psi_{\ell}\}_{\ell\in[L]}$, i.e., the follicular region on which focus is placed, changes depending on the clinical records. This case should have some difficulty in the diagnosis using only images, and explanatory attentions change according to the input clinical records.

Fig. 7 shows the result for a Reactive case. It is known that some Reactive cases have a similar appearance to FL cases. This case also has small follicular structures in the tissue, which is similar to FL, and exploratory attentions $\{\psi_{\ell}\}_{\ell\in[L]}$ are enhanced in those regions. Class-wise attentions for FL focus on the follicular regions and those for Reactive focus on the outside follicular regions. Detailed discussions for these results will be done with magnified image patches later.

The visualization results whose attentions were changed are observed for parts of the dataset, and not all cases changed their attentions depending on input clinical records. From the above results, it is expected that the changes of attentions are caused by whether the input WSI is pathologically typical or not; a case whose disease can clearly be determined only from a WSI does not change its attentions if a different clinical record is input together, and on the other hand, a case whose WSI has ambiguous features to identify the subtype has the possibility of changes of attentions depending on the input clinical record. To qualitatively confirm this, an expert hematopathologist (one of the authors, who is an institution member with over 15 years of experience diagnosing more than 10,000 cases of lymphoma) investigated whether each case was typical or not in both cases whose attention were changed and not. We targeted FL cases to easily interpret the observation results. The pathologist observed many WSIs of FL cases that changed attentions and did not change attentions when the different (fake) clinical records were input and evaluated their typicalness of FL with blind whether the attentions of each case changed or not. A case that was determined as FL only from a WSI was evaluated as a typical FL case, and a case that cannot be determined as FL only from a WSI and requires immunohistochemical (IHC) stains was evaluated as an atypical case. The results are discussed in §4.4.

We call self-attentions in the bag representation $\hat{\bm{H}}$ which indicates attentions from each clinical factor to image patches “clinical-record-to-patch attentions”. As an additional experiment, we visualized how clinical-record-to-patch attentions changed according to input clinical factors. Fig. 8 shows the visualization result of clinical-record-to-patch attentions for an FL case, where the clinical factor of an original (real) case is replaced with the those of representative (fake) case in the clustering results. Fake cases were used to confirm that clinical-record-to-patch attentions changed when different clinical records were input with the same WSI similarly to Figs. 6 and 7.

In the clustering, $k$-medoids method was applied to two-dimensional t-SNE embedded features that were calculated from 28-dimensional clinical factors. We determined the number of clusters by looking at t-SNE embedded features and set to $k=7$. Instead of the original clinical factors, the clinical factors of the representative cases in each cluster were input with a WSI of the original case into the PersAM model. The plots on the left represent the embedded clinical factors, in which $\star$ are the representative cases in each cluster. The images on the middle and right are the visualization results of clinical-record-to-patch attentions corresponding to blood test and interview, respectively. Attention weights of image patches are normalized in each case for visualization.

We can confirm that the proposed PersAM could adaptively change clinical-record-to-patch attentions depending on the clinical records that were input with an original WSI. The visualization results for the representative case 1, 2, 4, and 7 are similar to each other because the embedded clinical factors were located close, but the result for case 3 focuses on the follicular structures and the result for case 5 focuses on the outside follicular structures. We confirmed that clinical-record-to-patch attention also changed depending on clinical factors by effectively encoding the relationship between image patches and clinical factors.