Assessing the significance of longitudinal data in Alzheimer’s Disease forecasting

All participants used in this work are from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database [18].

Participants are chosen based on their clinical diagnosis status. Specifically, we include individuals who have not been diagnosed with clinical AD at their baseline visit. Additionally, these participants must have undergone at least one follow-up diagnostic assessment after this baseline visit. We exclude CN baseline subjects who converted to AD because they are very few (n=9). In addition, we exclude CN baseline participants who converted to MCI before reverting to CN (n=41), and MCI baseline participants who were diagnosed as CN in a later follow-up year (n=284) since these subjects might have been diagnosed incorrectly at some point. Therefore, we focus solely on CN baseline subjects who progress to MCI and MCI baseline subjects who advance to AD, reflecting the irreversible progression of Alzheimer’s disease. After the exclusions, we are left with 1404 participants. Table 1 lists summary statistics for the participants.

The data, typical of many longitudinal studies, includes missing follow-up visits, irregular timings, and subject dropouts before study completion. Table 2 details the number of subjects in each diagnostic group for annual follow-ups. In all analyses, including Table 2, subjects who progressed to a later stage (CN to MCI, or MCI to AD) before dropping out are assumed to remain at the advanced stage. Stable subjects not present in a specific follow-up year are excluded from training and testing for that year.

We use the clinical data and biomarkers as our input features. Clinical data includes subject demographics, genotype (the number of APOE4 alleles, specifically), cognitive test scores, and diagnosis (CN or MCI). The complete list of demographic features and cognitive tests can be found in  [17]. The biomarkers are Cerebrospinal Fluid (CSF) measurements and Magnetic Resonance Imaging (MRI) volume measurements [9] (computed using the FreeSurfer software [6]). In the ADNI study, subjects undergo multiple cognitive assessments and MRI scans across visits, providing a detailed longitudinal medical history. The demographic and genotype variables remain static, and CSF is not collected at every visit. ADNI is divided into four phases (ADNI-1, ADNI-GO, ADNI-2, and ADNI-3), each with its own data acquisition protocol. There is considerable heterogeneity in the follow-up data collection, including irregular visit intervals and frequent missing visits. The extent of data missingness across various modalities for the 1404 participants in our study is detailed in Table 3.

Following [11], we begin the input preprocessing by recording the binary missingness mask for the feature set. Each participant has their own binary missingness mask indicating what variable was observed or not for that particular individual at a specific visit. Then, we perform mode/mean substitution for missing categorical/numerical features, respectively, following [2]. The categorical variables except the diagnosis are one-hot encoded, and numerical variables are z-score normalized in the last step of feature processing. To prevent any information leakage, both mode/mean substitution and z-score normalization statistics are collected from training data and used for training, validation and test data. Concatenating the input features, and the binary missingness mask yields a feature vector of length 113.

We focus on predicting an individual’s future diagnostic status, categorizing it into one of three classes: CN, MCI, or AD. This prediction is based on data obtained from preceding annual medical visits. The chronological nature of these visits forms the basis of our sequential input, with the goal of generating a corresponding classification prediction. To achieve this, we implement a transformer-based sequence-to-class neural network, LongForMAD, wherein each medical visit is represented as a distinct input token. The detailed architecture of LongForMAD is illustrated in Fig. 1.

Our model enhances each visit token by appending the prediction time horizon ($\Delta$t in Figure 1), defined as the interval between the visit’s data collection and the intended future prediction time point. Following [11], this increases the input token length to 114. This adaptation allows our model to make predictions over various future time horizons without the need to train distinct networks for each horizon. Instead, a singular, comprehensive network is trained across the entire dataset, covering all labels from subsequent follow-up years. In terms of network architecture, we employ a transformer encoder layer, a sequence pooling layer, and a fully connected classifier. Notably, our network integrates a standard learnable position encoding layer (initialized with 0 vectors) and a parameterized linear age encoding layer, as proposed by [16].

Following the addition of positional and age encodings to the input visit tokens, which are projected to the same latent space by a linear layer, the transformer encoder processes the input utilizing self-attention. This process enables the efficient identification and extraction of pertinent features from each token by analyzing the patient’s AD longitudinal data comprehensively. The sequence pooling layer, which applies average pooling along the token axis, then consolidates the information gathered from the entire longitudinal data of the patient. The aggregated feature set is subsequently passed through the fully connected classifier with softmax activation, which computes a diagnosis probability distribution, which represents the probabilities of CN, MCI, and AD diagnoses at the predicted future time point. Our network incorporates element-wise rectified linear units (ReLUs) as nonlinear activation functions between layers.

Transformers, despite their effectiveness in various tasks, are notably data intensive and prone to overfitting due to their extensive use of self-attention mechanisms, which require substantial data to learn effectively. We have designed a training strategy incorporating several techniques to mitigate overfitting and enhance model performance.

To begin, we expand our training and validation datasets. We consider every visit with a diagnosis of CN or MCI within the training data as a present-time reference point, denoted as present point. Visits with a confirmed AD diagnosis are not designated as present point in our process, as projecting their future is beyond the purview of this study. For each visit designated as present point, we compile the patient’s historical data, which may extend to a maximum of three years prior. Consequently, the longest sequence of longitudinal data used for training encompasses four visits: the visit occurring three years before present point, the visit occurring two years before present point, the visit one year before present point, and the visit at the point of present point. Subsequently, we record all subsequent follow-up diagnoses occurring in the five years succeeding present point. By constructing these 9-year progression trajectories, we effectively triple the volume of training and validation samples, diversifying the dataset. To prevent information leakage, we ensure that all 9-year progression trajectories originating from the same subject are exclusively allocated to one of the splits: either training, validation, or test set.

We leverage the inherent capability of transformer encoders to process input sequences of variable lengths, utilizing this feature to perform data augmentation on longitudinal history records. We perform the augmentation by omitting randomly selected past visits from the training data in each training epoch. Such a technique prompts the model to generalize its learning across the full spectrum of a patient’s longitudinal data, rather than narrowly adapting to particular visits, thereby reducing the potential for overfitting.

Table 2 reveals two imbalances: the changing class label distribution over time and the decreasing availability of clinical labels due to dropouts. The fluctuation in the number of stable CN baseline individuals across different years can be linked to varying monitoring policies in the ADNI study phases. To address these imbalances, we employ a loss re-weighting scheme during training, similar to [11], which assigns higher penalties to misclassifications in minority classes by using sample-level weights. We categorize participants into four groups for each follow-up year (CN baseline non-converters, CN baseline converters, MCI baseline non-converters, and MCI baseline converters) and adjust the weights to ensure balanced representation across all years. Without this mitigation, the model might prioritize predicting follow-up years and patient groups with the highest number of samples, neglecting the minorities. The weights for each sample point are calculated based on the expanded datasets.

We calculate a single training loss using the sample points generated by our augmentation strategy. However, for validation, we evaluate the validation loss across every possible scenario regarding the presence or absence of a visit in a history year, which amounts to $2^{4}-1=15$ different scenarios. These scenarios include cases such as whether the visit at present point occurred, whether there was a visit 1 year before present point, and so forth. To generate these scenarios, we simply exclude the relevant visits from the history of the validation subjects. We then compute the average of these loss values and use this average as the criterion for early stopping.

In evaluation, our primary goal is to assess the influence of the presence of patient longitudinal history on the predictions made by our model. It is important to highlight that our transformer-based network is designed to handle cases where a patient’s longitudinal history is not fully complete, meaning it does not necessarily include all past visits before present point. This enables the model to generate predictions even with partial or no historical data. We perform the analysis across various longitudinal data modality cases present in the input. Our model is capable of making predictions with missing features in the visits, as discussed in Section 2.2. To create those “cases", we synthetically replace unwanted features with mean/mode values derived from the training set and flip the corresponding entries in the missingness mask. We calculate our performance scores in two patient groups (CN at present point and MCI at present point) and every follow-up year separately. Therefore, each score analysis we conduct is bifurcated, one corresponding to the CN-to-MCI conversion task and the other to the MCI-to-AD conversion task.

ADNI, like many real-world longitudinal studies, is subject to biases in subject recruitment and follow-up. A notable issue is “temporal bias" [22], arising from the non-uniform distribution of visits across disease stages. Employing an inference strategy that mitigates temporal bias is crucial when evaluating model performance. To compute the performance scores for a particular patient group, follow-up year, longitudinal data modality case, and longitudinal history scenario, the process begins by randomly selecting a single present point for each test subject. This selection is made by choosing from all available instances of present points, with each selected instance ensuring that the subject belongs to the specific patient group of interest (CN at present point or MCI at present point). Furthermore, each chosen instance should be linked with a diagnosis in the designated follow-up year. It is important to highlight that within a pseudo test set, each subject contributes a single follow-up diagnosis to be predicted from a certain historical instance. Thus, all sample points in a pseudo test set are independent. Then, we obtain predictions by utilizing all relevant input features and longitudinal history information implied by the longitudinal data modality case and history scenario. We record the performance scores and repeat this operation for multiple random pseudo test sets. In the final stage, we calculate the average of the performance score values obtained from those pseudo test sets. This average represents the final comprehensive score for that specific patient group, follow-up year, and longitudinal history scenario pair. The purpose of repeating the operation for multiple pseudo test sets is to broaden the range of disease progression trajectories we use in our evaluation. By doing so, we aim to mitigate the potential effects of temporal bias on our results. We note that since our dataset expansion strategy utilizes every possible visit as present point during training, as discussed in Section 2.4, it not only expands the dataset size but also addresses temporal bias.

We employed a randomized, diagnosis-stratified approach to divide the data into training and testing sets with an 80-20 ratio. This division process was replicated 200 times, with the results presented herein being the average outcomes of these iterations. Within each split, we conducted a 5-fold cross-validation on the training set, and the validation loss was used to determine early stopping for model training. For each fold, five models were trained with different random seeds. The final predictions for each test scenario were obtained by averaging the outputs of 25 models, combining 5 cross-validation folds and 5 unique initializations. We used Adam optimizer [12] for training. To tune the architecture of LongForMAD, we employ a grid search strategy across each of the train/test splits. The optimal architecture for each test set is determined based on the hyperparameter values that achieve the highest performance on a corresponding validation set. The details of fixed hyperparameters, along with the optimization grid used for the adjustable hyperparameters, are documented on our github repository. For the calculation of performance scores, we utilize 100 random pseudo test sets. Due to the unbalanced follow-up diagnoses in Table 2, we primarily use the area under the receiver operating characteristic curve (AUROC) to evaluate our model. Our analyses show nearly identical ROC curves for both one-versus-one and one-versus-rest approaches, indicating a low likelihood of CN subjects being predicted as AD and MCI subjects as CN. Therefore, we report the one-versus-rest analysis results, considering MCI as the positive class for CN-to-MCI conversions and AD for MCI-to-AD conversions.

A comparison of LongForMAD’s performance with other existing longitudinal models in the literature for MCI-to-AD conversion in the third follow-up year is presented in Table 4. It is important to note that our comparison is limited to this specific patient group and follow-up year because the models against which we can benchmark our performance are primarily focused on this particular scenario. We also emphasize that this comparison is made primarily to demonstrate that our modeling approach is sufficiently robust, justifying the analysis of its predictions.

We assess the impact of patient longitudinal history availability in two ways: We compare the performance of our model when patient history information is available starting from years -3, -2, -1, and 0 to investigate the impact of longitudinal history duration, year 0 being the present point. Then, by keeping the history duration constant at 2 years (i.e., using history starting from year -2), we analyze the effect of longitudinal history data collection frequency. We compare the model’s performance when data is collected annually (involving years -2, -1, and 0) versus biennially (involving years -2 and 0). We note that in the analysis of history duration, we maintain a constant data collection frequency of 1 visit per year.

As detailed in Section 2.2, the modalities that present a longitudinal history are cognitive test scores and MRI biomarkers. We assess the importance of patient longitudinal history across two time-varying (i.e., longitudinal) data modality “cases.” In the first case, referred as “MRI-only", we synthetically induce missingness in cognitive test scores for all patients, leaving MRI biomarkers as the sole time-varying modality, to assess the significance of historical MRI biomarkers. Then, for a comprehensive assessment of the impact of longitudinal history, we incorporate all available input features in the second scenario, referred as “complete-set" case.

We examine the effect of longitudinal history data collection frequency by maintaining a constant history duration of two years and comparing $\Delta$AUROC scores between scenarios of annual and biennial data collection against the scenario with no patient history. For the MRI-only case depicted in Figure 2, it’s evident that annual data collection offers a significantly greater improvement in predictive performance compared to biennial collection for both patient groups. Additionally, it is worth noting that the gain in performance with annual data collection is smaller in MCI-to-AD conversion when compared to CN-to-MCI conversion within the MRI-only case.

Figure 4 illustrates a similar comparison for the complete-set case. For the CN-to-MCI conversion, the trends are nearly identical to those observed in the MRI-only case, albeit with a smaller overall increase in performance. For the MCI-to-AD conversion, the differences between annual and biennial data collection frequencies are less stark, yet on average, annual data collection shows a slight advantage, particularly due to a notable improvement in the second follow-up year. The results demonstrate that both CN and MCI patient groups benefit from higher frequencies of historical data collection, with the CN group showing a more pronounced and consistent improvement in AUROC in both data modality cases. This again highlights that more sophisticated modeling, such as including longitudinal history, offers a larger boost for the harder CN-to-MCI conversion prediction task, as stated in Section 3.2.1.