Multi-Subset Approach to Early Sepsis Prediction

Some of the research done in sepsis prediction uses simple traditional machine learning models. For example, Zabihiet et al. [13] used a wrapper feature selection algorithm based on XGBoost to extract five different sets of features from clinical data to predict sepsis 6 hours before onset. Both valid and missing clinical data are used to derive the relevant attributes. Afterward, an ensemble model comprised of five XGBoost models is utilized to predict sepsis. In addition, Firoozabadi et al. [14] created models to predict sepsis as part of the PhysioNet/CinC Challenge 2019. In this study, the authors took hourly samples of 40 features from each patient’s data from three different ICUs. They processed the data to remove outliers, fill in missing data, and replace the remainder with the population average. They discovered similarly that an ensemble of bagged decision trees is effective for early sepsis prediction 6 hours before onset. A similar observation is shown by Fu et al. [15].

However, these models did not consider that the temporal change trend can be helpful for early sepsis onset prediction, and missing 6 hours of progressing information is challenging to make an accurate prediction. In this work, we address both by adding temporal change trend features and proposing a multi-subset approach.

Deep learning models have also been applied to address specific challenges of sepsis prediction. For example, Gilbertson et al. [16] used the Principal Component Analysis (PCA) to reduce the dimensionality of patient data before applying a simple RNN. The PCA method reduced the original 40 patient features to 10 main components. In addition to this, a fast Sequential Organ Failure Assessment (qSOFA) score generates one additional characteristic. These 11 characteristics are then loaded into a DNN classifier. However, the characteristics connected with each hour are examined independently. Later, Tsang et al. [17] applied LSTM which allows consideration of past time-step data embeddings within the prediction of the current time-step to capture the temporal pattern. Most recently, Shah et al. [18] proposed a DNN model to predict sepsis 6 hours in advance. The DNN model is used because of its feature learning capacity and its property of approximating functions.

Better sepsis onset prediction performances were observed using these DNNs. However, none of them has considered addressing the 6-hour gap challenge, which is explored in this paper. Moreover, training DNNs is often time, resource, and data expensive, which is avoided in this work.

In 2019 and 2021, Brekke et al. [19] and Mahta et al. [20] respectively proved that the temporal trend of vital signs and white blood cells could improve the performance of deterioration prediction by studying the retrospective cohort. In the machine learning area, Orphanou et al. [21] employed Temporal Association Rules (TAR) combined with Naïve Bayes classifiers for coronary heart disease diagnosis. They showed that higher levels of temporal abstraction improved the prediction performance when long sequences and distant events are critical.

In the sepsis-related prediction area, Ghosh et al. [22] adapted temporal patterns resulting from the contrast pattern mining approach in conjunction with Coupled Hidden Markov Model (CHMM) to predict septic shock 30-60 minutes prior to onset. This paper demonstrated that this performance is significantly higher than applying SVM directly on data or using CHMM with continuous multivariate data. Nonetheless, the 30-60 minutes prediction window is too narrow to let the doctor react.

We denote the ground truth^§ for a sepsis incident as $\hat{y}$ that has an indication of onset hour $t$. More than often, the hourly physiological patient data contain missing values, which are imputed and filled in. In order to train the model to predict the onset of a sepsis incident $h$ hours into the future, for each patient, we first shift the values in $\hat{y}$ earlier by $h$ hours and call this $\hat{y_{h}}$. This step creates a new target feature that will occur in $h$ hours. The leftover values on the table with no ground truth values are dropped from the data. This step is illustrated in Table I when we set $h=3$. From this example, we can observe that if we want to predict sepsis event onset in 3 hours, using observed patient data up to $t=1$, the onset prediction for $t=4$ should be 0, and using data up to $t=2$, the prediction of the onset of a sepsis incident for $t=5$ should be 1.

After shifting the label for $h$ hours to predict the onset of an incident in $h$ hours, for a certain hour $t$, we can use only the $x_{t,f}$ values across all of the features, and the new ground truth label $\hat{y_{h}}$ to train a classification model for the onset of a sepsis incident that will be able to make a prediction on new patients and provide the probability of the event in $h$ hours as $y_{h}$. This is illustrated as an example in Table II. Specifically, given the hard label of an incident, the trained model can provide the probability of the onset of such an event at each time stamp, which inspires our multi-subset approach.

To close the 6-hour gap of missing progressing information, we can include the probability of a sepsis incident in 3 hours as a new feature (i.e., one subset) to predict the incident in 6 hours in the target subset as shown in Table III. This procedure is also illustrated in Fig. 1. In a previous subset (e.g., Subset 1 in Fig. 1), we use the observed data up to the prediction time to determine the probabilities that a patient will experience a sepsis incident between the current time $t$ and the sepsis incident onset time $t+6$. For example, we can use the observed data to determine the probability of the onset of a sepsis incident at $t+3$. Then, this probability will be treated as an additional input feature in the target subset (e.g., Subset 2 in Fig. 1) of our approach. After that, the large 6-hour gap can be shrunk using an in-between measure with a smaller gap of 3. We can further shrink the gap by providing sepsis incident probabilities of each hour in-between to feed more features in the target subset, which is using multiple subsets to close the 6-hour gap.

It should be noted that the same absolute value for different patients can have totally different meanings, while changes over time can share the same indication. Therefore, we use delta values to capture the changes in feature values over time, which can provide valuable temporal patterns for sepsis prediction. The observation window size as shown in Fig. 1, denoted by $w$, is an important parameter that determines the duration of time over which the delta values are computed.

To compute the delta values, we create a new feature, denoted by $f^{\prime}$, which is computed as the difference between the value of $f$ at timestamp $t$ and the value of $f$ at the previous timestamp $t-1$ within the observation window $w$. In other words, the value of $f^{\prime}$ at timestamp $t$ is calculated as $x_{t,f^{\prime}}=x_{t,f}-x_{t-1,f}$. This computation is repeated for each feature and each timestamp within the observation window. For example, if the heart rate at timestamp 11 is 70 beats per minute (bpm) and the heart rate at timestamp 10 is 75 bpm, the delta value for the heart rate at timestamp 11 would be -5 bpm.

We leverage delta values to capture the temporal change trend within the observation window $w$. We can also incorporate statistical information computed from a feature $f$ within the same time window to describe the feature distribution. Specifically, for each feature $f$ within the observation window $w$, we can calculate six statistical values based on the sequence $S=[x_{t-w,f},x_{t-w+1,f},x_{t-w+2,f},...,x_{t,f}]$: the mean, minimum, maximum, standard deviation, skewness, and kurtosis. Variance, which is the expectation of the squared deviation, measures how spread out the data is, while skewness is a measure of the asymmetry of the variable’s probability distribution, and kurtosis measures the tailedness of the probability distribution. The formulas for calculating these statistical values are as follows:

• •

  Mean value: $\overline{x}=\frac{1}{N}\sum_{i=1}^{N}x_{i}$

• •

  Standard deviation: $\sigma=\sqrt{\frac{1}{N}\sum_{n=1}^{N}(x_{i}-\overline{x})^{2}}$

• •

  Kurtosis: $\text{K}=\frac{1}{N}\sum_{i=1}^{N}\frac{(x_{i}-\overline{x})^{4}}{\sigma^{4}}$

• •

  Skewness: $\text{Sk}=\frac{1}{N}\sum_{i=1}^{N}\frac{(x_{i}-\overline{x})^{3}}{\sigma^{3}}$

By incorporating both delta values and statistical information, our model is better equipped to capture the temporal change trend for early sepsis prediction. The overall process is illustrated in Fig. 2. First, delta and/or statistic values are generated. Then, a 3-hour prediction is generated to address the prediction gap and added as a previous subset for the target 6-hour prediction.

To evaluate the proposed method, we use trauma patients’ data from UW Harborview Medical Center. It contains prospectively collected, de-identified data from injured adults ages 16 years and older, who were admitted to the ICU between 2012 to 2019 and required at least three days of invasive mechanical ventilation. This data includes physiological data on 2,802 patients, 486 (17%) were identified to have sepsis during the first 14 days of admission. The patient data is sampled at different time intervals. Vital signs, for example, were sampled hourly, whereas laboratory tests were sampled daily or less frequently. As such, there are some features with many missing values. We address missing values using carry-forward imputation.

To find which features would be more useful for early sepsis prediction, we set groups of features that share common attributes. They are vital signs, static profile, cumulative exposures, and laboratory results, where static profile includes initial physiology from the first 48 hours and patient factors, and cumulative exposures are the summary of events. These 4 groups are denoted as G1, G2, G3, and G4. The detailed list is shown in the Appendix.

One of the primary problems with the sepsis patient data we received is predicting an outcome that occurs in a minority of the study population using an hour-by-hour framework. This can be a problem because it leads to imbalanced classes in the data. In this case, the outcome of interest, which is a sepsis event, occurs in only 17% of the patients, making it a minority class. When there is a class imbalance in the data, it can lead to biased model performance. Approximately 1.84% of the hour-by-hour records in the data have a sepsis indicator, while the rest does not have it. Most of the patients admitted to the ICU never develop sepsis. Some patients develop sepsis within an hour or two, and some develop sepsis after a more extended period.

To account for the imbalanced nature of the data, we use the following methods. First, we only include hourly patient records from day 2 until day 14 of the patient’s stay in the ICU. This is because the scope of this study is patients developing sepsis after being in the ICU. Limiting the data will reduce the number of hourly records with no sepsis that does not need to be included. Second, if a patient has an hourly record with sepsis, we remove all the subsequent hourly records for that patient. For example, if we see a patient record with sepsis at 4:00, all observations after that time are removed. It is because we are only interested in the early sepsis onset prediction, that is, the first onset. In addition, to further account for the imbalance of data, we use a random oversampling of 0.8. To illustrate this further, suppose there are 20 minority samples and 1000 majority samples. Random oversampling duplicates the 20 minority samples without replacing them until there are 800 samples. As a result, there are 800 minority samples and 1000 majority samples. Finally, we use random under-sampling to make the number of minority class data the same as the number of majority class data in the sample.

To evaluate the performance of the proposed approach, we use metrics of Area Under the Receiver Operator Curve (AUROC), Sensitivity, and Specificity, where the average performance under 5-fold cross-validation is reported. Specifically, $Sensitivity=\frac{TP}{TP+FN}$, where $TP$ is the number of true positives and $FN$ is the number of false negatives, is defined as a measure of how well the model can recognize positive examples.

At the same time, $Specificity=\frac{TN}{FP+TN}$ is the fraction of real negatives properly detected by the model. Finally, AUROC is a useful metric for evaluating the performance of a binary classifier because it provides a single number that summarizes the classifier’s ability to discriminate between positive and negative samples over all possible decision thresholds.

Considering the feature selection property of XGBoost that can benefit from more features, we use the largest set of features (i.e., Group 1+2+3+4 with delta and statistic values in the observation window, which is set to $w=6$) to demonstrate the effect of our multi-subset approach. In the comparison, we use ‘1 Subset’ to denote the method using only the target subset, ‘2 Subsets’ to denote the method adding the probability of sepsis in 3 hours to the target subset, and ‘6 Subsets’ to denote the method adding probabilities of sepsis in 1 hour, 2 hours, 3 hours, 4 hours, and 5 hours to the target subset.

Table IV compares the performance for both the tasks of early sepsis prediction and early septic shock prediction. We can observe that our multi-subset approach is definitely helpful for the task of early septic shock prediction, where adding the probability of shock in 3 hours improves the performance compared to using only the target subset and adding more probabilities can further improve. However, this is not the case for early sepsis prediction. In the experiment, we observed that more features does not mean more choices of features for XGBoost that can help improve the performance. For the task of early sepsis prediction, we find that using only Group 1 (i.e., vital signs) and Group 2 (i.e., static profile) features are more helpful as shown in Table V. More importantly, we can observe the similar phenomenon that adding the probability of sepsis in 3 hours improves the performance compared to using only the target subset and adding more probabilities can further improve.

Fig. 3 further demonstrates the observation that for early sepsis prediction, certain features are not helpful and even harmful. In general, the model with more subsets performs better than the model that has only 1 subset. However, we find that the group combinations with G3 and G4 may affect the result lower than fewer subsets. For example, Fig. 3 shows the result of different group combinations with delta and stats using different subsets. Group 1+2 and Group 1+3 are getting better when adding subsets, but Group 1+3+4 is getting lower, and Group 1+2+3+4 is getting higher first, then getting lower. This is because that G4 has much more missing values other groups, which may mislead the model learning.

In this sub-section, we evaluate the effect of using delta, stats, or a combination of delta and stats in the prediction for sepsis and septic shock. We take the best feature group combination for each task with ‘6 Subsets’ to compare, that is, Group 1+2 for sepsis and Group 1+2+3+4 for septic shock, respectively.

Table VI shows the results. First, for sepsis prediction, we can observe that both AUROC and Sensitivity are increasing when incorporating delta and stats. This indicates that using delta and stats can help improve the detection performance of sepsis patients, while the baseline can be biased to non-sepsis patients with a way higher Specificity compared to the Sensitivity. This demonstrates that using temporal change trend is helpful for early sepsis prediction.

For the task of septic shock, we have a different observation. Although the combination of Delta and Stats provides better performance compared to using only one of them, the baseline without using any additional temporal trend features gives the best performance. In fact, this is a reasonable observation, since shock as a sudden event is more related to the most recent conditions of the patient, while sepsis can be more observable using the progressing status of infections. In this case, using multiple subsets to shrink the 6-hour gap is more effective.

Moreover, considering the out-performance of DNNs, we apply RNN to the temporal features and DNN to static features. Then, we concatenate them for sepsis and septic prediction using the same setup. However, DNN performs worse. The potential reason is that DNNs often require sufficient training data to be successful, while we have very limited amount of sepsis cases, resulting in models significantly biased to Specificity.

To further verify if these probabilities added from the previous subsets are helpful for making predictions, we apply LIME [23], a tool that attempts to explain the weights in a predictive model, and check the importance of features for sepsis onset prediction. Specifically, we take a random sample of the data and apply LIME to get the importance of each feature for both models of ‘2 Subsets’ and ‘6 Subsets’ shown in Table VII. The top-5 important features of each model are listed.

The results show that the probability of sepsis occurring in 3 hours contributed significantly to the prediction of sepsis onset for the method using 2 subsets. Similarly, the likelihood of sepsis occurring in 5 hours contributed significantly to the prediction for the method using 6 subsets. Moreover, the rest top features are delta values. This further demonstrates our proposal of shrinking the gap and using the temporal change trend.

G1 (Vital Signs)*:

• •

  Heart Rate (hr)

• •

  Diastolic Blood Pressure (dbp)

• •

  Mean Arterial Pressure (map)

• •

  Respiratory Rate (rr)

• •

  Temperature (temp)

• •

  FIO2 (fio2)

G2 (Static Profile):

• •

  Age

• •

  Sex (male or female)

• •

  Transfer from another hospital (transfer)

• •

  Mechanism (Injury Mechanism)

• •

  Head Injury

• •

  ED Systolic Blood Pressure (Initial.ED.SBPCat)

• •

  Reverse Shock Index (rSICat)

• •

  Max Base Deficit Cat (baseDef48Cat)

• •

  Lactate Category (lactate48Cat)

• •

  Red blood cell count (rbc48)

• •

  Crystalloids count (crys48Cat)

• •

  Apache

• •

  Antibiotics 48 (abx48)

• •

  Surgery 48 (surg48)

• •

  ER Disposition (er dispCat)

G3 (Cumulative Exposures):

• •

  IV Fluid BOLUS (bolusSum) *

• •

  Red Blood Cell Count (RBCsum)

• •

  Days in Ventilator (ventDaySum)

• •

  Days in Surgery (surgSum)

• •

  Hours in Surgery (surgHours)

G4 (Laboratory Result):

• •

  Bicarbonate (bicarb) *

• •

  Acidosis (acidosisCat)

• •

  Strong Ion (StrongIon) *

• •

  Blood Urea Nitrogen (bun) *

• •

  Creatinine (creatinine) *

• •

  White blood cell count (wbc)

• •

  Urine Output (uop)