Undersampling and Cumulative Class Re-decision Methods to Improve Detection of Agitation in People with Dementia

Our previous work completed the collection of data using the multimodal sensor framework for detecting agitation and aggression in PwD installed at a Specialized Dementia Unit at the Toronto Rehabilitation Institute (SDU-TRI), Canada khan2017daad . The framework included wearable Empatica E4 wristbands and 15 cameras installed in public areas. We recruited 20 participants after obtaining the informed consents from their substitute decision-makers. The duration of each participant’s study was set to a maximum of two months as per the ethics approval.

This paper focuses on using Empatica E4 data to detect agitation including acceleration, blood volume pulse, electrodermal activity and skin temperature. This type of multimodal data has been previously used for stress detection van2020standardized , daily activity identification pietila2017evaluation , and other clinical applications wang2015low ; park2019smart . In our study, Empatica E4 was put on the participant’s dominant hand in the morning and removed at night for charging and backup on the following day.

The labeling mechanism was manually implemented. Nurses in SDU-TRI were trained to record participants’ behaviours and the approximate start and end times of agitation events in their nursing charts. The researchers further fine-tuned the agitation start and end using the video recordings of cameras. However, in certain cases the manual labeling mechanism would inevitably lead to inaccuracy or even lack of start and end times, which was a challenge to be evaluated in this study. These situations could happen when the PwDs were in private areas of SDU-TRI or video feeds were unavailable. Ultimately, 3 participants didn’t show any agitation behaviours throughout the collection, so our studies were based on the data from 17 participants. The participants demographic information is shown in Table 1, where agitation day is defined as a day on which at least one agitation event occurred, and each agitation event is fully annotated with both start and end times. Similarly, a “normal” day is defined as a day without agitation events.

Data processing and feature extraction adopted the same processes as described by Spasojevic et al. spasojevic2021pilot . All sensors were resampled to 64 Hz and the high-frequency noises were removed by a first-order Butterworth low-pass filter. In this study, we used a combination of 67 features and 1-minute sliding window, which was found to be most effective window length in this dataset. The windows were non-overlapped to prevents information leakage during training and testing classifiers. To further verify the influence of the amount of data on the classifier training, the data from both agitation days and “normal” days were combined to form a dataset. The results by Spasojevic et al. spasojevic2021pilot were conducted on only agitation days, whereas we showed new results on the larger combined dataset. In addition, it is worth mentioning that we found three labeling oversights from previous implementation spasojevic2021pilot , including,

• •

  Agitations that occurred during the wristbands were being put on or taken off were all assigned normal label;

• •

  Data collected earlier than 10:00 a.m. were all assigned normal label; and

• •

  Agitations that lasted less than or equal to the length of sliding window were incorrectly assigned normal label.

We have made corrections so that the assigned labels are consistent with the nursing charts. The overall size of the dataset in this paper is 205,427 rows over 67 features. In the supporting supplementary document, the data sizes cross participants are reported in Table S1, and the features are listed in Table S2 along with their definitions.

We further discussed the decision-making process of agitation detection. Compared with using the default empirical threshold ($Th$, for example: $Th=0.5$ for random forests), we explored the relationship between different thresholds and the corresponding binary classification performance. Inspired by the continuity of agitation, we proposed a postprocessing method named CCR. The class label of each instance were decided by the classifier’s output and former instances in a decision window as well. The idea of CCR is illustrated in Figure 3. The specific rules were implemented through equations (4) - (6).

where $S_{i}$ is classifier score of instance $i$, $f\left(\cdot\right)$ is the interim decision function, $L_{i}^{temp}$ is the interim label of instance $i$ decided by the given threshed $Th$, $flag_{i}$ represents the vote of instances in the $win$-minute decision window, $L_{i}$ is the final label decision, $g\left(\cdot\right)$ is the final decision function, decided by $flag_{i}$. CCR firstly generates label reference of current instance based on the score of the trained classifier and takes account of the historic information of instances in the past $win$ minutes. According to the continuity characteristic of agitation, current instance is reported to be agitation when more than half of instances in the decision window are detected as agitation by the classifier. On the contrary, current instance tends to be normal if all the instances in the decision window stayed normal. Since the average duration of agitation is 8.62 minutes, the length $win$ is set to be 5 by ceiling the half of the average duration. CCR does not peek into any future information to make class decision for current instance. To the best of our knowledge, it is the first time to incorporate historical data information to develop decision-making to detect agitation using multimodal sensors.

RUS and AEF were separately applied to generate the rebuilt training sets. RFC models were trained on the two new training sets and obtained optimal parameters respectively. Eventually, the optimal classifiers were tested on the same test set for fair comparison. The processes are shown in Figure 4. The proportions of selection for RUS and AEF are listed in Table 2. Among them, 1.3% made equal size of normal and agitation set used for training. The proportions of AEF were calculated corresponding to each $k$ in (2) and the AUROC is reported respectively. The random seed 1 was kept constant to ensure that the cross-validation partitions were consistent under different settings. For the RUS method, we repeated 5 times of experiments by changing Random Seed 2 and reported the average AUROC in Figure 5. The IQR analysis was shown by boxplots for better comparison when AUROC was close to the results of RFC without RUS, which was equivalent to 100% selection proportion for RUS. The average training time is shown in Table 3.

It can be seen from Figure 5 that using RFC with RUS, even if only 20% of the normal instances were retained for training, the classifier still performed equivalently as compared to RFC without RUS. Especially when the selection proportion was between 30% to 70%, the performance of the classifier was even higher. The reason is that the rebuilt training set not only ensured the variation diversity of the normal activities, but also potentially mitigated the problem of imbalance to a certain extent. Table 3 shows the training time of RFC with RUS is linearly related to selection proportion, indicating that an appropriate lower number of normal instances can reduce the time cost of training and improve the classifier’s performance by addressing the data imbalance problem. However, the performance of AEF is worse compared to RUS as the dotted curve is under the blue curve except when the proportion is 98.7%. It implicitly shows that the number of mislabeled instances in our dataset is quite small. While AE tried to filter out more mislabeled instances, it eliminated normal instances with potential similarity, which reduced the diversity of normal activities and caused underfitting. In short, to obtain better performance of RFC requires adequate size of the dataset and rich diversity of instances to improve the model training. When the diversity of the dataset stays unchanged, increasing its size only increases the imbalance and time cost.

To validate the idea above, we compared the results with that of spasojevic2021pilot , where the normal and agitation sets were obtained from agitation days only. By setting the selection proportion as 0.4 in RUS, we adjusted the size of our dataset to be the same as that of spasojevic2021pilot , as well as the ratio of normal set size to agitation set size. We also applied RUS method to our test set ensuring the number of test instances was equal to that of spasojevic2021pilot for comparison. The AUROC was 0.865 for spasojevic2021pilot , while our setting increased the AUROC to 0.880. The only difference between two settings was our dataset included a richer diversity of normal behaviours from “normal” days, which also showed the idea that both diversity and quantity of dataset were important.

To further evaluate the impacts of potential mislabeled instances in ATI before and after the known agitations, we replaced the RUS with WRUS and followed the rest of experiment steps shown in Figure 4. The proportion of selection was set to [30%:70%] with separation of 10%. The AUROC results are shown in Table 4. The better results in comparison are bold-faced.

Table 4 shows that WRUS preprocessing gives slightly better results of RFC under most selection proportions compared to RUS. It indicates that there might exist a few mislabeled instances in ATI, however, the classifier is not significantly improved by reducing the contribution of those instances to training. In other word, our dataset is quite reliable for developing agitation detection algorithms.

The performance of CCR method was validated based on extended RUS experiment with 20% as selection proportion in Section 4.1. To quantify the performance of decision-making, precision ($P$), recall ($R$), and F1-score ($F1$), calculated by equations (7) - (9) respectively, were introduced. After obtaining the trained RFC model with RUS, we applied it on the test set and obtained the score of each instance. We firstly explored the relationship between thresholds and decision-making performance by adjusting different threshold values. Then, we followed steps (4) - (6) in Section 3.2 to remake decision and assign class label for each instance in test set in chronological order, only relying on present and past information. It is worth noting that reporting performance on the test set is considered possible as it does not involve tuning hyperparameters or training the classifier. The curves of F1-score versus thresholds between [0.10,0.80] with separation of 0.01 are plotted in Figure 6 for both RUS without CCR and RUS with CCR methods.

Table 5 shows part of metrics when the threshold is between [0.10,0.80] with separation of 0.10, while 0.43 is the threshold that gives the best F1-score for both RUS with CCR and without CCR. The best F1-score after applying CCR increases by 26.6% comparing $F1$ with underlines in Table 5. Figure 6 shows the curves of F1-score first rise and then drop after their peaks as threshold increases. It means that having high precision and recall values at the same time is a challenge in this application because a larger threshold makes more instances classified as normal by the decision equation (4), resulting in smaller false positive (FP) and larger false negative (FN). However, these results should be treated with caution as the overall F1-score obtained with CCR are still deemed low to be used in clinical practice. The low precision values at optimal threshold indicated the presence of large amounts of false positives. A clinical evaluation may be necessary to understand these false positives and how they can impact the clinical burden.