In-vehicle alertness monitoring for older adults

Multiple approaches were used in order to detect drivers’ attentiveness using sensorial, physiological, behavioral, visual, and environmental modalities. Early methods considered the steering motion of vehicles as an indicator of drowsiness (Sahayadhas et al., 2012). More recent approaches used face monitoring and tracking to detect the alertness levels of drivers. These methods relied on extracting facial features related to yawning and eye closure (Park et al., 2016; Reddy et al., 2017; Smith et al., 2000). Additionally, approaches that measured physiological signals such as brain waves and heart rates were introduced in order to correlate them to states of drowsiness (Kokonozi et al., 2008).

Several studies have been conducted on the state of the driver’s eye based on a video surveillance system to detect the eyes and calculate their frequency of blinking to check the driver’s fatigue level (Massoz et al., 2018; Mandal et al., 2016; Al-Rahayfeh and Faezipour, 2013; Kurylyak et al., 2012). Some studies have used a cascade of classifiers to detect the ocular area’s rapid detection (Kurylyak et al., 2012). A Haar-like descriptor and an AdaBoost classification algorithm have been widely used for face and eye-tracking by using Percent Eye closure (PERCLOS) to evaluate driver tiredness (Lienhart and Maydt, 2002; Freund et al., 1996). PERCLOS evaluates the proportion of the total time that a drivers’ eyelids are $\geq$80% closed and reflects a slow closing of the eyelids rather than a standard blink (Wierwille et al., 1994).

Wahlstrom et al. developed a gaze detection system to determine the activity of drivers in real-time in order to send a warning in states of drowsiness (Wahlstrom et al., 2003). Gundgurti et al. extracted geometric features of the mouth in addition to features related to head movements to detect drowsiness effectively. However their method suffered in situations involving poor illumination and variations in the skin color (Gundgurti et al., 2013). Sigari et al. developed an efficient adaptive fuzzy system that derived features from the eyes and the face regions such as eye closure rates and head orientation to estimate the drivers’ fatigue level using video sequences captured in laboratory and real conditions (Sigari et al., 2013). Rahman et al. proposed a progressive haptic alerting system which detected the eyes state using Local Binary Pattern and warned the drivers using a vibration-like alert (Rahman et al., 2011). Jo et al. introduced a system to separate between drowsiness and distraction using the driver’s face direction (Jo et al., 2011). The system utilized template matching and appearance-based features to detect and validate the location of the eyes using a dataset collected from 22 subjects (Jo et al., 2011). Taken together, these approaches faced challenges in practical adoption due to errors resulting from signal disambiguation, loss of facial tracking arising from sudden movements, and the invasiveness of signal extraction using contact-based sensors.

The detection of facial features (also called landmarks) is an essential part of many face monitoring systems. Eyes, as one of the most salient facial features reflecting individuals’ affective states and focus of attention (Song et al., 2014), have become one of the most remarkable information sources in the face. Eye tracking serves as the first step in order to get glance behaviour, which is of most interest because it is a good indicator of the direction of the driver’s attention (Devi and Bajaj, 2008). Glance behaviour can be used to detect both visual and cognitive distraction (Fernández et al., 2016). It has also been used by many studies as an indicator of distraction while driving and has been evaluated in numerous ways (Fernández et al., 2016). Therefore, both eye detection and tracking form the basis for further analysis to get glance behaviour, which can be used for both cognitive and visual distraction.

A typical face processing scheme in alertness monitoring systems involves the following steps:

• •

  Face detection and head tracking: In many cases a face detection algorithm is used as a face tracking one. In other cases, a face detection algorithm is used as an input for a more robust face tracking algorithm. When the tracking is lost, a face detection call is usually involved.

• •

  Localization of facial features (e.g., eyes). Facial landmarks localization is usually performed, but it should be noted that, in some cases, no specific landmarks are localized. So, in such cases, estimation of specific cues are extracted based on anthropometric measures from both face and head.

In general, “detection” processes are machine-learning based classifications that classify between object or non-object images. For example, whether a picture has a face on it or not, and where the face is if it does. To classify, we need a classifier.

Our implementation is based on the Viola-Jones algorithm, which has made object detection practically feasible in real-world applications (Vikram and Padmavathi, 2017). The Viola-Jones algorithm contains three main ideas that make it possible to build and run in real-time: the integral image, classifier learning with AdaBoost, and the attentional cascade structure (Heydarzadeh and Haghighat, 2011). This framework is used to create state-of-the-art detectors (e.g., face detectors (Fan et al., 2012)), such as in the Opencv library. However, cascade detectors work well on frontal faces but sometimes, they fail to detect profile or partially occluded faces. Thus, we used the Histogram of Oriented Gradients (HOG), which is a feature descriptor used in computer vision and image processing for the purpose of object detection. This approach can be trained with fewer images and faster (Surasak et al., 2018).

The majority of methods for alertness detection in the last decade have utilized deep networks in one form or another. For extracting the intermediate representation, algorithms consist of proprietary software (Wang and Xu, 2016; Liang et al., 2019; François et al., 2016), or a pre-trained CNN (Shih and Hsu, 2016) such as the VGG-16 (Simonyan and Zisserman, 2014). For characterizing alertness, models consist of logistic regression (Liang et al., 2019), artificial neural network (ANN) (Wang and Xu, 2016), support vector machine (SVM) (Cristianini et al., 2000), hidden Markov model (HMM) (Weng et al., 2016), Multi-Timescale by CNN (Massoz et al., 2018), long-short term memory (LSTM) network smoothed by a temporal CNN (Shih and Hsu, 2016), or end-to-end 3D-CNN (Huynh et al., 2016) are conducted.

For the purpose of our use case, the Level 5 CAV was assumed to be personally owned, which meant that we expected that parameters could be personalized to one traveler. The vehicle was expected to be a mid-sized sedan or larger, such as an SUV or minivan. Because the design study is for SAE level 5 vehicles, we expected that travelers will sit in the vehicle in the same way as we sit in the front passenger seat in manually-driven vehicles of today. Therefore, no steering wheels or other vehicle control devices would be in front of the traveler. Instead, the main features of the vehicle cab would be a dashboard, with an interactive display that presents trip relevant information to the traveler. Sensors to monitor the traveler would be integrated into the dashboard or the seat, and finally feedback from the driver monitoring system would be shown on a display located on the dashboard.

In the list below, we discuss design variables specific to traveler alertness monitoring.

1. (1)

   Sensor

   1. (a)

      Type: Alertness is a user state that can be operationalized in several ways, for example, as alertness versus sleepiness, or as situational attentiveness, or as focusing (foveating) on a key region of interest. For each of these definitions, there is a range of sensor types to achieve the goal. In this study, we chose visible light cameras as our primary traveler monitoring sensor for attentivenessWe selected a standard webcam (Logitech C270 HD Webcam) as it is a low-cost sensor which can additionally be used for head pose detection and facial landmark tracking.

   2. (b)

      Placement: The placement of the sensor within the vehicle must be done in such a way that it provides a natural advantage for recording, while at the same time minimally occluding the traveler’s experience. We experimented with three configurations of camera placement and three distances from the traveler.

2. (2)

   Traveler

   1. (a)

      Expectations from the traveler: This design variable refers to the extent to which the designer and developer can expect the user to cooperate with requests to align themselves to a tracking box, or submit to a calibration procedure, or sit still while a baseline is being established, or respond to audiovisual alerts, etc.

   2. (b)

      Characteristics of the user: Older people have different physical characteristics compared to younger people, which can impact the outcome of a traveler monitoring system. For example, older adults will have wrinkles around the eye and are more likely to be wearing glasses. Other characteristics include the distribution of activities older adults perform, which can change the distribution of head and body poses the traveler monitoring system will receive compared to young users. Furthermore, there is some evidence that older adults with MCI may have different eye movements than healthy individuals (Nie et al., 2020; Wilcockson et al., 2019; Molitor et al., 2015). There are also likely to be changes in the distribution of activities, attentional focus, and lucidity for older adults with MCI.

3. (3)

   Data

   1. (a)

      Raw data: This variable refers to the data that is recorded from the traveler. In our study, raw data consists of RGB video recorded at 24fps. No audio is required for alertness detection.

   2. (b)

      Features: This variable refers to how the raw data is processed, the time scale over which data is smoothed or otherwise preprocessed to reduce noise, and the extent to which features of interest are passed up the software stack. For example, video data may be processed to extract facial landmark points, which are sent to one computational sub-routine, and to extract the subset of pixels that comprise the eye region and mouth area, which are sent to a different computational sub-routine. Raw data may be secured on the vehicle while only specific features are permitted to be passed upstream to protect user privacy.

4. (4)

   Context: In addition to the features extracted from the primary sensor, other data may be used in the decision making process. For example, for alertness monitoring, the predictions from the video channel could be checked against a physiological sensor to rule out false alarms as a result of lighting and shadow.

5. (5)

   Intelligence: Data recorded by one or more sensors needs to be fused for reliable predictions. In addition, this variable refers to the extent to which the underlying algorithm incorporates computational mechanisms to adapt to the primary traveler (see note on personally owned vehicle) or adopts a best fit (one-size-fits-all) approach.

6. (6)

   Feedback: This design variable considers the extent to which the system is permitted to ask the traveler to reorient themselves for facial tracking, as well as the types of cues that the system will provide to the traveler, for example, visual and audio prompts to assess if the traveler simply dozed off or needs medical assistance can be provided by the vehicle.

7. (7)

   Privacy: Because the traveler alertness monitoring system will record highly personal data, there is a trade-off between local processing versus cloud-based processing in terms of user privacy and algorithmic improvements which leverage the cloud-based computational back end. For example, a system where all facial landmark features are processed locally, and no face videos are stored, would address privacy concerns related with out of context use of the facial video stream.

8. (8)

   Security: Because the goal of an alertness monitoring system is to escalate the alerts if the traveler does not respond to a nudge, the design and implementation could create a novel attack space from the cybersecurity perspective. For example, a malicious adversary might gain access to the the alert code to get the vehicle to repeatedly call the caregiver, or perform a man-in-the-middle attack on the camera feed causing the vehicle suddenly come to a stop because it thinks the traveler needs medical attention.

The purpose of the proposed system is to gather implicit cues from the older adult traveler to assess their degree of alertness, which allows the automated vehicle to monitor the passenger’s state and act accordingly. Several aspects of the traveler can be monitored and tracked such as: expressions, body positions, eye-tracking monitoring, physiological signals, etc. In our prototype, we have focused on alertness as our target population is susceptible to motion-induced sleepiness and distraction (Ren et al., 2019; Lin et al., 2016). Discussions with the physicians on the team, the advisory board members, and driver rehabilitation specialists revealed that taking a nap may be considered entirely reasonable while on a long trip. However, it would be concerning for a short (¡20 minutes) trip if the older adult with MCI traveling alone displayed signs of low alertness.

Figure 1 shows the system diagram. The prototype system uses computer vision libraries to monitor features associated with low alertness, such as eye aspect ratio, mouth aspect ratio, and pupil circularity, by using a remote camera embedded within the vehicle’s dashboard. These features are combined to predict an overall level of alertness of the traveler in real time. Each frame of the camera feed goes through the following processing steps: contrast enhancement, face detection, feature extraction, and alertness level prediction. Figure 2 illustrates the dashboard of the prototype vehicle. There is a small 7” screen placed adjacent to a larger in-vehicle information system. The system plays a beep sound through a sound bar in the dashboard when it cannot detect the traveler’s face, to remind them to reposition back to the center of the screen. In addition to the auditory cue, the system provides visual cues as shown in Figure 3.

To account for individual differences in facial structure and the facial cues that indicate low alertness, the traveler monitoring system starts with a calibration step for each new traveler. During calibration, the traveler is asked to keep their face steady and centered on the screen in front of them for around 30 seconds. The algorithm assumes that these data contain exemplars of a high alertness state for this traveler. When the features deviate from this exemplar, the algorithm detects a loss in alertness level. The features are calculated at a frame-by-frame level. We average the features in ten consecutive frames. When the average features deviate from the exemplar capture in the calibration stage, the algorithm detects a loss in alertness. Examples of the detection result between alert and drowsy states are presented in Figure 4.

Detection accuracy is highly dependent on the quality of input to the feature detection algorithm. We tested different locations of the camera in a car aiming to capture the traveler’s front face. A front face will enable us to extract features we need to predict the traveler’s alertness level. Other than the camera, we also need to consider the location of visual displays in the dashboard, as the traveler will likely be looking at those displays.

To determine an optimal location of the camera for our system, we tested three different locations (In the middle of the dashboard - Figure 5.(a) and Figure 5.(d); In front of the traveler - Figure 5.(b) and Figure 5.(e); On the right side of the front windshield - Figure 5.(c) and Figure 5.(f)). We tested two locations for the screen (In the middle of the dashboard - Figure 5.(a), Figure 5.(b) and Figure 5.(c); In front of the traveler - Figure 5.(d), Figure 5.(e) and Figure 5.(f)). Table 1 shows the face detection rate in different camera and screen placement locations. Based on the percentage of successful face detections, the optimal location for both camera and screen is determined to be in front of the traveler.

To identify the optimal distance between users’ eyes and the camera for our system, we tested four distances (44.3 inches, 39.2 inches, 30.5 inches, 28.2 inches). We recorded videos at these four distances and analyzed the facial landmark detection rate for each video. We recorded two videos for each distance while the user was wearing or not wearing glasses. We converted the video to image frames. We extracted five frames per second. For all the videos, we followed the following procedure to mimic a traveler’s behaviors.

• •

  0-10s: preparation (we removed this part before analyzing the face detection rate)

• •

  10-40s: looking ahead

• •

  40-70s: looking at the main display (the laptop)

• •

  70-100s: using a phone

• •

  100-130s: looking at the front left passenger seat

• •

  130-160s: looking ahead

• •

  160-190s: looking at the main display

• •

  190-220s: using a phone

• •

  220-250s: looking at the front left passenger seat

Table 2 shows the face detection rate for all image frames of each video under different distances:

The result showed that the face detection rate increased when the camera was moved closer to the participant when they were not wearing glasses. When wearing glasses, the face detection rate generally increased as the camera was moved closer, but the accuracy dropped off at the closest distance (28.2 inches) suggesting that glasses may be interfering with face detection at this close range. Among the four tested distances, 30.5 inches appears to be the most optimal. Under this distance, we could detect facial landmarks for almost all frames (99.86%) for both wearing and not wearing glasses conditions. Cameras placed at closer distances may be impacted by glasses and it the cameras may also be more obvious to users leading to annoyance.

We used dlib’s pre-trained facial landmark to detect a face area and extract features. We selected four features from the previous literature to train our classification model (Sathasivam et al., 2020; Singh et al., 2018). We also tested the performance of classifiers trained using different combinations of these features.

For our training and test data, we used the Real-Life Drowsiness Dataset (UTA-RLDD) (Ghoddoosian et al., 2019) created specifically for detecting multi-stage drowsiness. The dataset consists of around 30 hours of videos of 60 unique participants. There were 51 men and 9 women, from different ethnicities (10 Caucasian, 5 non-white Hispanic, 30 Indo-Aryan and Dravidian, 8 Middle Eastern, and 7 East Asian) and ages (from 20 to 59 years old with a mean of 25 and standard deviation of 6). The subjects wore glasses in 21 of the 180 videos and had considerable facial hair in 72 out of the 180 videos. Videos were taken from roughly different angles in different real-life environments and backgrounds. Participants were instructed to take three videos of themselves by their phone/web camera (of any model or type) in three different drowsiness states, based on the Karolinska Sleep Scale (Shahid et al., 2011), for around 10 minutes each. All videos were recorded at such an angle that both eyes were visible, and the camera was placed within one arm length away from the subject. These instructions were used to make the videos similar to videos that would be obtained in a car, by phone placed in a phone holder on the dash of the car while driving. The proposed setup was to lay the phone against the display of their laptop while they were watching or reading something on their computer.

For each video, we used OpenCV to extract 1 frame per second starting at the 40 seconds mark until the end of the video. The end goal is to detect not only extreme and visible cases of drowsiness but allow our system to detect softer signals of drowsiness as well.

In this section, we reported the accuracy, F1-score, precision, and recall of the models we trained using the K-nearest neighbors algorithm to predict the drowsy state when using different sets of features. Table 3 shows the detailed information.

We extracted 13235 frames from the UTA-RLDD dataset. We used 9264 frames as the training data and the remaining 3971 frames as test data. Among the 9264 frames in the training dataset, 51% were extracted from the alert state videos, and 49% were extracted from the drowsy state videos. Among the 3971 frames in the test dataset, 45.5% were extracted from the alert state videos, and 54.5% were extracted from the drowsy state videos.

The result shows that MOE (the ratio of MAR divided by EAR) seems to be the most sensitive feature among the four features we tested. The result aligns with the fact that MOE captures the subtle changes in both EAR and MAR and will exaggerate the changes as the denominator and numerator move in opposite directions. Though the feature MAR is the least sensitive feature among the four features (accuracy=49.94%, F1 score=0.4), the highest accuracy (86.33%) and F1 score (0.86) were found when using the feature combination of MAR and MOE, followed by using the feature combination of EAR, MAR, and MOE (accuracy=85.72%, F1 score=0.86). The third highest accuracy was found using the feature combination of EAR and MOE (accuracy=85.63%, F1 score=0.85).

We also found that in some cases, individual features work better than a combination of features. For example, when using the feature combinations EAR+MAR, EAR+PUC, MAR+PUC, and EAR+MAR+PUC, the accuracy, and F1 score are lower than using these features individually.

We investigated the mean and standard deviation of the EAR, MAR, PUC, and MOE for alert and drowsy conditions. Table 4 shows that EAR and PUC are larger in the alert condition than in the drowsy condition on average, while the MAR is smaller in the alert condition than in the drowsy state. The result also shows that the MOE is larger in the drowsy condition than in the alert condition on average. Moreover, the difference in MOE’s mean value in the alert and the drowsy condition is larger than other than three features. This aligns with the fact that MOE could exaggerate the changes of EAR and MAR.

We tested our algorithm on older adult participants to evaluate how well we could detect drowsiness in older adults. Older adults were recruited from the general population. Ten participants agreed to participate in three Zoom sessions, one in the morning, one around noon, and one in the evening. These times were selected so as to record videos of participants’ faces when they were alert and drowsy. In the Zoom meeting, participants were asked to watch a car driving video ¹¹1https://www.youtube.com/watch?v=fkps18H3SXY. Participants were instructed to feel free to talk or look around as they usually do when traveling as a passenger. Participants were asked to report their sleepiness level based on the Karonlinksa Sleep Scale (Shahid et al., 2011) before and after watching the car driving video. Figure 10 shows the detailed study procedure. Table 6 shows the meeting time with participants and their self-reported sleepiness level.

We collected 33 videos from 10 older adults without MCI. Table 6 shows the percentage of frames that the traveler monitoring system could detect the participant’s facial landmark. The results show that the facial landmark detection rate for most participants is higher than 90% except for participants 7, 8, and 9. The main reasons that the results were low for these three participants are that (1) their video resolution was low and (2) their face was out of the viewport of the camera. To address these concerns, we used a higher-resolution Logitech c920 webcam in our final prototype system while placing the camera directly in front of the user to capture their face during the entire process; thus, ensuring our system can detect greater than 90% of facial landmarks.

Participant 2 reported they were extremely alert in the first session, alert in the second session, very alert in the third session, and sleepy in the fourth session. Based on their reported subjective alertness level, their alertness level is highest in session A and lowest in session D. Table 5 shows the mean and standard deviation of the EAR, MAR, PUC, and MOE for that participant. The average EAR and PUC are the highest, while the average MOE is the lowest in session A. The average MAR and MOE are the highest, while the average PUC is the lowest in session D. The bolded values in table 5 show that the MOE features can distinguish the different alertness levels, and that MOE is a good feature to predict the alertness level.