AnxietyFaceTrack: A Smartphone-Based Non-Intrusive Approach for Detecting Social Anxiety Using Facial Features

Over the last five decades, researchers have studied nonverbal communication in mental disorders (Waxer, 1977; Argyle, 1978; Perez and Riggio, 2003; Foley and Gentile, 2010; Schneier et al., 2011; Gilboa-Schechtman and Shachar-Lavie, 2013; Weeks et al., 2019; Asher et al., 2020; Shatz et al., 2024). These studies have found that nonverbal cues can play a significant role in diagnosing mental disorders and contribute to therapeutic processes. For instance, nonverbal signs such as a patient’s appearance, behavior, and eye contact are routinely assessed during psychiatric evaluations as part of the mental status examination (Foley and Gentile, 2010).

Existing research has analyzed video recordings of individuals with anxiety disorders in various scenarios, such as therapy sessions, task performance, video watching, dyadic conversations, etc. One of the earliest studies on this topic was conducted by Waxer (Waxer, 1977), who analyzed the nonverbal cues of individuals with anxiety. In this study, anxious and non-anxious participants (20 participants: 5 anxious males, 5 non-anxious males, 5 anxious females, and 5 non-anxious females) were videotaped at different times during the admission period. One-minute silent session videos were then shared with 46 senior psychologists. They rated ten behavior cue areas on a 10-point scale ranging from “not anxious at all” to “highly anxious” and described how these features conveyed anxiety. The ten behavior cue areas were the forehead, eyebrows, eyelids, eyes, mouth, head angle, shoulder posture, arm position, torso position, and hands. Further, using Linear regression analysis, hands, eyes, mouth, and torso were identified as a key nonverbal indicator of anxiety.

A major focus of recent studies has been on gaze behavior and its relationship with social anxiety. Schneier et al. (Schneier et al., 2011) explored gaze avoidance in individuals with generalized social anxiety disorder, healthy controls, and undergraduate students. Their findings indicate that avoiding eye contact is associated with social anxiety. Similarly, Weeks et al. (Weeks et al., 2011, 2019) conducted multiple studies on behavioral submissiveness in social anxiety. In one study, participants engaged in a role-play task with unfamiliar individuals (confederates), revealing that body collapse and gaze avoidance are linked to social anxiety (Weeks et al., 2011). In another study, Weeks et al. used eye-tracking systems to examine participants watching positive and negative video clips, further identifying gaze avoidance as a prominent marker of SAD (Weeks et al., 2019).

Nonverbal synchrony is another area of investigation in SAD. Asher et al. (Asher et al., 2020) analyzed dyadic conversations between individuals with SAD and non-anxious individuals, finding impaired nonverbal synchrony among those with SAD. Similarly, Shatz et al. (Shatz et al., 2024) examined nonverbal synchrony during diagnostic interviews, showing that individuals with SAD displayed lower levels of synchrony and reduced pacing compared to non-anxious counterparts. An in-depth review by Gilboa et al. (Gilboa-Schechtman and Shachar-Lavie, 2013) provides a comprehensive understanding of nonverbal social cues in SAD, synthesizing findings from various studies and emphasizing the role of nonverbal behaviors in the disorder. The findings from these studies underscore the role of nonverbal cues, such as gaze behavior and body posture in understanding and diagnosing SAD. These studies relied on recorded videos and human inference, thus highlighting the need for an automated tool.

To the best of our knowledge, Cohn et al. (Cohn et al., 2009) were the first to explore the use of automated visual features from videos for research in mental health detection. They recorded interviews between clinically depressive participants and interviewers. The study used manual and automated facial analysis coding systems (FACS) as feature inputs for machine learning. They achieved accuracies of 88% with manual features and 79% with automated features in depression detection. Furthermore, ‘AVEC 2011 – The First International Audio/Visual Emotion Challenge’ introduced automated visual features, calculated using dense local appearance descriptors, for affective computing (Schuller et al., 2011). This was done through a workshop challenge that provided an open dataset to the research community. Later, the development of OpenFace¹¹1https://cmusatyalab.github.io/openface/, based on FaceNet (Schroff et al., 2015), an advanced deep learning model, offered a unified system for detecting facial features. Later, the updated version, OpenFace 2.0 (Baltrušaitis et al., 2016), emerged as a state-of-the-art computer vision toolkit. It enabled researchers to analyze facial behavior and study nonverbal communication without requiring comprehensive programming knowledge.

Most studies that use visual features for mental health detection have focused on depression and stress (Cohn et al., 2009; Schuller et al., 2011; Valstar et al., 2014; Ringeval et al., 2019, 2017; Valstar et al., 2016, 2013; Wang et al., 2021; Gavrilescu and Vizireanu, 2019; Grimm et al., 2022; Giannakakis et al., 2017; Sun et al., 2022), with limited attention given to anxiety (Wang et al., 2021; Gavrilescu and Vizireanu, 2019; Grimm et al., 2022; Mo et al., 2024; Giannakakis et al., 2017) and even less to SAD (Harshit et al., 2024; Shafique et al., 2022). Giannakakis et al. (Giannakakis et al., 2017) used an open-source model to detect the face region of interest and applied Active Appearance Models (AAM) for emotion recognition and facial expression analysis. In their study, participants were recorded with a video camera with extra lighting while undergoing three experimental phases: a social exposure phase, an emotion recall phase, and a stress/mental task phase. The computed features were then used to detect emotional states related to stress and anxiety. They achieved an average accuracy of 87.72% in stress detection across these phases. Similarly, Sun et al. (Sun et al., 2022) utilized visual features for remote stress detection. Participants attended an online meeting and self-reported their stress levels on a scale of 1 to 10, which served as the ground truth for a binary stress classifier. The study reported an accuracy of 70.00% using motion features (eye and head movements) and 73.75% using facial expressions (action units). In another study, Grimm et al. (Grimm et al., 2022) analyzed participants’ videos captured while they answered open-ended questions. A classifier was trained using GAD-7 scores as ground truth, achieving an area under the curve (AUC) score of 0.71 for the binary classification of anxiety characteristics. Similarly, Gavrilescu et al. (Gavrilescu and Vizireanu, 2019) predicted depression, anxiety, and stress using videos captured with high-end cameras while participants watched emotion-inducing clips. They achieved accuracies of 87.2% for depression, 77.9% for anxiety, and 90.2% for stress.

Existing studies on SAD have predominantly focused on eye gaze. In these studies, participants typically complete a performance task involving interviews or the Trier Social Stress Test (TSST). For example, Shafique et al. (Shafique et al., 2022) used participants’ eye gaze data captured during a 5-minute general conversation with an examiner, covering topics such as introductions, support, and conflict. Their method achieved an accuracy of 80% in detecting the severity of SAD. In another study, Harshit et al. (Harshit et al., 2024) analyzed participant’s eye gaze while they performed a speech task as part of the TSST. Using an autoencoder, they extracted latent feature representations (deep features) from the eye gaze data, which was used as features for machine learning models, and achieved 100% accuracy in detecting participants’ anxiety.

In summary, most existing studies focus on interview-based videos or externally induced anxiety tasks, limiting their relevance to real-world, everyday scenarios. Additionally, non-verbal cues, such as facial expressions and head movements, remain underexplored in detecting SAD. To address these gaps, we designed a study set in a social environment where participants were surrounded by unfamiliar individuals and instructed to remain idle without engaging in any activity. Using a low-cost smartphone camera, we recorded videos of the participants’ faces, extracted facial features, and analyzed them for insights.

Participants were recruited from the home institute through an email advertisement, following approval from the Institutional Review Board. A dedicated email was sent to the student community with information related to the study and the Google form to fill out for the interested participants. The participants filled out their demographic information, such as age, gender, current educational program, location of home residence, and preferred time slot. Additionally, the participants’ email and phone numbers were collected so the research assistant (RA) could contact them on the study day.

The day before the study, the RA sent an email to the participants to confirm their availability for a specific time slot. Further, a text message was sent one hour before the study session, confirming the location and time for participation. Three participants were invited to the lab for each study session. The RA ensured that the three participants were unfamiliar and did not know each other. The purpose of inviting three unfamiliar participants was to create a socially anxious situation during the study.

Upon arrival, the participants were seated around a rectangular table with rounded edges. Each session involved three participants and a RA. The participants were labeled as P1, P2, and P3 for each study session. The seating arrangement is shown in Figure 2: P1 was seated to the left of the RA, P2 was directly opposite the RA, and P3 was to the right of the RA, where P1 and P3 faced each other, while P2 faced the RA. This arrangement ensured that each participant faced an unfamiliar person, creating a socially anxious scenario. The RA explained the study to the participants and distributed the Participant Information Sheet (PIS) and the Informed Consent Form (ICF). After collecting the signed consent forms, the RA obtained permission to start camera recordings. Three individual smartphones, labeled C1, C2, and C3, with 13-megapixel back cameras were used to record P1, P2, and P3, respectively (see Figure 2). The “Background Video Recorder” app was selected for its ability to record video even with the screen off, which was not possible with the smartphone’s default camera app. The app was configured to use the highest possible sampling rate of 30 frames per second (FPS).

The RA instructed participants to remain seated for two minutes without interacting with others. Participants were free to look around but remained idle. After two minutes, the RA concluded the session, distributed a self-reported survey (discussed later), and recorded the session’s start and end times. Finally, the RA thanked the participants and provided refreshments as a token of appreciation for their time and participation.

The participants in our study are the student population of the author’s institute. Most participants were male (#58, 63.74%), followed by female (#33, 36.26%). In terms of education status, 71.43% (#65) were undergraduate students, while the remaining 28.57% (#26) were graduate students. Regarding home location, 76.92% (#70) were from urban areas, and 23.08% (#21) were from rural areas. A detailed breakdown of the participants’ demographic information is provided in Table 1.

The self-reported survey collected during the study was used as the ground truth. The survey included a single question asking participants about their anxiety levels during the studys session (i.e., sitting idle for 2 minutes). Participants rated their anxiety on a Likert scale from 1 to 5, where: 1: Very nervous, 2: Somewhat nervous, 3: Neither relaxed nor nervous, 4: Somewhat relaxed, and 5: Very relaxed. The distribution of self-reported anxiety ratings is shown in Figure 3.

For the ground truth, participants who rated their anxiety as 1 or 2 were labeled as anxious, while those who rated their anxiety as 4 or 5 were labeled as non-anxious. A considerable number of participants rated their anxiety as 3, so instead of grouping these participants into either the anxious or non-anxious categories, they were labeled as neutral.

During data processing, the recorded videos had an average duration of 124.21 seconds and a mean sampling rate of 20.02 FPS. To ensure uniformity, we selected the first 90 seconds of each video for analysis, as participants generally exhibited reduced anxiety over time. Videos shorter than 90 seconds were excluded, while longer videos were trimmed to the initial 90 seconds. Data from six participants were lost due to issues such as delayed recording initiation by the research assistant or technical problems, including incomplete recordings caused by memory errors or full storage. Ultimately, data from 85 participants were retained for further analysis. This approach minimized data loss while maintaining a consistent dataset.

To extract features from participant’s videos, we used the open-source OpenFace²²2https://github.com/TadasBaltrusaitis/OpenFace Python toolkit (Baltrušaitis et al., 2016). OpenFace is a well-validated Python-based tool for facial analysis tasks and has been widely used in various behavioral studies, including depression and anxiety detection. The toolkit processes video inputs and generates time-series data consisting of 714 features. In our analysis, we excluded metadata information features (#5) and rigidity features (#40) due to limited relevance in existing literature and lack of interpretability. The remaining 669 features used in the analysis are described in Table 2.

Following methodologies proposed by Bhatti et al. (Bhatti et al., 2024) and Schmidt et al. (Schmidt et al., 2018), the data was prepared for classification through two key steps: (i) Chunking: The 669 OpenFace features were segmented using non-overlapping windows of 10 seconds to create data chunks. (ii) Flattening: For each chunk, the mean values of all features were computed along the time dimension, resulting in a data sample of size $1\times 669$.

This process produced a dataset of 1,173 samples. Ground truth labels were then associated with the prepared dataset, resulting in the following class distribution: anxious (314 samples), neutral (384 samples), and non-anxious (475 samples). Additional details on sample distributions are provided in Table 3.

In our analysis, we used classical ML and DL classification models to evaluate the ability of AnxietyFaceTrack to detect anxiety in a lab setting without introducing additional anxiety-provoking situations. Table 4 presents the performance metrics for all the classification models used. Specifically, Table 4 shows each class’s average precision and recall to assess the model’s performance for each class. In the case of ML, we found that KNN and RF performed well, while LR showed the poorest performance in terms of accuracy. Further inspection of the other metrics revealed that RF and KNN achieved almost identical average precision and recall. However, when focusing on the “anxious” label, RF outperformed KNN, achieving higher average precision for this label.

Moreover, Table 4 provides additional insights. Firstly, RF outperformed DT across all metrics, suggesting that the single-tree structure of DT suffered from overfitting, whereas the ensemble approach of RF (using multiple trees) effectively handled high-dimensional data without overfitting. Secondly, RF outperformed LR on all metrics. This is likely because LR relies on linear decision boundaries, which failed to capture the complex and non-linear patterns in the data, while RF could handle these complexities. The overall best performance of RF in multiclass classification inspired us to conduct an ablation study to assess its effectiveness in binary classification scenarios (see Section 4.2.3), such as distinguishing between “anxious” and “non-anxious” participants.

In the case of DL models, the 1D CNN outperformed the MLP on most evaluation metrics. The 1D CNN achieved an average accuracy of 84%, compared to 80% for the MLP. Although these DL models performed better than most used ML models, they still lagged behind the RF and KNN. This could be due to the ability of machine learning models to learn effectively with smaller datasets, while deep learning models require larger amounts of data.

Given the overall best performance of RF across all evaluation metrics, we will now use the RF model in various ablation studies, which are discussed later in the paper.

In this work, we use facial video features for anxiety detection, and it is important to understand which facial features are linked to anxiety. To explain the results, we conducted a post-hoc analysis using SHapley Additive exPlanations (SHAP) (Lundberg, 2017) to examine the classification model. SHAP quantifies the contribution of each feature to the model’s prediction using game-theoretic Shapley values. Additionally, the Shapley values provide insights into how each feature affects the model’s decision-making. We will use a Random Forest model trained on the full feature set to identify the important features.

Literature suggests that ML models can sometimes be biased due to factors such as gender, age, etc (Cheong et al., 2023; Chu et al., 2023). Moreover, in AnxietyFaceTrack, we use facial features, which can vary based on gender and age (Bannister et al., 2022). This highlights the need to assess biases in Random Forest model related to these factors. To investigate potential bias in our trained model, we checked if AnxietyFaceTrack suffers from any bias. We split our test data into two gender groups: Male and Female. For age, we categorized the test data into Undergrad and Graduate. Additionally, we examined bias based on participants’ home locations by dividing the test data into Rural and Urban groups.

Figure 8 shows the performance results of our Random Forest model, revealing several key observations. First, for gender (see Figure 8(a)), the classification metrics were higher for females, with a difference of 5-7% compared to males. This suggests that AnxietyFaceTrack works better for females, possibly because females tend to display emotions more prominently through facial expressions than males (Parkins, 2012; Fischer and LaFrance, 2015; Kring and Gordon, 1998). Second, for education level (see Figure 8(b)), we observed about a 10% difference in classification metrics between graduate and undergrad participants, indicating that the model was better at learning the more subtle behaviors of graduate participants. Lastly, for location (see Figure 8(c)), the model performed similarly for both rural and urban groups. However, precision was higher for the rural group. Upon closer inspection of the precision for individual labels, we found that precision for rural participants was above 95%, while for urban participants, it was around 85%. Our analysis of these biases aims to increase transparency in ML models for anxiety detection and provides valuable insights for future research on anxiety detection.

Early detection is crucial for enabling timely interventions and promoting recovery for mental disorders (Colizzi et al., 2020). This study used facial videos captured through a low-cost smartphone camera for anxiety detection. Our promising results highlight the potential of smartphone-recorded facial videos and machine learning for the early detection of anxiety. This innovative approach can complement existing mental health assessments. Although our study was conducted in a controlled setting, the results pave the way for future research to explore our methodology in real-world settings, leading to a better understanding of anxiety and its early detection in fully naturalistic settings.

Furthermore, our use of low-cost smartphone cameras opens the possibility for anxiety detection through facial features to be feasibly integrated into everyday settings. This technology has the potential to be incorporated into smartphones, enabling early detection and monitoring of anxiety. Extending this work could also assist mental health professionals in routine assessments and interventions, helping to reduce the mental healthcare gap, especially in low-income and developing countries.

AnxietyFaceTrack study provides valuable insights into an unobtrusive mental health assessment tool for anxiety detection. However, it has certain limitations. First, the study was conducted in a controlled setting, which might limit the study findings in larger settings. However, the findings can serve as a baseline for future research conducted in either controlled or uncontrolled settings for anxiety detection. Second, the dataset size is limited due to the small number of participants. However, it is worth noting that machine-learning models were able to identify patterns associated with anxiety. Finally, our study used a self-reported survey questionnaire to create the ground truth, which may be subject to recall bias. Future studies could incorporate multiple questionnaires to reduce potential bias in participants’ responses.