A Multimodal Data-driven Framework for Anxiety Screening

Anxiety is a feeling of tension, worry, or restlessness. It occurs frequently in a variety of mental conditions, including phobias, panic disorders, and generalized anxiety disorders [23]. Anxiety is a typical response to risk and mental stress. The amygdala and hippocampus are activated by the feelings of fear and dread brought on by stress, which also affects the autonomic and parasympathetic nervous systems [24]. Patients with anxiety disorders exhibit physical symptoms that are linked to the disease, such as rapid breathing [7], heartbeat, BP [8], and additional symptoms [25] such as perspiration, tightness in the muscles, and vertigo. The physiological signals that are most frequently utilized in physiological research to evaluate mental health include ECG [26], heart rate, heart rate variability [8], EEG [27], and electrode signals, as shown in Table 1.

Patients with anxiety disorders exhibit structural and functional abnormalities in the nerve system that regulates emotion, according to brain imaging studies. As shown in Table 1, a person’s ability to manage their emotions can be plainly noticed in their facial and behavioral characteristics [10] as well as audio indicators (such as intonation and speech tempo) [28]. For example, the insula, frontal orbital cortex, anterior cingulate cortex, striatum, and amygdala all exhibit diminished response to unfavorable emotional stimuli [29]. Due to physiological respiratory issues, individuals with anxiety disorders have voices that are a mirror of their circumstances. Related studies have shown that anxious patients exhibit elevated wavelet, jitter, shimmer, and fundamental frequency (F0) mean coefficients [30]. Mel-Frequency Cepstral Coefficients (MFCCs) declines in the presence of anxiety [28]. The main signs of facial anxiety include changes in the eyes, including pupil size variations, blink rates [19], and gaze distribution [31], as well as in the lips, including lip twisting and mouth movement. Other key facial anxiety indicators include changes in the cheeks, head movement, and head speed [32]. Additional facial signs of anxiety include pallor, twitching eyelids, and stiffness in the face. Numerous studies have shown a link between the size of the pupil and emotional or mental activity. Dilated pupils may be a sign of higher anxiety levels [33]. The coherence and direction of the eyes, are also impacted by anxiety [31]. Increased gaze volatility during voluntary and stimulus-driven gazing are correlated with high levels of trait anxiety [34]. For instance, those who are nervous typically scan negative stuff more than those who are not anxious [10].

Traditional and noncontact screening methods are based on machine learning or deep learning techniques, which are the two primary categories of anxiety screening techniques. Psychological scales, [6] and assisting technologies (such as MRI [11], CT, ECG [26], and EEG [27], and biochemical indicators [44]) are frequently used in traditional screening approaches to evaluate anxiety levels. Noncontact screening methods mainly use computer vision [25] or deep learning techniques [13], [42] to extract behavioral characteristics and physiological signals related to anxiety.

As shown in Table 1, the feature extraction methods for anxiety screening are mainly classified into three categories: neural network-based, correlation analysis, and signal processing methods. High data dimensionality and data redundancy are properties of the time series data (such as ECG [19] and EEG [27]) and image data (such as CT, MRI [11]) utilized for anxiety inference. The performance of screening anxiety methods may thus be adversely affected. To improve the performance of the traditional screening methods, the feature extraction methods in Table 1, such as Neural Networks (NN) [9], [39], [22] and correlation analysis [15], [36], are used to extract features useful for anxiety inference. Nonlinear information useful for anxiety inference is frequently extracted using neural network-based feature extraction techniques including CNN, LSTM, and Radial Basis Function (RBF).

Canonical Correlation Analysis (CCA), Sparse Canonical Correlation Analysis (SCCA) [19], and Principal Component Analysis (PCA) [35] are correlation analysis techniques for the extraction of anxiety-related features, which are helpful for inferring anxiety. In [35], PCA uses orthogonal transformations to project observations of a group of possibly correlated variables (such as the time and frequency domain statistical from respiratory signals) into the principle components of a group of linearly uncorrelated variables. The number of features in a dataset can be controlled by imposing a threshold using PCA, which is frequently used to minimize the dataset’s dimensions.

Due to acquisition equipment or environmental factors, the quality of physiological signal extraction or analysis is easily affected. For example, the iPPG signals used for anxiety inference is particularly susceptible to ambient lighting and motion artifacts [42]. So these physiological features from the iPPG signals contain a lot of noise. The signal processing techniques in Table 1, such as Kalman filter [13] and fast Fourier transform [42], are typically used to reduce noise and eliminate its detrimental effects on the anxiety screening model. In [13], the enhanced Kalman filter processes the heart rate and accelerometry signals to follow the user’s heart rate information in various contexts and choose the best model for anxiety detection based on the user’s exercise conditions.

Due to complex etiology and long development cycle, the diagnosis of mental illness usually requires a multidisciplinary approach that combines biomedicine, psychology, and social medicine [17]. Moreover, multimodal data can provide more objective evidence for anxiety screening [18]. The multimodal data-driven approaches are promising for anxiety screening. By using the correlation analysis approach to examine the structural information between the EEG and eye movement data, it is possible to combine these two types of variables and detect anxiety more precisely [19]. Biophysical signals such as heart rate, skin electricity, and EEG in virtual reality applications, are extracted as features of different dimensions [9]. The features from the time domain and frequency domain are then fused to achieve high-precision anxiety detection. Several biosignals, including EEG, iPPG, Electrodermal Activity (EDA), and pupil size [18], are used to measure anxiety in various driving scenarios.

However, the use of multimodal data will necessarily result in a rapid increase in data dimensions and feature redundancy. The increase in data dimensions leads to curse of dimensionality. The model’s accuracy may be affected since redundant features may have a lot of noise. Decision Trees-based (DTs) approaches such as Random Forest (RF), Adaptive Boosting (AdaBoost) [10], Naïve Bayes (NB) [38], extreme gradient boost (XGBoost) [40], can select features useful for anxiety detection. They use information attributes (such as information gain, information entropy, or Gini index [22]) to lessen the dimensionality and number of redundant features in the data. Besides, unbalanced datasets are very common, particularly in the medical industry. It also poses a significant obstacle to anxiety screening. The existing machine learning techniques [10], [13], [39], [38], [21] and neural network techniques [10], [39], [36], [22], [18] used for anxiety screening are therefore challenging to apply to this unique situation.

Table 2 shows the different features extracted in Feature Extraction. The heart rate and respiration rate features from iPPG signals, BF, AF, TF, and QF features from facial video and questionnaires, which are used for anxiety inference.

The majority of features are time series data. The data dimensions of the original HR and RR features from the time domain and frequency domain, behavioral features, and text features from audio are often rather high. For example, a total of 18 AUs can be extracted from each frame of a facial picture given in Figure 3, so the total number of dimensions of AUs obtained from a video with a sampling rate of 25 Frames Per Second (FPS) and a duration of one second is 18 $\times$ 25. It is similar to the dimensions of other time series features. To reduce the data dimensions and the cost of feature selection in the next step, the original features are processed by deep learning networks [9], [39] for dimension reduction, which is shown in Figure 5.

The Convolutional Neural Network (CNN) can effectively extract spatial features of high-dimensional data [41], while Gated Recurrent Units (GRU) network is a variant of the Long Short-Term Memory (LSTM), which can extract temporal features in time series data [61]. Compared with time series data, the dimensions of questionnaire features are not high, so it does not need dimension reduction. Therefore, as depicted in Figure 5, the feature vector $[F_{iPPG},F_{BF},F_{AF}]$ and $F_{TF}$ are produced when the iPPG, behavior, and audio, and text features are processed by the ”1DCNN+GRU” and ”CNN_Text” networks, accordingly. Then, the feature vector $F=[F_{iPPG},F_{BF},F_{AF},F_{TF},F_{QF}]$ is composed of $[F_{iPPG},F_{BF},F_{AF}]$, $F_{TF}$, and $F_{QF}$.

Several ”1DCNN+GRU” networks with the same structure are employed to extract spatiotemporal features from iPPG, behavior, and audio respectively to reduce data dimensions. The ”1DCNN+GRU” network consists of three 1D-MaxPool layers, three 1D-convolutional layers, and two GRU layers. A Rectified Linear Unit (ReLU) layer and a dropout layer are added after each convolutional layer and MaxPool layer, respectively. The Conv-k(p1, p2) denotes the k-th convolutional layer. The p1 and p2 represent the out channels and kernel size, respectively. Additionally, the $TF$ features are processed by the ”CNN_Text” network to obtain the feature vector $F_{TF}$. The ”CNN_Text” network shown in Figure 5 consists of one embedding layer, two convolutional layers, two ReLU layers, one pooling layer, one flatten layer, two dropout layers, and two Dense layers.

In Figure 5, convolutional layers are used to extract different features of the input. The first layer of convolutional layers may only be able to extract some low-level features, and more layers of networks can iteratively extract more complex features from low-level features. The max-pooling layer not only reduces the feature dimension but also retains more texture information. The ReLU layer is to increase the nonlinear fitting ability of the neural network. The dropout layer can enhance the generalization ability of the model. The embedding layer is to encode sentences or words. The flattened layer is to make the multi-dimensional input one-dimensional, and it is often used in the transition from the convolutional layer to the fully connected layer. The dense layer is to change the previously extracted features through nonlinear changes, extract the association between these features, and finally maps them to the output space.

There are still a lot of redundant features in $F$=[$F_{iPPG}$, $F_{BF}$, $F_{AF}$, $F_{TF}$, $F_{QF}$] extracted from the original data. In addition, due to the data imbalance problem, the parameter learning of the model is more biased toward the majority classes during the training process. As such, the performance of the model will be adversely affected. The feature selection [22] approaches, such as the filter and wrapper methods, are used for anxiety screening to remove redundant features from the original data to enhance the effectiveness of models. Embedding methods, which combine the advantages of filter and wrapper techniques, can also be used for anxiety screening. Moreover, bio-inspired methods that introduce randomness into the search process to avoid local optimum to learn the model parameters are more conducive for predicting the minority class. Therefore, the feature selection method based on our Improved Fireworks Algorithm (IFA) is used to search for feature subsets to solve the feature combination optimization problem, which is non-differentiable.

The parameter learning for anxiety inference models can be viewed as a 0-1 programming problem. If the feature vector $F$ in Figure 6 has d-dimensional features, there are $2^{d}$ possible feature combinations. The feature subset is searched by IFA from the solution space of $2^{d}$ feature combinations to find the one that can achieve the best model’s overall classification evaluation metrics. In previous studies [19], the model’s performance usually uses one or more classification evaluation metrics, such as accuracy, precision, sensitivity, specificity, or F1-score. However, when the samples in the data set are unbalanced, these metrics can hardly distinguish the model’s performance [63]. In addition, too low sensitivity or specificity may cause adverse consequences. For instance, a test with low sensitivity may cause errors, and fail to detect correctly. The test’s low specificity might lead to a lot of false positive results, which is quite stressful for patients. Therefore, it is necessary to measure the performance of the model by multiple classification evaluation metrics, such as the Area Under Curve (AUC), accuracy (Acc), precision (Pre), sensitivity (Sen), specificity (Spe), and F1-score (F1), and use Equation (9) as the loss function for model optimization. A penalty factor $\lambda=0.2\times d$ needs to be introduced to limit the dimensions of the selected features.

where True-Positive (TP) and True-Negative (TN) denote the accurate classification of the anxiety and anxiety-free samples, respectively. False-Negative (FN) and False-Positive (FP) suggest that the anxiety and anxiety-free samples are wrongly categorized respectively. In addition, AdaBoost [64] is a technique for ensemble learning that uses an iterative process to improve weak classifiers by learning from their mistakes [10], [13]. Due to its effective classification performance and the interpretability of the inference process, Adaboost is utilized as the classifier for anxiety screening.

Figure 7 depicts the anxiety inference process. The feature subset produced by the feature selection method can be used for determining the selected features. The data corresponding to these features is divided into training and test sets. The training set is used to train the classifier. The trained classifier assesses the feature subset based on the predictions from the test set. When a feature subset is evaluated once, the number of evaluations is increased accordingly, ie, $Eve=Eve+1$. If the total number of evaluations reaches the predefined maximum number of evaluations, ie, $Eve>MaxEve$, the selected features and trained classifier are used for anxiety inference. Otherwise, the feature selection method process searches for a new feature subset.

We have designed a system that can be accessed using mobile devices based on the WeChat applet platform. With their smartphones, seafarers can then check their physical and mental well-being. First, seafarers need to fill in a questionnaire. As shown in Table 2, the content of the questionnaires includes personality traits [54], poor sleep quality (leading to fatigue) [45], bad emotional state, attitude to life [58], family relationship, social support [60], etc.

Next, the seafarers are then asked additional questions on their work status, and relationships with family and friends after completing the questionnaires. The seafarer’s responses are captured on camera by their phones. Each video capture lasts for 30 seconds. The smartphone concurrently records the audio data while recording the video. The sampling rates for audio and video are 25 FPS and 22.05 kHz, respectively. The video has a 480 $\times$ 480 pixel resolution. The average, maximum, and minimum audio durations are 29.31 seconds, 30.44 seconds, and 28.96 seconds, respectively. When responding to the inquiries, the seafarer’s face is kept as visible as possible on the recorded screen, as shown on the right side of the interface in Figure 8. One frame per second is taken to assess the acquired video’s quality before the video is transferred to the server. The video is uploaded to the server if the rate of faces being detected in it exceeds 90%. Otherwise, it needs to be captured again.

In the experiments, all seafarers taking part in the physical and mental health evaluation were male. The age range of seafarers was 19 to 58. There were 2 seafarers’ age between 18 and 20, 167 people between the ages of 20 and 40, and 20 seafarers’ age between 40 and 60. The seafarers’ collective age was known to be 31, on average. The age information for other seafarers was not recorded due to errors. The start and end dates for data collection were from June 2020 to June 2021. A total of 227 of seafarers’ data were recorded: 189 of them had no anxiety, 33 had mild anxiety, and 5 had moderate anxiety. In other words, 189 people were labeled ”anxiety-free” and the remaining 38 people were labeled ”anxiety”. The GAD-7 [6] was used to measure seafarers’ anxiety levels. Additionally, psychiatrists were invited to validate the outcomes of the GAD-7 scale that the seafarers had completed.

Our proposed MMD-AS framework consists of the dimension reduction component ”1DCNN+GRU+CNN_Text”, feature selection component ”IFA”, and anxiety inference component ”AdaBoost”. Table 4 shows the performance results of different methods for different components. The performance is computed using Average (Avg) = (Acc + AUC + Pre + Sen + F1 + Spe)/6. $\Delta$Avg shows the average performance difference of the method when compared to the proposed MMD-AS framework. Overall, the proposed MMD-AS has achieved the best performance with Avg = 97.55%.

We have conducted an experiment for an ablation study to evaluate the effectiveness of each component in our proposed MMD-AS framework: (1) M15 is MMD-AS without feature selection. (2) M16 is MMD-AS without feature selection and anxiety inference. As M16 only has a dimension reduction component ”1DCNN+GRU+CNN_Text”, it cannot perform classification. So we add a fully connected layer behind the component to enable M16 with a classification function. (3) M17 is MMD-AS without dimension reduction and feature selection.

Table 5 shows the results of the ablation experiments. We have the following observations. First, MMD-AS outperforms M15 by 5.65%, which demonstrates the capabilities of the feature selection method IFA in terms of feature selection and feature denoising. Second, MMD-AS outperforms M16 by 3.56%, which shows that the components IFA and AdaBoost improve the model’s performance. Third, MMD-AS outperforms M17 by 8.25%, which shows that the components ”1DCNN+GRU+CNN_Text” and IFA can improve the model’s performance. Overall, the different components of the proposed MMD-AS framework are important for the framework to achieve the best performance.

Feature importance scores are used to indicate which features are useful for anxiety screening. When the feature selection algorithm selects a feature of a certain dimension in the feature $F$, the importance score of the feature is increased by one. Figures 9(a) to 9(d) show the ranking of our proposed MMD-AS framework on the importance scores of different types of features such as iPPG features (including HR and RR features), BF, AF, TF, and QF for anxiety screening.

In Figure 9(a), the importance of QF and BF features are ranked the highest and the second highest among the five categories of features, respectively. Figure 9(b) shows the feature importance scores of physiological representations in anxiety screening, such as audio features, and iPPG containing heart rate and respiration rate features. Audio features, such as Prosody, fundamental frequency features (including F0 and F0¹), Pitch Perturbation Quotient (PPQ), Jitter, and Amplitude Perturbation Quotient (APQ), play a more important role in anxiety screening. Their feature importance scores are all greater than 150. The iPPG features from frequency domain signals including iPPG${}_{Fore}^{FD}$ and iPPG${}_{Nose}^{FD}$ are more important for anxiety screening. Compared with the iPPG features from the nose area, iPPG features from the forehead are more important for anxiety screening, since the forehead is dense with blood vessels [42]. Figure 9(c) demonstrates that anxiety screening is more heavily influenced by characteristics in the chin (e.g. AU17), eyes (e.g. AU02, AU05, AU01, AU45, AU04), and lips (e.g. AU10, AU23, AU20) areas. The scores of these features are mostly distributed in the range of [160, 206]. Figure 9(d) shows the contribution of text and questionnaire features to anxiety screening. The feature importance scores of SSRS, MFI, PF, HPLP, and PSQI features ranked among the top in anxiety screening. The importance scores of these features are all greater than 1400.

Due to the complex etiology and long development cycle of anxiety, it usually requires a combination of multidisciplinary knowledge such as biomedicine, psychology, and social medicine to assist diagnosis [17]. In clinical practice, multimodal data are essential for anxiety screening [19], [18]. Therefore, based on the results of the feature analysis, we suggest that screening patients with anxiety should be combined with multimodal information. In addition, behavioral features (such as chin, eyes, lip area), physiological signals from the frequency domain (such as heart rate, respiration rate), audio characteristics (such as the Prosody, fundamental frequency features, PPQ, Jitter, and APQ), social support, fatigue status, sleep quality [45], lifestyle and other important modal characteristics can be used as indicators for clinical practice.