Towards More Efficient Depression Risk Recognition via Gait

Due to the significant harm caused by depression, researchers have conducted numerous studies in the field of depression risk recognition. The application of machine learning methods has propelled advancements in this field. Machine learning-based approaches for recognizing depression risk encompass various modalities of data.

Physiological signals such as electroencephalogram (EEG) and electrocardiogram (ECG) can be utilized for the recognition of depression risk. Depression is associated with abnormal brain activity, which can be reflected through EEG signals. Additionally, due to disruptions in the autonomic nervous system, depressive episodes may be accompanied by symptoms such as nausea, vomiting, chest tightness, and sweating[42], which can be observed in ECG signals. Common machine learning methods, including principal component analysis (PCA), support vector machines (SVM), random forests, and long short-term memory (LSTM), have been employed to analyze the relationship between these physiological signals and depression risk[43, 44, 45].

Brain imaging techniques are also among the means used for recognizing depression risk. This primarily includes magnetic resonance imaging (MRI), functional near-infrared spectroscopy (fNIRS), and positron emission tomography (PET), among others. Depression can potentially lead to structural and functional abnormalities in the brain, such as hippocampal atrophy, neurotransmitter changes, and chronic inflammation. Brain imaging techniques can assess the condition of patients by detecting these symptoms. In order to achieve automated processing and analysis of brain images, image processing methods from the field of machine learning have been introduced in this domain[46, 47, 48, 49].

Furthermore, depression can potentially lead to changes in facial expression patterns and speech patterns of patients. Researchers have proposed the use of facial expression analysis techniques[50] and speech analysis techniques[51] for the recognition of depression risk.

The aforementioned exploration of depression risk recognition through machine learning methods has explored various approaches. However, the data modalities used by these methods require participants to undergo data collection in specific scenarios and with specific devices, demanding a high level of cooperation from the participants. As a result, these methods incur higher time and manpower costs, ultimately making them no more efficient than traditional psychological questionnaires in practice.

Gait analysis can be broadly categorized into two main streams of research methods: model-based gait analysis and appearance-based gait analysis. In the realm of model-based gait analysis research, a pioneering study by Wang et al.[52] proposed a method that combines static and dynamic human body information. Subsequently, Li et al.[53] introduced an end-to-end model-based gait analysis and recognition method. Given an RGB gait sequence, it first extracts pose and shape features by fitting the SMPL model, and then provides these features to a deep learning model for analysis. The model-based approach in gait analysis research aims to model the structure of the human body, resulting in extracted features that possess better invariance. These approaches demonstrate good performance even in the presence of occlusion, noise, scale variations, and rotations.

In contrast, appearance-based gait analysis methods analyze human gait based on the appearance of the human body. Early representative research includes the work of Kusakunnira et al.[54], who proposed a support vector regression-based view transformation model to address the view variation problem. Bashir et al.[55] estimated the view of each gait sequence using Gaussian processes and used correlation analysis to model gait sequences from different view, enabling cross-view gait analysis. With the rise of deep learning, methods based on deep neural networks for appearance analysis have gradually become mainstream in the field of gait analysis. Wu et al.[56] implemented gait recognition by learning the similarity between two input samples using deep convolutional networks. Their research found that deep neural networks can effectively extract gait features. Subsequent gait analysis studies have focused primarily on overcoming the challenges posed by view and attire variations. Yu et al.[57] employed generative adversarial networks as regressors to generate gait images, addressing gait analysis under different views, attire variations, and carrying objects. Chao et al.[58] proposed the GaitSet model, which treats gait data as an unordered set of gait silhouettes and models them accordingly. Song et al.[59] proposed a fully end-to-end gait analysis model, greatly streamlining the process of gait analysis. Fan et al.[60] proposed dividing the human body into patches to extract features separately, addressing the variations caused by different perspectives and attire through a divide-and-conquer approach. In response to the issue of high dimensionality in gait features, Hou et al.[61] presented a method for compressing gait features. To better model global and local information, Lin et al.[62] proposed a global-local feature extractor.

While there has been rapid development in general gait analysis methods in recent years, most current approaches primarily focus on identity recognition tasks. They lack the exploration of fine-grained information contained in human gait, such as emotions, psychological states, and health conditions. Therefore, there is a need for systematic integration of relevant domain knowledge to further research in this area.

Several researchers have explored the relationship between gait and depression. For instance, Sloman et al.[63] conducted an analysis of photograms capturing a single stride during a natural walk and observed that individuals with depression exhibited slower walks with a lifting motion of the leg. Another study utilized electronic walkways and photogrammetry to demonstrate that depressed patients had shorter strides and slower gait velocity compared to healthy controls[41]. Michalak et al.[40] utilized 3D motion capture and showed that individuals with depression displayed reduced vertical head movements, a more slumped posture, and lower gait velocity compared to controls.

Some researchers have also attempted to recognize depression risk through the recognition of gait features. For example, Wang et al.[64] utilized Support Vector Machines (SVM) to construct a machine learning model that could recognize depression risk by extracting spatio-temporal, time-domain, and frequency-domain features. Lu et al.[65] proposed a model based on human body skeleton, aiming to extract gait features related to depression risk through Fast Fourier Transforms (FFT). Fang et al.[66], on the other hand, employed Random Forests in an attempt to recognize depression risk.

However, the existing studies suffer from two noteworthy limitations: Firstly, these studies are often constrained by a dearth of heterogeneous gait data, which frequently fails to incorporate varying perspectives and attire. Consequently, the generalizability of their findings remains uncertain, as the conclusions drawn may not be readily applicable to a broader range of gait data. Secondly, these studies commonly rely on hand-crafted gait features for analysis. Nevertheless, human gait is an intricate and multifaceted form of motion, and such manually constructed features often offer a limited portrayal, capturing only a fraction of the intricate correlations between gait and the risk of depression.

The subjects chosen for the collection of gait data are consenting adult volunteers who have been informed about the nature of our research endeavors. These volunteers are instructed to traverse a predetermined walking route while being recorded by six cameras placed strategically from distinct views, as shown in Fig. 1. The six cameras are divided into three groups, each consisting of one high and one low camera. The two cameras of each group are positioned at heights of 2.5 meters and 3.5 meters above the ground, respectively, in order to capture data from different vertical views. These three groups of cameras are evenly distributed along an arc ranging from the front to the side, enabling the capture of gait videos from various horizontal views. The designated walking route spans a length of six meters, requiring the participants to traverse from one end to the other and subsequently return to the starting point. Each volunteer is required to undergo three data collection sessions, with each session involving a distinct attire. Each volunteer is required to undergo three sessions of data collection, with each session involving different attire, including wearing a coat, not wearing a coat, and carrying a backpack. This is done to obtain gait videos under varied dressing conditions.

The pre-processing of the video gait data primarily involves two steps: Firstly, tracking the subjects in the videos. Subsequently, in order to preserve the privacy of the individuals being captured, we perform human segmentation on the tracked tracklets, retaining only the silhouettes that encompass the contour information of the human body, as shown in Fig. 2. The private information is eliminated during data pre-processing, and the original data is permanently removed. Through the aforementioned process, the privacy of the subjects involved has been effectively safeguarded.

Two widely used psychological scales are used to assess the depression risk of the participants:

• •

  The Self-Rating Depression Scale (SDS): SDS is one of the most renowned self-report measures for depression. It is recommended by the U.S. Department of Health and Human Services for research in psychopharmacology. It is widely utilized for screening and assessing depression due to its ease of use and ability to reflect the subjective experience of patients. The SDS consists of 20 items and provides a quick assessment of an individual’s risk of depression.

• •

  The 9-question Patient Health Questionnaire (PHQ-9): PHQ-9 is a commonly used scale in outpatient settings for the rapid preliminary screening of depression risk in individuals.

During the depression risk identification experiment, participants are classified into an experimental group, indicating the presence of depression risk, and a control group, indicating the absence of depression risk, based on their scores on these scales. Participants who meet both of the following criteria are assigned to the experimental group: SDS score greater than 58 and PHQ-9 score greater than 8. Participants who meet both of the following criteria are assigned to the control group: SDS score less than 47 and PHQ-9 score less than 2.

The statistical information of the dataset proposed in this paper is presented in Table 1. A comparison with existing datasets is illustrated in Table 2. In the existing dataset, the proposed dataset has the largest number of participants and gait sequences. It also includes the highest number of camera views and attires conditions. Due to privacy concerns, the majority of existing datasets cannot be openly accessed, which significantly impedes the progress in this field. To address this issue, we have implemented effective privacy protection measures during the data preprocessing stage. We will release gait silhouette data along with corresponding annotations of depression risk, but only with the informed consent of the volunteers. This study aims to facilitate the advancement of research in this field while ensuring the protection of participants’ privacy.

Compared to general gait analysis tasks, there are two notable characteristics in depression risk recognition based on gait analysis: Firstly, the main features associated with depression risk are dynamic features, which refer to the temporal aspects of gait during the walking process[63, 41, 40, 64]. Static features of the human body have less correlation with the depression risk, e.g., shape. Secondly, gait features that are associated with depression risk can manifest as either local details or involve the entire body[63, 41, 40]. Based on these two characteristics, this paper proposes a deep learning model with dynamic feature modeling as its core. In the feature extraction process, both local dynamic features and global dynamic features are effectively integrated.

This deep learning model consists of three kinds of modules: the dynamic feature extraction module, the feature compression module, and the recognition head, as shown in Fig 3. The dynamic feature extraction module is designed to extract the dynamic features of gait, and it consists of two branches: local dynamic feature branch and global dynamic feature branch. The feature compression module compresses the dynamic features to obtain a more compact and effective representation. The recognition head maps the final features to the task space of depression risk recognition. The following subsections will provide a detailed description to each of these modules.

Existing research on the association between depression and gait suggests that the correlation between depression risk and gait primarily manifests in the dynamic characteristics of gait. Therefore, it is crucial to effectively extract and model the dynamic features of gait for the purpose of depression risk recognition based on gait. To achieve this, we propose a novel dynamic feature extraction module. Furthermore, the manifestation of gait features associated with depression risk varies, encompassing both the entire body and specific local movements. Therefore, the dynamic feature extraction module consists of two branches: the local feature branch and the global feature branch, which are dedicated to extracting local and global features, respectively.

When given input $x_{In}\in\mathbb{R}^{C\times T\times H\times W}$, where $C$ is the number of channels, $T$ is temporal scale of $x_{In}$, i.e., the length of the sequence, $H$ and $W$ are the spatial scale, the local feature branch initially divides it into $K$ parts in the vertical direction. $x^{k}_{In}\in\mathbb{R}^{C\times T\times\frac{H}{K}\times W}$ is the $k$-th part, $k\in\{1,2,...,K\}$. Subsequently, a local 3D convolutional layer is employed to extract dynamic features from each part:

$$x^{k}_{Loc}=M^{s\times s\times\tau}_{Loc}(x^{k}_{In})$$

(1)

where $M^{s\times s\times\tau}_{Loc}$ is the local 3D convolutional layer, $s$ is the spatial kernel size of the 3D convolution operation, $\tau$ is the temporal kernel size.

Most existing studies on gait analysis primarily employ 2D convolutions for the process of low-level feature extraction. However, considering the significance of dynamic features in depression risk recognition, we have opted for 3D convolutions, enabling the model to directly capture gait patterns in the temporal dimension. The parameter $\tau$ is used to regulate the temporal range during dynamic feature extraction. When $\tau$ is set to a higher value, the model tends to capture longer-term gait patterns, whereas a smaller $\tau$ value favors the capture of gait patterns within shorter time intervals.

By partitioning the input into K parts, we ensure that the feature extraction process between each part is independent, allowing the model to focus on a part, i.e., a local region of the input, for dynamic feature extraction. This enables the local feature branch to concentrate on the localized features associated with depression risk. After local feature extraction, the local feature maps of different parts are concatenated:

$$x_{Loc}=Concat(x^{1}_{Loc},x^{2}_{Loc},...,x^{K}_{Loc})$$

(2)

The global feature branch also employs 3D convolutions for dynamic feature extraction, without the need to partition the input into parts.

$$x_{Glob}=M^{s\times s\times\tau}_{Glob}(x_{In})$$

(3)

where $M^{s\times s\times\tau}_{Glob}$ is the global 3D convolutional layer, $s$ is the spatial kernel size, $\tau$ is the temporal kernel size. $\tau$ is also to regulate the temporal range during dynamic feature extraction.

The output of the dynamic feature extraction module is obtained by merging local features and global features:

$$x_{Out}=x_{Loc}+x_{Glob}$$

(4)

By integrating local and global features, the dynamic feature extraction process is accomplished ultimately.

From the aforementioned description, it can be found that the dynamic feature extraction module achieves efficient extraction of dynamic features through the application of 3D convolution. Additionally, the local and global branches model the local and global features separately. These targeted designs tailor the feature extraction model to the characteristics of depression risk recognition task.

Due to the typically high dimensionality of gait data and the presence of significant information redundancy, it is necessary to perform feature compression during the feature extraction process to mitigate the curse of dimensionality and enhance the effectiveness of feature representation. Information redundancy in gait silhouette data primarily stems from two aspects. Firstly, there is temporal redundancy as gait data is collected at a rate of 30 frames per second, resulting in short time intervals between adjacent frames and a substantial amount of repeated information. Secondly, there is spatial redundancy. Silhouette data only captures the contour information of the human body, which has lower information density compared to RGB data. However, the dimensionality of silhouette data remains the same as RGB data, leading to a significant amount of spatial information redundancy. Building upon the above insights, we propose two types of feature compression layers in this paper: one for temporal compression and the other for spatial compression of features.

Temporal Compression Layer (TCL) is proposed to accomplish feature compression in the temporal dimension, as shown in Fig. 5. For the feature map $x\in\mathbb{R}^{C\times T\times H\times W}$, dimensions other than the temporal dimension are omitted for the sake of simplicity: $x\in\mathbb{R}^{T}$. A straightforward method for feature compression is linear compression:

$$x_{C}=Wx$$

(5)

where $x_{C}\in\mathbb{R}^{N}$ is the compressed feature, $N$ is the temporal dimension after compression, $W\in\mathbb{R}^{N\times T}$ is the compression parameter. However, this approach would result in a significant increase in the number of model parameters, which is not conducive to model training and efficiency. To address this issue, we propose to partition the input along the temporal dimension and perform separate compression for each group:

$$x_{C}^{i}=W^{i}x^{i}$$

(6)

where $x^{i}\in\mathbb{R}^{\frac{T}{N}}$ is the $i$-th group, $i\in\{1,2,...,N\}$, $W^{i}\in\mathbb{R}^{1\times\frac{T}{N}}$ is the compression parameter, $x_{C}^{i}\in\mathbb{R}^{1\times 1}$ is the $i$-th compressed feature. Furthermore, we adopt a weight sharing strategy: :

$$\forall i\in\{1,2,...,N\},W^{i}=W$$

(7)

Finally, the compressed features are concatenated along the temporal dimension to obtain the compressed feature:

$$x_{C}=Concat(x_{C}^{1},x_{C}^{2},...,x_{C}^{N})$$

(8)

where $x_{C}\in\mathbb{R}^{N}$. The rationale behind the aforementioned method of compressing temporal features is derived from insights into gait data and related features associated with depression risk. Specifically, it is observed that gait is continuous over time and there is minimal variation between adjacent frames. Hence, we partition the inputs that are temporally adjacent into the same group for compression. Compared to directly performing linear compression, the proposed method significantly reduces the number of parameters, decreasing it from $T\cdot N$ to $T/N$.

Spatial Compression Layer (SCL) is employed to accomplish feature compression in the spatial dimensions. Spatial feature compression is a common operation in deep learning models, and max pooling is the most commonly used spatial compression method. Therefore, in the proposed model, max pooling is utilized to implement spatial compression.

Upon completion of feature extraction, the extracted feature $f$ is fed into the recognition head for the depression risk recognition. The recognition head initially performs classification to determine the presence of depression risk:

$$y=M_{Head}(f)$$

(9)

where $M_{Head}$ is a fully connected layer, $y\in\mathbb{R}^{2}$ is the output of $M_{Head}$. During the training process, the model is trained using the cross-entropy loss function:

$$L_{CE}=-\sum_{i=1}^{N}y^{*}_{i}ln\frac{e^{y_{i}}}{\sum_{j}e^{y_{j}}}$$

(10)

where $N$ is the number of classes and $N=2$, $y^{*}$ is the depression risk label, when a sample is at risk of depression $y^{*}_{1}=1,~{}y^{*}_{2}=0$, and when a sample is not at risk of depression $y^{*}_{1}=0,~{}y^{*}_{2}=1$.

Meanwhile, the depression risk recognition cannot be simply generalized as a binary classification problem. This is due to the uniqueness of individuals’ psychological states, where even for two participants both at risk of depression, their psychological states may vary significantly. Hence, in addition to the aforementioned classification loss, we employ triplet loss to constrain the features extracted by the model:

$$L_{Tri}=max(D(f_{A},f_{P})-D(f_{A},f_{N})+m,~{}0)$$

(11)

where $D(\cdot,\cdot)$ represents the distance metric function in the feature space, $f_{A}$ denotes the anchor, $f_{P}$ denotes the feature of the same subject as $f_{A}$, and $f_{N}$ denotes the feature of a different subject than $f_{A}$, $m$ is the distance margin. This loss function requires that for the same individual, the extracted depression risk features by the model should remain relatively stable and consistent, while features of different individuals should exhibit distinctiveness.

The training objective of the model is composed of the aforementioned two loss functions:

$$L=L_{CE}+L_{Tri}$$

(12)

This subsection introduces the implementation details of the proposed depression risk recognition model, as well as the training and testing configurations.

The spatial dimensions of the silhouette data are normalized to $64\times 44$. During the training process, a random sequence of 60 consecutive frames is extracted as the input sequence. If the sequence is less than 60 frames, it is padded by repeating the sequence until it reaches 60 frames. During testing, all frames of the test data are inputted into the model for testing. For training, the Adam optimizer is used with an initial learning rate of 10-4 and a weight decay of 5e-4. The batch size is set to 64, and a total of 80,000 steps are performed. The learning rate is decayed to one-tenth of the original value at the 70,000th step.

The channel of the three blocks are set as 32, 64, and 128 respectively. In the dynamic feature extraction module, the local feature branch divides the input into 16 parts. Both the spatial kernel size of the local and global 3D convolutional layers are set to 3, and the temporal kernel size are set to 5. In block 1, the temporal feature compression layer divides the features into 20 groups for compression in the temporal dimension. In block 3, the time feature compression layer divides the features into one group for compression. Euclidean distance is utilized as the distance metric function $D(\cdot,\cdot)$ in Eq.11, with $m$ set to 0.2.

During the experiment, the collected gait dataset is divided into a training set and a test set. The training set consisted of 831 subjects, while the test set consisted of 396 subjects. The training set and test set do not have any overlapping subjects. The proposed depression risk recognition model is trained on the training set and its recognition performance is quantitatively validated on the test set. Each subject’s gait data is divided into two sequences: walking towards the cameras and walking away from the cameras during training and testing. During the testing process, the probability of the class with depression risk, as outputted by the recognition model, is considered as the probability of having depression risk.

The model’s performance is evaluated using the following performance metrics:

$$Acc=\frac{TP+TN}{TP+FP+TN+FN}\times 100\%$$

(13)

where $TP$ is the number of true positive samples, $TN$ is the number of true negative samples, $FP$ is the number of false positive samples, $FN$ is the false negative samples. Accuracy ($Acc$) is an intuitive evaluation metric that directly calculates the proportion of correctly recognized samples.

$$Prec=\frac{TP}{TP+FP}\times 100\%$$

(14)

When the cost of incorrectly predicting negative samples as positive ones ($FP$) is high, precision ($Prec$) becomes a crucial metric.

$$Recall=\frac{TP}{TP+FN}\times 100\%$$

(15)

Conversely, when the cost of predicting positive samples as negative ones ($FN$) is high, recall rate ($Recall$) becomes an important metric.

$$F1=\frac{2\times Prec\times Recall}{Prec+Recall}$$

(16)

$F1$ score is an evaluation metric that combines both precision and recall rate, providing a comprehensive assessment of a model’s recognition performance.

In addition, the area under the curve (AUC) of the Receiver Operating Characteristics (ROC) curve is also used to quantitatively evaluate the recognition performance of the model. This metric is comprehensive and has the advantage of being unaffected by threshold selection.

The performance of the proposed depression risk recognition model on the collected dataset is shown in Table 3. As can be observed, the metrics of the proposed model are distributed around 80%, indicating that the model is able to distinguish between most positive and negative samples, resulting in a passable outcome. These experimental results further validate the feasibility of depression risk recognition based on gait and also confirm the effectiveness of the proposed deep learning model.

From the experimental results, it can be observed that the recall of positive samples in the depression risk recognition is significantly higher than the precision. This phenomenon indicates that the features extracted by the model are more adept at clustering positive samples with depression risk together, while negative samples without depression risk are relatively scattered. This is intuitive as positive samples with depression-related characteristics exhibit common feature representations, resulting in strong similarities between them. On the other hand, negative samples without depression risk may not have the same set of features. The similarities among negative samples are much less pronounced compared to positive samples, resulting in a more dispersed distribution in the feature space.

As depicted in Fig. 6, positive samples tend to cluster together in the feature space, while negative samples are more dispersed. Consequently, the optimal classification interface that the model can learn in this scenario would result in a larger number of false positive samples ($FP$) compared to false negative samples ($FN$), leading to a higher recall rate than precision rate.

In general gait analysis tasks, the view of camera plays a crucial role as an influencing factor. This is because there can be significant differences in the observed gait patterns across different views. Certain gait features may be easily observable from specific views, while they may be challenging to capture from other views. Therefore, it is necessary to further explore the impact of views on depression risk recognition.

To accomplish this, we divide the training and testing datasets into six subsets based on views of camera. Each subset exclusively contains gait data captured from a specific view. We then proceed to train and test the proposed depression risk recognition model on these six subsets independently. The experimental results are presented in Table 4, with the angle numbering consistent with Fig. 1.

From these results, we can observe that the performance of the recognition models based on each separate view of camera is inferior to the performance achieved with all views combined. This indicates that the features related to depression risk captured by each individual view are not as comprehensive as those captured by the complete set of views. Different angles provide complementary information of gait features related to depression risk. However, it is important to note that the difference in performance between the individual views of camera and the overall views is not significant. This suggests that the effective complementary information between different views is limited for the task of depression risk identification. Most of the depression-related features can be observed in all views. Static features in the spatial dimension, such as arm swing angles, are more influenced by view of camera, while dynamic features in the temporal dimension can be captured in multi-views, such as arm swing frequency. This further verifies that the gait features related to depression risk are primarily manifested in dynamic features in the temporal dimension rather than static features in the spatial dimension.

Additionally, we can also observe that the performance of the models captured from higher views (# 1, # 3, # 5) is inferior compared to the corresponding lower views (# 2, # 4, # 6). This indicates that lower views of camera are more favorable for capturing depression-related features. This is because higher views of camera contribute to a more severe self-occlusion phenomenon in the human body, particularly impeding the observation of leg movements.

The grading of depression is an integral aspect of depression research. This subsection explores the relationship between the predictions made by the proposed model and the grading of depression risk. To achieve this, the participants’ depression risk is classified based on the scores obtained from the SDS and PHQ-9 scales. For the SDS, a score of 70 or above indicates a severe risk of depression, while a score below 70 but above 58 indicates a moderate risk of depression. As for the PHQ-9, a score of 15 or above indicates a severe risk of depression, while a score below 15 but above 8 indicates a moderate risk of depression.

The predictions made by the proposed model for samples with moderate and severe depression risks were separately analyzed. The depression probabilities predicted by the model are categorized into 10 intervals, as shown in Fig. 7. From the figures, it can be observed that samples with a severe depression risk exhibit a more concentrated distribution around 0.7 in the model predictions. In contrast, samples with a moderate depression risk show a bimodal distribution. One part of the distribution closely resembles that of samples with severe depression risk, while the other part is centered around lower probabilities. Since the proposed deep learning model was not explicitly trained to consider the grading of depression during its training process, this phenomenon suggests that only a portion of the features related to severe and moderate depression risks are shared in the proposed deep learning model. As a result, samples with moderate risk that lack these shared features exhibit differences in the predictions compared to samples with these features, leading to the observed bimodal distribution in the model’s predictions for samples with moderate depression risk.