Pervasive Lying Posture Tracking

We extract 48 time-domain features from a sliding window over each episode of data (lying supine, prone, and on the left side) with a 50% overlap. We set the window size to the minimum episode length in the training dataset (e.g., 96 samples equal to 3.8 seconds). The selected features, shown in Table 1, are proven to be useful in human posture and activity recognition applications such as amplitude, mean, standard deviation, and angle of the signal [23, 24]. We compute a set of meta-features $f_{i}=\{f^{1}_{i},f^{2}_{i},...,f^{48}_{i}\}$ for $i_{th}$ episode by averaging of all the extracted features from multiple windows over that episode.

A total of 48 features are obtained from the previous step. We train an ensemble of 100 decision trees on the features and lying posture labels. We choose the bagging technique as the ensemble method to reduce the variance of the decision tree and overfitting to the existing data. A decision tree is selected as the weak learner because of the high dimension of the input features. As shown in Figure 1, we fit 100 decision trees on 100 random subsets of the original dataset with a randomly selected subset of features to minimize the correlation between individual trees. The final prediction is the majority voting on the decision of the individual trees [25].

Recurrent Neural Networks (RNNs) are a type of deep learners that are well-suited to model sequential data. However, RNNs fail to learn long-term dependencies in the data due to the problem of vanishing/exploding gradients. Long Short-term Memory (LSTM) network has been introduced to address this issue and capture long-term dependencies from the sequential time series data [26]. LSTM networks have shown promising results on time series classification tasks [27, 28]. LSTM captures long-distance dependencies from sequential data through the integration of memory cells and RRNs [29]. Bidirectional long short-term memory (bi-LSTM) networks were introduced as an extension to the LSTM networks. The bi-LSTM architecture consists of two LSTMs that train in two directions; therefore, it is capable of extracting long-term data dependencies in both forward and backward directions [30]. Equation 1 and Equation 2 show the mathematical formulation of a bidirectional LSTM with $L$ hidden layers. At time-step $t$, each unit in layer $(i)$ receives a set of parameters from the previous time-step (previous state $h_{t-1}^{(i)}$), and two set of parameter from the previous layer (state of previous layer $h_{t}^{(i-1)}$ and ); one input from the left-to-right RNN and one from the right-to-left RNN. The input to each unit at level $(i)$ is the output of RNN at layer $(i-1)$ at the same time-step $t$. The output, $\widehat{y}$ at each time-step $t$, in Equation 3, is the result of propagating the input parameters through all hidden layers.

In the next section, we define lying posture prediction from the sequences of raw sensor data as an optimization problem, then design a deep learning architecture using Bi-LSTM networks as the solution.

Problem: We have $N$ sequences of variable lengths where each sequence $X_{i}$ is assigned a label $Y_{i}=(y_{i1},y_{i2},...,y{ik})$ using max-likelihood classification, where $y_{ij}$ shows the likelihood of $j_{th}$ class for $i_{th}$ sequence. Given these, the problem is to estimate a set of labels $\hat{Y}=\{\hat{y_{1}},\hat{y_{2}},...,\hat{y_{N}}\}$, such that the difference between the actual and estimated label sets is minimized. We compute this difference as the cross-entropy of the estimated labels $\hat{Y}$ and actual label $Y$ for summed over the sequences.

Equation 4 is not an accurate representation of the error as the sequences might adopt different lengths. Therefore, given a set of scalar numbers $M=\{m_{1},m_{2},...,m_{i}\}$ where $m_{i}$ is the length of the sequence $X_{i}$, Equation 4 can be formulated as below.

Deep Learning Architecture: To solve this problem, We design an Adaptive LSTM (AdaLSTM), an LSTM Network with an adaptive learning rate method with a decaying learning rate schedule. AdaLSTM receives the sequences of raw accelerometer sensor data as the input and estimates one label for each sequence. As shown in Figure 1, the input episodes/sequences of raw accelerometer data are fed to a Bi-LSTM layer with ten hidden units. The training process of the bi-LSTM includes back-propagation processes in two directions with the objective of minimizing the error. Three fully connected layers multiply the output of the Bi-LSTM layer (e.g., a sequence of tri-axial accelerometer data) by the matrix of weights and add it by the vector of bias. The output of the fully connected layer is fed to a softmax layer that is a multi-class generalization of the logistic sigmoid function. We compute the cross-entropy loss for multi-class classification based on the likelihood of the softmax function. We set the maximum number of the epochs equal to $100$. We set the initial learning rate and decay rate of the squared gradient moving average to $0.01$ and $0.99$, respectively. To shorten the amount of padding in the mini-batches and make the training more suitable for CPU, we chose the mini-batches to be short sequences of size $27$. Adam optimizer [31] is used for training the neural network through backpropagation.

Class-Act is a human posture and activity classification dataset from 22 healthy participants (7 females and 15 males, ages between 20 and 36) [32]. The participants worn nine accelerometer sensors sampled at 30 Hz on nine different body locations, including the chest, left and right thigh, left and right ankle, left and right arm, left and right wrist during the activities, as shown in Figure 2(a). Class-Act was collected based on three pre-defined protocols with different combinations of activities or postures in a controlled manner. The activities were walking, sitting, standing, lying supine, lying prone, lying on the left side, kneeling, and crawling. Figure 2(b) shows the prevalence of the extracted episodes for only lying postures. The duration of different episodes for lying supine, prone, and on the left side were 12.2$\pm$3.6, 11.9$\pm$3.6, and 12.10$\pm$3.3 seconds, respectively.

The DAS dataset contains data from eight subjects (four females and four males, ages between 20 and 30) that performed 19 activities of daily living for five minutes each [33]. The participants wore five inertial sensor units embedding a tri-axial accelerometer on the chest, right and left wrist, and right and left thigh. The sensors were calibrated to acquire data at a sampling frequency of 25 Hz. We only used lying supine and lying on the right side postures in this study. The dataset is a balanced dataset, including 16 episodes of data collected from 8 participants while performing lying supine and lying on one side postures, where there were eight episodes for each lying posture. Each episode has a length of 7500 samples (300 seconds).

We combined the lying posture episodes from the Class-Act and DAS datasets for the common sensor locations (i.e., chest, right and left wrists, and right and left thighs). We segmented each episode in the DAS dataset into 15 episodes of 500 samples (i.e. 20 seconds). Since the DAS dataset only contained episodes of lying on the back (i.e., supine) and lying on the right side, we performed under-sampling in the DAS dataset prior to combining the datasets to maintain the balance. The final dataset contains 75 episodes of supine (i.e., 26.3$\%$), 57 episodes of prone (i.e., 20.0$\%$), 73 episodes of left side (i.e., 25.6$\%$), and 80 episodes of right side (i.e., 28.1$\%$) for the chest, right and left wrists, and right and left thighs.

Figure 4 shows the feature importance of lying posture tracking as determined by the ensemble tree classifier that is trained on the data from the chest, left thigh and wrist sensors. The importance of the features is one of the outputs of the ensemble tree classification. The y-axis in this figure is the estimation of feature importance from the ensemble tree by summing over the changes in the mean squared error because of splits on every feature and dividing the sum by the number of the branch nodes in the tree [37]. The x-axis shows features 1 to feature 48 as in Table 1. Based on the results, features 4, 7, 10, and 13 are the sets with the highest importance. Table 1 shows that these features are the median, mean, maximum, and minimum of the vertical axis (e.g., x-axis) of the tri-axial accelerometer sensor. Moreover, features 6, 9, 12, and 15 are the second important set of features. Based on Table 1. These predictors refer to features median, mean, maximum, and minimum of the z-axis of the tri-axial accelerometer signal. These results match the observations on the sensitivity of the lying on the left side to the x-axis and lying supine and lying prone postures to the frontal axis (e.g., z-axis) in Figure 3.

Table 2 reports the average and standard deviation of accuracy, balanced accuracy, and F1-score of lying posture detection using the ensemble tree classifier. Overall, the body locations such as the chest and the left thigh, which are less susceptible to nocturnal movement during sleep, demonstrate high performance in lying posture detection, While, sensor locations such as the arms and the wrists are poor in lying posture detection. In particular, the ensemble tree classifiers trained on the data collected from the chest, thighs or ankles achieve 89.8% –96.2% accuracy, 89.8%–96.2% balanced accuracy and, 89.8%–96.2% F1-Score. While, these values drop when the sensor is placed on the upper body parts such as the arms and the wrists (78.6%–89.5% accuracy, 62.9%–84.0% balanced accuracy, and 64.1%–81.6% F1-Score).

The performance decline in the upper body parts is originated from inter-subject variations in the placement of the arms and the wrists and nocturnal movements of them such as bending and rotating while maintaining the same lying posture. The trend in the standard deviation of accuracy, balanced accuracy, and F1-Score across different subjects is also in concordance with the hypothesis that more within-subject variation (e.g., $9.2\%$ to $26.7\%$ standard deviation in accuracy, balanced accuracy, and F1-Score) is observed when the sensor is worn on the wrists and the arms comparing to the chest, thighs and ankles (e.g., $6.2\%$ to $15.6\%$ standard deviation in accuracy, balanced accuracy and F1-Score). Moreover, Figure 5 visualizes the confusion matrix of lying posture detection training the ensemble tree classifier using the Class-Act dataset. The confusion matrices of the classifying the thighs, chest, and ankles data show more promising results than the arms and wrists. In particular, the classifiers trained on the left thigh, right thigh, and chest misclassify 5.5%, 8.1%, 5.5%, and 7.6% of the lying episodes. The misclassification rate increases to 15.2% and 15.7% for the left ankle and left arm locations, and 39.1%, 31.9%, and 28.93% for the right ankle, right arm, and right wrist locations.

Table 4 compares the F1-score mean and CoV of the proposed models AdaLSTM and ET against the state-of-the-art deep learning and feature-based models on the Class-Act dataset. The class-act dataset contains data from three major lying postures including supine, prone, and left side and nine different sensor locations. Since CoV shows the ratio of variation to mean for a metric, a lower F1-Score CoV value represents a more promising model. The linear feature-based classifiers including LDA and SVM obtain $>$88.3% F1-Scores nd $<$0.26 CoV when applied to data from the thighs, ankles and chest, however, their performance significantly drops to 55.0%–82.2% F1-Score and 0.22–0.36 CoV when the sensor is moved to the arms or wrists. The competing deep learning model, LSTM, maintains 83.7%–92.5% mean F1-Score for the thighs, ankles and chest locations, and 51.6%–75.5% mean F1-Score for the arms, and wrists location. AdaLSTM outperforms the competing deep learning and feature-based classifiers over all the body locations except for the Right thigh and right wrist. It achieves 91.5%–98.2% mean F1-Score and 0.06–0.17 CoV for the thighs, ankles and chest body locations. It shows the most promising result when applied to the sensor on the left thigh with 98.2% F1-Score and 0.06. These results demonstrate the power of deep learning and salient of the left thigh in detecting the lying postures for a new subject. We note that neural networks with simpler structures such as LSTM with a fixed learning rate in this paper could not extract useful features and patterns from the raw data automatically from limited data, therefore choosing the proper parameters for the deep learning models is a crucial factor in their performance.

Figure 5, and Figure 6 show the confusion matrix of the ET and AdaLSTM classifiers for the sensor on the thighs, and the wrists locations. As shown, both classifiers mislabel a few of the episodes when applied to data from the sensor on the thighs.

Table 5 shows the mean F1-score and CoV values of lying posture detection on the dataset of four major lying postures including supine, prone, left side and right side and five sensor locations including thighs, wrists, and chest. The results are leave-one-subject-out validated because it is a more realistic validation scenario for the application of human lying posture tracking.

As shown in Table 5, ET and AdaLSTM classifiers achieve a promising range of F1-Score (i.e., $63.3\%$ for the right wrist to $97.0\%$ for the chest) across all the body locations. ET classifier obtains the highest mean F1-score when the sensor is placed on the right thigh (i.e., $97.3\%$) and right wrist (i.e., $78.6\%$) locations, while AdaLSTM achieves the highest F1-Score values for the chest (i.e., $95.3\%$) and the right wrist(i.e., $74.2\%$) among all the algorithms. The linear classifiers such as LDA [7] and SVM [22] achieve higher F1-Score (i.e ranged 90.4%-97.2%) than the proposed models when applied to data collected from the sensor on the thighs and the chest. The linear relationship between the lying posture and accelerometer readings causes the superiority of state-of-the-art for these locations. On the other hand, extra movements of the hands during lying introduce noise and non-linearity to the data collected by the sensor placed on these locations, therefore, F1-Score values of linear classifiers drop to $33.5\%$, and $44.6\%$ for the left and right wrists.

Moreover, the proposed models show lower F1-Score variation to mean ratio comparing to the state-of-the-art techniques. Ensemble tree classifier achieves CoV of $0.14$, and $0.15$, for the right thigh, the chest, respectively, and AdaLSTM achieves CoV of $0.39$, and $0.44$ for the left and the right wrist, respectively. While CoV of the linear models including LDA and SVM increases to the range $0.53-0.80$ for the left and right wrist locations.