Leveraging Multiple Legacy Wi-Fi Links for Human Behavior Sensing

The RL agent first needs to encode the explored environment into a feature vector reflecting the context information. Therefore, we adopt an observation network $f_{o}$ and a context-aware network $f_{h}$ as the mixed CNN-LSTM network, parameterized by $\theta_{o}$ and $\theta_{h}$ respectively.

The observation network $f_{o}$, composed of three convolution layers, is designed at the beginning of the mixed CNN-LSTM network to effectively reduce the dimension of the data and significantly simplify the training procedure. At the time step $t_{th}$, the agent observes a state:

for $i=1,2$ and $\xi=1,2,\ldots,M\times Q$, where $f_{o}$ is a linear weighted sum function, weights $W_{o}$ and biases $b_{o}$ constitute the parameter $\theta_{o}$.

Furthermore, on the one hand, human behavior sensing should be sensitively context-aware in our situation. On the other hand, multi-LSTM models are capable of learning and recognizing hidden patterns. Therefore, we design a context-aware network $f_{h}$, using three LSTM layers upon the observation network, to be aware of the context information and recognize the hidden patterns. The agent observes both a state $s_{t}$ and its previous hidden states $h_{t-1}$ as inputs of the context-aware network, then produces its current hidden states $h_{t}$:

Action $U$ is the action space covers all the combination of Wi-Fi links, e.g., all possible link groups formed, and represents a set of discrete actions that our agent can take. At time step $t_{th}$, $u_{t}\in U$ decides a Wi-Fi link group from the set of possible actions according to the state $s_{t}$.

In response to the selected action $u_{t}$, function $\mathcal{T}:S\times U\rightarrow S$ maps a state ${s}_{t}$ into a new state ${s}_{t+1}$:

Then the agent feeds the feature vector $h_{t}$ into the following policy network to generate a proper action from the action space. This action adjusts the link group that the agent decides at the next time.

The policy network is composed of a fully connected layer parameterized by $\theta_{u}$ and uses a sigmoid activation function to output the probability value of each action.

Clearly, the generation of an action indicates that the agent decides the link group corresponding to the action. The policy is as follows:

where $\pi\left(u_{t}|h_{t};\theta_{u}\right)$ is the action probability distribution, $sigmoid\left(\cdot\right)$ is a sigmoid function, ${W_{u}}$ and ${b_{u}}$ are the weights and biases, which together constitute the parameter $\theta_{u}$. The action with the highest probability, i.e., $u_{t}^{*}=argmax_{u_{t}}\pi\left(u_{t}|h_{t};\theta_{u}\right)$ is estimated as the proper action.

When the link groups decided by agent keep unchanged after performing a series of actions, we consider the link group as the effective link group $\Gamma_{lb}\left(t\right)$ at this time. A classification network will emit the prediction based on the $\Gamma_{lb}\left(t\right)$.

The CNN-based classification network $f_{p}$ is parameterized by $\theta_{p}$. Supposing there are a total of $\Omega$ links in $\Gamma_{lb}\left(t\right)$, for each link $\omega$ ($\omega=1,2,...,\Omega$), we firstly obtain the two CSI streams $x_{\omega,t}^{i}$ $(i=1,2)$ as mentioned before. Then we use Discrete Wavelet Transform (DWT) [29] to extract the temporal-frequency features $y_{\omega,t}^{i}$ $(i=1,2)$ from CSI phase differences and amplitudes. Based on this, we extract the DWT spectrum images at the ${t_{th}}$ time step from the link ${\omega}$, defined as:

for $i=1,2$ and $\omega=1,2,\ldots,\Omega$.

As the input data of the CNN-based classification network, the DWT images are then processed in convolution layers, dropout layers, maximum pooling layers and fully connected layers. The detailed architecture of the CNN-based classification network is presented in Section V. Note that we deploy the Rectified Linear Unit (ReLU) function in each convolution layer. The last fully connected layer uses a softmax function to output the probability value of each class.

The predicted distribution according to link $\omega$ is:

for $i=1,2$ and $\omega=1,2,\ldots,\Omega$.

We then calculate the average predicted distribution over all the $\Omega$ links in $\Gamma_{lb}\left(t\right)$ as the final predicted distribution:

The reward will be calculated from the predicted distribution $p_{t}$, taking into account the performance of link selection. In this paper, we propose a strategy which is inspired from recent work in the area of computer vision where authors in [50] have addressed a problem of video frame selection for lowering the complexity of video processing. Albeit driven by completely independent applications and data scenarios, the problems of frame selection and Wi-Fi link selection bear some interesting analogies in that the objective is to identify a sub-sampled version of an original measured data that capture the greatest amount of relevance to a given data-drive classification problem. The goal of the reward function is to push the agent toward finding an effective link group that will gradually improve the classification accuracy. In other words, the reward is set to provoke the largest possible prediction gains (in identifying the correct ground-truth class) as compares with the previous iteration [50]. Hence, we set the reward received by the agent at the ${t_{th}}$ step as:

where $p_{t,\delta}$ is the probability value of class $\delta$ at step ${t_{th}}$, and $gt$ represents the ground-truth label of the behavior.

Inspired by the REINFORCE [54] algorithm of policy gradient, our agent objective is defined as:

where $G$ is a long-term discounted reward. Since the link selection is considered as a sequential decision-making problem, a long-term discounted reward is more suitable here, i.e., the further the reward, the lower the contribution to the current step.

To be specific, the discounted return at the ${t_{th}}$ step can be expressed as:

where $\psi$ is the discount rate. As can be seen from Equation (12), the parameter gets the most movement in directions, so that the favorable action can get the highest reward.

We aim at learning the parameters $\theta_{\pi}$ which can maximize Equation (11). The gradient of $J\left(\theta_{\pi}\right)$ is:

As the dimension of the action space is high, Equation (13) results in a non-trivial optimization problem. Therefore, according to REINFORCE, a Monte-Carlo sampling method is used to estimate the gradient:

where $K$ is the number of samples. Via Monte-Carlo stochastic gradient descent, we can minimize the loss function, i.e., Equation (15), to updates the parameters $\theta_{\pi}$.

We define the loss as the mean of binary cross entropy loss for each action in the action space:

where ${u}_{t,\xi}$ is the probability of ${\xi_{th}}$ link at the ${t_{th}}$ step, $\hat{u}_{t,\xi}$ is the true value of ${\xi_{th}}$ link at the ${t_{th}}$ step, which can only be 0 or 1. Therefore, the loss of the agent is a weighted sum of the two losses:

where ${\lambda}_{1}$ is a constant.

We also define the loss as the mean of binary cross entropy loss for each class:

where $C$ is the number of classes, ${p}_{\delta}$ is the probability of ${\delta_{th}}$ class, $\hat{p}_{\delta}$ is the true value of ${\delta_{th}}$ class, which can only be 0 or 1.

The overall objective of our model is to minimize the loss function:

where ${\lambda}_{2}$ is a constant.

We explore the relationship between different types of CNN-based classification networks and the overall accuracies of human activity recognition in the five cases. The overall classification accuracies of Case a $(a=1;2;3;4;5)$ under CNN c $(c=1;2;3;4)$ are expressed through a radar graph (Figure 4) firstly. Through the radar graph, we can see that the performance of Case 1 is the best, and the performance of Case 5 mostly exceeds that of Case 2 even if it uses less CSI data.

Then we draw the accuracy curves of different cases with a decreasing number of CNN network layers, as shown in Figure 5. It can be seen that the accuracy of Case 1 declines the slowest and remains at the highest level from Figure 5. We can also see that the accuracy of CNN 2 $\bullet$ Case 1 is much higher than that of CNN 3 $\bullet$ Case 3, the accuracy of CNN 3 $\bullet$ Case 1 is comparable to that of CNN 4 $\bullet$ Case 3, which shows that the link selection framework can utilize a simpler network to achieve higher accuracy.

Our framework can significantly outperform the baseline methods. Even if only one single link is decided (Case 5), the accuracy is higher than using two orthogonal links (Case 2). Furthermore, the numbers of Wi-Fi links selected by our agent under CNN c $(c=1;2;3;4)$ are 2.90, 3.14, 2.49 and 2.35, respectively.

Due to the limited space, in what follows below we only focus on CNN 4, as it seems to give the best compromise between complexity and performance.

We examine the characteristics obtained by our proposed framework that underpins the successful performance of the agent in the Wi-Fi-based human activity recognition. The t-SNE algorithm is a technique developed for the visualization of high-dimensional data [56]. As expected, the t-SNE embeddings tend to map the characteristics of perceptually similar activities to nearby points. Thus we can utilize the t-SNE embeddings to estimate the performance of the human activity recognition. As shown in Figure 6, we plot the t-SNE embeddings transformed by the last layer features of the CNN classification network (CNN 4) in different cases.

The higher performance achieved by our method (Case 1) appears clearly from Figure 6(a). In that plot, the embeddings associated with each one of the 5 activities appear as fairly compact and well distinct groups of points, hence helping the classification task.

In contrast, the other methods result in embeddings that show up as groups of points with significant overlap or tend to be more scattered. This effect is particularly pronounced in Case 4, shown in Figure 6(d), where it is seen that totally random selection of one of the Wi-Fi links gives the worst performance. In comparison the selection of just one link but based on our RL strategy (Case 5) gives a clear improvement over Case 4, as see in Figure 6(e). Interestingly, Case 3, which systematically exploits all radio links performs better that a single random random link approach but not as well as a carefully selected subset of links, as shown in Figure 6(c) because some of the links tend to be more noisy or carry data less relevant to the classification task at hand, hence tend to confuse the network. Finally Case 2 corresponds to a widely used approach to heuristically select two radio links (based on orthogonality [35, 29, 31]), but again results in greater confusion probability as shown in Figure 6(b) when compared to our Case 1 in Figure 6(a).

To evaluate a fine-grained performance, we present the confusion matrix of accuracy for each case under CNN 4 in Figure 7.

Case 1: Figure 7(a) shows the confusion matrix of accuracy for CNN 4 $\bullet$ Case 1. Obviously, our proposed framework outperforms all others, reaching a recognition rate of 96.50$\%$. To be specific, the accuracy of each activity is 99.38$\%$, 98.75$\%$, 99.38$\%$, 93.75$\%$ and 91.25$\%$, respectively. The accuracy of each activity ranges from 91$\%$ to 100$\%$, indicating that the effective link group $\Gamma_{lb}$ decided by our agent can not only accurately but also comprehensively classify the overwhelming majority of corresponding activity.

Case 2: Figure 7(b) shows the confusion matrix of accuracy for CNN 4 $\bullet$ Case 2. The overall recognition accuracy for all activities in Case 2 reaches 91.75$\%$. Although Case 2 has a similar recognition accuracy to Case 1 in most activities, it has a poor recognition effect for “Walk” and “Sit”. As reflected in existing papers [35, 40, 29], Case 2 is applicable to some daily activities. However, it does not guarantee that the two orthogonal Wi-Fi links closest to the human location can extract the optimal activity characteristics in a legacy network of multiple APs. Especially in the application of wireless signals for human sensing, it is more important to choose links flexibly than to use fixed links.

Case 3: Figure 7(c) shows the confusion matrix of accuracy for CNN 4 $\bullet$ Case 3. The overall recognition accuracy is 95.13$\%$, which is lower than that of Case 1. It can be seen that, “Sit” samples are more likely to be mistaken for “Bend”, which may be due to the fact that the duration of the two activities are quite similar. Some Wi-Fi links may not be sensitive enough to small differences in these characteristics, so such links need to be eliminated.

Case 4: Figure 7(d) shows the confusion matrix of accuracy for CNN 4 $\bullet$ Case 4. We can see that randomly selecting a link is the least effective way to classify activities in all cases, with the overall accuracy of 85.75$\%$.

In conclusion, it is not feasible to use too much or too little information without filtering.

Case 5: Figure 7(e) shows the confusion matrix of accuracy for CNN 4 $\bullet$ Case 5. This case performs slightly worse than Case 1 and Case 3, but better than the other two cases, with the overall recognition accuracy of 92.13$\%$. This illustrates, on the one hand, the limitation of using a single link for human sensing and, on the other hand, the effectiveness of our framework.

Then we calculate the human activity recognition accuracy of different locations under the five cases, as shown in Figure 8.

Case 1: Figure 8(a) shows the recognition accuracy of the 16 locations under CNN 4 $\bullet$ Case 1. The recognition accuracy of each location reaches more than 90%, among which the lowest accuracies are in Location 9, but it could still reach 90%. In contrast, in other cases (except Case 3), the recognition accuracy in some locations are less than 90%, or even less than 80%. We also note that the performance of Case 1 is better than that of Case 3, as the accuracies of Case 1 over all locations distribute uniformly and the accuracy of each location is generally higher than that of Case 3. This indicates that our framework can select the effective link group ($\Gamma_{lb}$) for different locations ($l$), thus effectively improving the recognition accuracy.

Case 2: Figure 8(b) shows the recognition accuracy of the 16 locations under CNN 4 $\bullet$ Case 2. In this case, the recognition accuracy of each location is lower than Case 1, especially the Location 8, Location 9, Location 10, Location 11, Location 13, Location 15, whose accuracies are lower than 90%. This shows that the two orthogonal links are not enough for each location to get the optimal sensing results. In addition, it shows that our framework can effectively improve the recognition accuracy of the difficult areas.

Case 3: Figure 8(c) shows the recognition accuracy of the 16 locations under CNN 4 $\bullet$ Case 3. In this case, the recognition accuracies of most locations are lower than Case 1. Among them, the locations with the worst recognition accuracies are Location 9, Location 11 and Location 13, all are 90%. This indicates that there are some redundant links that affect the recognition accuracy, i.e., the links themselves do not contain activity-related features.

Case 4: Figure 8(d) shows the recognition accuracy of the 16 locations under CNN 4 $\bullet$ Case 4. This case has the worst performance, which indicates that selecting a single link randomly has obvious disadvantages.

Case 5: Figure 8(e) shows the recognition accuracy of the 16 locations under CNN 4 $\bullet$ Case 5. The performance of this case is slightly worse than Case 1 and Case 3, but better than Case 2, illustrating the necessity of link selection. In addition, for some locations, such as Location 2, Location 9 and Location 16, the recognition accuracy is lower, indicating that using one single link for recognition is not enough for these locations, i.e., utilizing only one single link for human sensing has limitations.

To further evaluate the effectiveness of our framework, we make the agent select a fixed number of links in $\Gamma_{lb}$ and then calculate the recognition accuracy. That is, our agent decides the best two or three links for classification each time. The experiment results under different layers of CNN-based classification networks are shown in TABLE X. The results demonstrate the effectiveness of our framework.

Finally, we evaluate the computational cost of the five cases using different layers of CNN-based classification networks. We calculate the average test time over all the test samples and the results are shown in TABLE IX.

For one thing, the link selection process takes about 15$ms$, which is still in the acceptable range. Further, our framework brings a high gain in accuracy, which illustrates the significance of our framework.

For the other thing, the time required for the link selection process is constant. Because of the fixed numbers of links used, the test time of the classified network in Case 2-Case 4 increases with the complexity of the network. Therefore, in more densely deployed Wi-Fi environments, our framework will be more advantageous.