Classifying In-Place Gestures with End-to-End Point Cloud Learning

Virtual locomotion is a technique by which the user controls the virtual avatar to travel in a 3D virtual environment. It has a significant impact on the users’ sense of presence [36]. Real walking is an immersive and natural approach to navigate in the virtual environment [32, 42], but this method requires the same large physical space as the virtual environment for users to move, which in most cases can not be satisfied because the virtual space can be infinite and the room space is limited [2, 5].

WIP is a natural way of virtual locomotion using leg motions while remaining in place, which makes it easy to be applied in limited physical space. WIP is easy to control the direction and distance of movement [50, 42, 37] and it can also reduce simulation sickness [40].

WIP motion is usually detected by wearable sensors, such as inertial measurement units (IMUs) embedded in HMD, smartphones, and trackers, etc. We can get the information of leg position, linear velocity, acceleration, angular velocity etc by attaching the wearable sensors to the user’s body [14]. Tregillus et al. [40] used the smartphone’s inertial sensors which include a 3-axis accelerometer and 3-axis gyroscope to collect the acceleration signal from the HMD in real-time and used Zhao’s proposed pedometry algorithm [56] to implement VR-Step. Wendt et al. [46] built a model that uses only the bobbing head-track position data to estimate the walking speed, step frequency and direction. Paris et al. [27] used the IMU of the smartphone for Samsung Gear VR to detect whether the user is stepping by setting threshold values of the acceleration signal. Feasel et al. [5] proposed a LLCM-WIP System to reduce the system latency by using a chest-orientation tracker and foot trackers. Lee et al. [20] used the vertical position of the HMD to detect the jogging in place step and considered eliminating the impact of the HMD’s pitch angle on the vertical position. Wendt et al. [47] implemented a Gait Understanding-Driven(GUD) WIP model with a fairly low start-stop latency by attaching two 6-DoF trackers to the user’s shins. With prior knowledge of biomechanics, they implemented a state machine with manual-tuning of parameters, then walking speed was estimated by the proposed GUD WIP model from step frequency and foot position.

Most of the aforementioned methods rely greatly on empirical experience and require the experimenter to be familiar with the biomechanical characteristics of the gait in order to manually adjust the parameters of the WIP model such as threshold values. Moreover, as suggested by [36], there are inter-person and intra-person variations. The intra-person variation refers to the fact that the sensor signal of the same person may change significantly for multiple repetitions of the same experiment due to physical exertion, and the inter-person variation refers to the fact that the sensor signal of the same gesture may vary significantly for different users, due to the differences in height, weight, gender and movement habits among users. These factors lead to significant differences in the effectiveness of manually setting model parameters for different users [14].

Previous data-driven approaches for WIP were usually used to recognize certain patterns of motion. Slater et al. [31] used the position of the HMD as input to a simple neural network to identify whether the user is walking in place or not. Hanson et al. [14] utilized the linear acceleration of the x, y, z axes of the HMD as input and fed it into a convolutional neural network to predict whether the user is currently walking or standing in real-time. Their approach improves the generalization of the WIP model and reduces the impact of inter-person variation across different users on the model. Lee et al. [22] used the position values of the x-axis and z-axis of the HMD labeled with the torso direction of the user to train a model based on the LSTM network to predict the current WIP direction of the user. Lee et al. [21] attached the VIVE tracker to the user’s left ankle and used the continuous x-y-z coordinates and yaw angles of the HMD and tracker labeled with the user’s posture in the gait cycle to train a convolutional neural network for predicting the gait state of the user’s WIP posture. Maghoumi et al. [24] proposed an end-to-end gesture recognition network consisting of an encoder network of stacked gated recurrent units, attention module and classification layers. It directly took as input a sequence of raw 3D gesture vector features collected by sensors to predict gesture labels.

Shi et al. [36] attached a microchip and three pressure sensors to the insole, which were then used to collect pressure signals from users under seven different gait patterns to create a dataset. This dataset was used to train a pattern classifier for real-time classification of user’s gait, which is based on the LSTM. Shi et al. then proposed a Dual-Check Till Consensus (DCTC) method, which dynamically adjusted the segment length of the input data according to the classifier’s probability, to improve the classification accuracy of the LSTM and to reduce the system latency. Their approach can achieve excellent generalization across sensors and individuals.

A point cloud is a set of unordered 3D points, which is the most direct way to represent 3D geometric information [13]. PointNet [28] was pioneer in directly consuming raw point cloud representation without either voxelization or projection. It obtains permutation invariance by a symmetric function. Qi et al. further proposed PointNet++ [29], a hierarchical network that extracts the geometric features of each point’s neighborhood. DGCNN [45] is a graph-based method which creates a dynamic graph in the feature space and uses the EdgeConv layer to learn the edge features of the graph. Guo et al. [10] proposed Point Cloud Transformer (PCT) to apply the traditional Transformer [43] to the field of point cloud learning. They constructed an attention mechanism based on the Transformer for the learning of point cloud features. The Transformer has the order-invariant property, which is favorable for processing point clouds. Zhang et al. proposed to filter 3D point clouds with a point-based network [55]. Vatavu et al. [44] proposed to consider the two-dimensional strokes generated by gestural input as a disordered set of points, and introduced a $P recognizer, a member of the $-family, to recognize the input stroke gestures.

Qin et al.[30] proposed PointDAN, which aligns the local and global features of point clouds together using a Self-Adaptive (SA) node module with an adjusted receptive field. They also used an adversarial training strategy to learn and align features across domains. Achituve et al. [1] applied self-supervised learning(SSL) to point cloud processing. They came up with a multi-task architecture with a multi-head network, where one head is used to train the classifier and one head is trained using the loss caused by the self-supervised Deformation Reconstruction (DefRec) task. SqueezeSetV2 [52] used geodesic correlation alignment and progressive domain calibration to bridge the domain gap between synthetic and real point clouds.

We implement our method on PyTorch. We choose Adam [19] as the optimizer and train the model on an Nvidia TITAN RTX GPU. The learning rate is set to 0.001 and the weight decay is assigned to 0.0001. The batch size is set to 64. Similar to [38], we train an individual model for each selected user in the dataset. Each model is trained for 250 epochs. Note that for a given user, we first remove his/her data from the dataset and use his/her unlabeled data as target and the remaining labeled data as source.

We selected 10 subjects from the dataset marked as S1 to S10. 4 subjects in the dataset were not selected because of missing data for the squatting gesture due to a systematic error during data collection. For these subjects, we evaluated the classification results after training the individual models separately, including the classification accuracy of each class, the mean class accuracy (the average of each accuracy per class), and the overall accuracy (the number of correctly predicted samples divided by the number of total samples to predict). The classification results obtained for each selected subject are listed in Table 2. The results show that the average classification accuracy of keeping squatting is the lowest (85.9%), which is caused by the relatively small amount of data of keeping squatting. During the data collection phase, subjects tended to get up immediately after squatting down, resulting in a relatively short period to maintain the keeping squatting posture. Except for keeping squatting, the average classification accuracy of all other classes is greater than 92.5%. Jogging has the highest mean classification accuracy of 97.7%. Note that a window size of only 180ms enables an accuracy of 94.4% with 12ms model inference time (192ms in total), which satisfies the requirement of real-time applications.

The average confusion matrix normalized over the true (rows) condition for all the 10 selected classification models is shown in Figure 5. The figure shows that stepping forward achieves the lowest recall of 87%, and stepping backward is the second-lowest recall of 90%, and these two postures are easily misclassified as standing. This is probably because users are allowed to do some relaxing actions such as turning in place or resting when standing to better observe the virtual environment. These unintentional relaxed turns are sort of similar to forward movements or backward movements, which leads to relatively low recall of these two gestures. The recall of squatting is also relatively low (90%) due to the relatively small sample number in the dataset.

The results of the model (mean class accuracy and overall accuracy) for different sliding window sizes (number of consecutive frames) are shown in Figure 6. The sequence duration time for generating a sample with a window size of 3 is 90ms, but the corresponding model has the lowest mean class accuracy and overall accuracy, which are only 92.3% and 92.0%, respectively. This is probably because a sample contains only 9 points which represent little spatial information in making gesture classification. When the window size is 6, the mean class accuracy and overall accuracy reach 94.4% and 95.0%, respectively, inducing a moderate sequence duration time (180ms). When the window size is increased to 10 and 16, we do not observe any remarkable improvement in the classification accuracy. Therefore, we choose a window size of 6 in this work.

For latency, we experimentally examine the inference time of PointNet, PointNet++, DGCNN, and PCT+MCD networks on the same sample, and the results show that they all consume around 10ms. Shi et al. [36] also mentioned that the maximum inference time of DCTC is around 15ms. This means the networks have very similar inference time. To achieve an accuracy of about 95%, our method requires a sequence duration of 180ms, while DCTC requires 500ms. The inference time of these networks is significantly smaller than the sequence duration time, which indicates that the delay of ours and other machine learning methods mainly arises from the sequence duration time rather than the inference time. In short, to achieve similar accuracy, our method induces a much smaller delay compared to CNN and LSTM. This is mainly because our method only requires the collection of a few data points in a sample.

Hanson et al. [15] applied convolutional neural networks to WIP research. They classify walking or standing using a simple CNN structure and achieved 98.6% accuracy on the test set with a latency of 680ms (a sample is composed of 40 consecutive triaxial data and data is polled every 0.017 seconds) for the Oculus Go and 92.6% accuracy with a latency of 1020ms (60 consecutive triaxial data) for the Samsung Gear VR. For comparison purpose, we trained a binary classification model using only the samples of walking and standing in the dataset. In contrast, our method is more accurate for classifying standing and walking, with a mean class accuracy of 99.2%, and our method significantly shortens the latency (taking only about 192ms).

LLCM-WIP [5] achieves a start latency of 138ms and a stop latency of 96ms, which is faster than our method. However, LLCM-WIP requires manual adjustment of thresholds for the model, such as the speed-offset threshold. Due to inter-person variations, the threshold of the model needs to be adjusted across users [36] which actually consumes time. In comparison, our approach avoids the need for manual setting of threshold parameters and can personalize the model using only the user’s unlabeled data, which is more flexible and automatic.

DCTC [36] can classify 7 foot patterns based on the LSTM model using the pressure sensor data. They achieve 95% accuracy at 500ms delay and 85.3% accuracy at 300ms delay. In contrast, our method can achieve 94.4% mean class accuracy and 95.0% overall accuracy at 192ms latency, which has overall better performance than DCTC with considering the latency. Moreover, DCTC provides different general models according to the shoe size. By contrast, our method needs to collect unlabeled data for each user to train an individual model.

We compare our method (PCT + MCD) with a series of point cloud classification methods, including PointNet [28], PointNet++ [29], DGCNN [45] and PointDan [30]. PointDan also involves unsupervised domain adaptation. These experiments are all using Adam as the optimization strategy, with a learning rate of 0.001 and a batch size of 64. The cosine annealing schedule is adopted to adjust the learning rate for each epoch. The models are trained using the same 10 subjects’ posture data as shown in Table 2 and the train-test splitting method as described in Section 2.2.1. All models are evaluated on the test set of the target domain. The comparison results are summarized in Table 3.

As illustrated in Table 3, our method achieves the best result of 95.0% overall accuracy and 94.4% mean class accuracy. Compared to PointNet, our method achieves a 3.7% improvement in mean class accuracy and 2.0% improvement in overall accuracy, respectively. Compared with PointDAN which also uses the unsupervised domain adaptation, our method achieves 4.7% gain on mean class accuracy and 3.1% gain on overall accuracy, respectively. We also perform an ablation experiment, eliminating MCD and using only PCT for classification, and obtains a mean class accuracy of 92.2% and an overall accuracy of 93.7%, which are lower than PCT+MCD. This demonstrates the necessity of MCD.

To verify the significance of the improvement of our introduced PCT+MCD, we perform two-tailed t-tests for mean class accuracy of different methods. The t-test results show that the mean class accuracy of PCT+MCD is significantly higher than that of PointNet ($t=-3.56$, $p=.002$), PointNet++ ($t=-4.31$, $p<.001$), DGCNN ($t=-3.90$, $p<.001$), PointDAN ($t=-4.49$, $p<.001$) and PCT ($t=-3.12$, $p=.006$).

15 volunteers participated in this experiment, each of them use their individual model trained by labeled dataset and their unlabeled data to perform the virtual locomotion experiment. Note that if the participant participated in the dataset collection experiment as mentioned in Section 2.1, we remove his/her data from the dataset before training. There are 4 female volunteers and 11 male volunteers, and their mean age is 25.6 and the standard deviation is 2.8. The mean value of their familiarity with VR was 2.8, and the standard deviation was 0.82. None of these volunteers suffered from severe motion sickness.

We utilize 1 square meter of space in the center of the room for moving in place and install two SteamVR Base Stations around the space. Participants wear HTC VIVE Pro HMD and are attached two VIVE trackers to their left and right thighs respectively. Participants then performed a series of movements in place as described in Section 2.1.2.

We develop the virtual locomotion system in Unity3D. Our model is deployed to the server through the torchserve³³3https://pytorch.org/serve/ framework, which allows Unity clients to call the prediction model via HTTP requests. We sample the user’s motion data at 30Hz and input samples to the model every 180ms to predict the user’s gesture class. The user’s forward direction is defined as the average value of the z-axis rotation of the two VIVE trackers. When a backward step is detected, we reverse the forward direction, and when a forward step is detected, we reset the forward direction. When detecting a user walking in place or jogging in place, we calculate the time between steps ($t_{step}$), which is defined as:

where $t_{maxposy}$ refers to the time corresponding to the last maximum peak of the vertical position of the HMD, and $t_{minposy}$ refers to the time corresponding to the last minimum peak of the vertical position of the HMD. A large $t_{step}$ means that the user is walking slowly and vice versa. Then we calculate the virtual locomotion speed of walking ($v_{walking}$) and jogging ($v_{jogging}$ ) with the equation proposed by [40]:

where $[I_{max},I_{min}]$ are respectively the upper and lower bounds of $t_{step}$, and $[V_{max},V_{min}]$ are respectively the maximum and minimum values of velocity. $k$ is a constant greater than 1. When $t_{step}$ is less than $I_{min}$, we set $v_{walking}$ to $V_{max}$, and when the user is detected to be standing, we set $v_{walking}$ to 0. When a user is detected to be jumping, we apply a forward and upward force to the rigid body of the user’s virtual avatar, so that the avatar jumps forward. Finally, when the user is detected to be squatting down or keeping squatting , we lower the avatar’s view and collision body, and when the user is detected to be squatting up, we reset the avatar’s view and collision body.

We selected several scenarios and tasks from the testbed [4] which are relevant to our locomotion gestures for experiments. The first scenario is called Straight Line Movement and consists of 4 tasks. Task 1 requires the user to walk towards the target spot, following the green straight line on the ground as closely as possible. Task 2 requires the user to walk through three progressively smaller circular areas in sequence. Task 3 requires the user to follow a circle, staying inside the circle as much as possible, which moves forward at a variable speed. Task 4 requires the user to dash through a straight corridor as fast as possible. The second direction control scenario is used to evaluate the directional control performance of the locomotion method and includes 5 tasks. First, the user walks along the folded lines marked on the ground to reach several targets, then the user must walk backward through a door while gazing at the target ahead, after that, the user adjusts the appropriate direction through a narrow and curved tunnel, without hitting the walls as much as possible. Next, the user jumps through a staircase and walks across a ramp. Finally, the user crosses a dangerous area on the edge of the roof of a tall building and avoids falling from the roof. The third agility scenario is used to evaluate the agility of the locomotion method. This scenario requires the user to cross a corridor with blocks of various shapes moving towards the user. The user has to control the direction of movement and squat to avoid the oncoming blocks.

Following the instructions [4], we collect some objective metrics which focus on accuracy and latency for each task during the experiment, which are classified as the Accuracy (AC) and the Error-proneness (EP). We recommend that readers refer to [4] for detailed explanation of these objective metrics. The questionnaire provided by the testbed [4] is used to investigate user’s subjective feelings, including Input sensitivity, Input responsiveness, Ease of use, Perceived errors, Appropriateness and Satisfaction, and so on. Then we use the SSQ tool [18] to assess the symptoms of motion sickness caused by our experiment (if any).

For comparison, we choose the improved LLCM-WIP [5] method embedded in the testbed [4]. This method attaches two VIVE trackers to the back of the user’s left and right calves. To be fair, we set its forward direction to be the average of the z-axis value of rotation of the two trackers, which is the same as our method. We also trained a model of DGCNN [45] for comparison purpose. Before the experiment, participants will be assigned one of these three methods without knowing which one it is.