Fast User Adaptation for Human Motion Prediction in Physical Human–Robot Interaction

Early studies on human motion prediction were developed based on probabilistic models such as the hidden Markov model [14, 15], Gaussian mixture regression [10], and probabilistic movement primitives [16]. The recent use of deep neural network architectures has resulted in a remarkable improvement in prediction performance. Fragkiadaki et al. [17] first proposed an encoder-recurrent-decoder model structure, which successfully predicted human body movements using a dataset spanning multiple subjects and activity domains. Subsequent studies have focused on utilizing context information as clues for prediction, such as the motion data of nearby people [18], or multimodal responses of the user [19]. In addition, there have been attempts to consider other learning techniques that increase prediction accuracy and robustness. For example, variational autoencoder [20] or adversarial learning [21, 22] frameworks have been adopted in motion prediction.

We focus on a learning framework that enables the proposed model to effectively respond to differences between individuals for human motion prediction, which has not been addressed in the aforementioned studies that employed fixed model parameters for all users. To solve this problem, we present a meta-learning approach that can quickly adapt the prediction model to novel users.

There have been attempts to apply a meta-learning technique to adapt the prediction model to novel tasks. Proactive and adaptive meta-learning (PAML) [23] integrates MAML and model regression networks [24] to learn an effective adaptation strategy. MoPredNet [25] is a method with a parameter generation module that utilizes external memory. These two previous studies [23, 25] focused on predicting human motion across specific categories (e.g., walking, eating, or smoking) without any interaction with a robot.

Our work is distinguished from the prior works in that we consider a situation wherein a human and a robot physically interact. In the pHRI situation of our work (i.e., virtual air hockey environment, described in Section IV-A), the user behavior is strongly affected by the robotic guidance at every timestep, making it difficult to predict the user behavior over a long time horizon because the interacting robot’s guidance over the horizon is already unpredictable. The robotic guidance depends on the opponent’s play, which is obviously unpredictable. In [8], it has been shown that even one-step human motion prediction can improve the user’s task performance under a certain pHRI situation. Therefore, we focus on predicting the user’s movement at the immediate next timestep given the dynamically changing and unpredictable robotic guidance of the current timestep. It is worth mentioning that we have tested and compared the prediction performance when using the data of the past several timesteps as the input and when using the data of the current timestep only, but there was no significant performance difference. Therefore, we decided to use the knowledge at the current timestep for the prediction.

Optimization-based meta-learning is a technique that allows a model to learn a new task quickly via an optimization procedure based on small data samples. A powerful and representative example is MAML [13], a meta-learning algorithm with a dual-structured training procedure consisting of inner and outer loop updates. The training procedure for MAML aims to attain model parameters that can reach task-specific parameters of a new task within a few gradient steps. Since MAML outperformed previous methods, such as the meta-learner with recurrent layers [26], various other optimization-based algorithms have followed, for example, Reptile [27], which simplifies the second-order gradient computation of the MAML; LEO [28], which performs adaptation in the low-dimensional embedding of model parameters; and multimodal MAML [29], which pursues a more diverse task distribution through parameter modulation.

The aforementioned meta-learning algorithms updated all of the parameters of every layer during the adaptation process. However, Raghu et al. [30] discovered that the entire parameter adaptation changed the parameters of the last layer. Therefore, the authors proposed the ANIL algorithm that adapts only the parameters of the last layer while fixing the parameters of the body layers. Even with this simplification, ANIL exhibited a performance comparable to that of the other meta-learning algorithms utilizing the entire parameter adaptation. CAVIA [31] separates adaptive parameters (which are updated during the adaptation and condition the body layers) and shared parameters (which are fixed). By separating these parameters, CAVIA outperformed MAML despite adapting fewer parameters.

Fast user adaption can be a new application of existing meta-learning algorithms. However, the applicability of the existing meta-learning algorithms and a comparison of their performance have not yet been studied. We propose a novel meta-learning framework specialized in user movement prediction after investigating the user prediction performance of existing meta-learning algorithms. Inspired by the CAVIA model, our prediction model consists of shared and adaptive parameters. Shared and adaptive parameters predict general and individual user movements, respectively. Our model responds to movement differences between individual users by adjusting adaptive parameters. In the CAVIA algorithm, the adaptive parameters are updated for each new task from a zero vector. This initialization method can impede the ability to adapt to various tasks within a few gradient steps. Hence, we propose a model structure that determines the effective user-specific initialization of adaptive parameters. In addition, we suggest a meta-loss function for the meta-update that induces the adaptive parameters to handle individual differences exclusively.

The goal of our meta-learning approach is to train a human motion prediction model that can quickly adapt to a new user with small data samples. In a situation wherein a user performs a task while being guided by a robot, the prediction problem can be formulated as follows: to predict $\mathbf{y}$, which indicates the human action at the next timestep, when given an input $\mathbf{x}=\{\mathbf{x}_{s},\mathbf{x}_{r},\mathbf{x}_{h}\}$, which indicates knowledge at the current timestep consisting of $\mathbf{x}_{s}$, interaction state (e.g., state of the cooperative task), $\mathbf{x}_{r}$, robot action, and $\mathbf{x}_{h}$, human action. We let $\mathcal{U}_{i}$ denote the dataset collected from one user of index $i$. $\mathcal{U}_{i}$ consists of the interaction data of the $N_{i}$ timestep length (i.e., $(\mathbf{x},\mathbf{y})_{i}$ pairs of size $N_{i}$).

For the fast user adaptation problem, two batches $\mathcal{D}^{spt}_{i}$ and $\mathcal{D}^{qry}_{i}$ are given, each consisting of $(\mathbf{x},\mathbf{y})_{i}$ pairs of size $B$ ($B\ll N_{i}$) and sampled from the same dataset $\mathcal{U}_{i}$ without overlapping. $\mathcal{D}^{spt}_{i}$ is employed to adapt the parameters of the prediction model $f$ to user $i$, and the accuracy of the $\mathbf{x}\rightarrow\hat{\mathbf{y}}$ mapping of the adapted model $f^{\prime}$ for $\mathcal{D}^{qry}_{i}$ is investigated. For the model training, a meta-dataset $\mathcal{M}^{tr}$ is employed, which is composed of datasets spanning multiple users (e.g., $\{\mathcal{U}_{1},\mathcal{U}_{2},...,\mathcal{U}_{m}\}$). Another meta-dataset $\mathcal{M}^{test}$, collected from users who do not overlap with the users of $\mathcal{M}^{tr}$, is adopted to evaluate the performance of the fast user adaptation on previously unseen users.

Our model parameters consist of $\theta$, which is shared across all users, and $\phi$, which is adapted according to each user. Because $\phi$ exhibits user-specific values after a series of initialization and adaptation processes, we refer to $\phi$ as the user embedding (UE). To implement a prediction model conditioned by $\phi$, we concatenated $\phi$ to a hidden state vector of the model’s middle layer and adopted the concatenated vector as the input of the next layer, which was originally proposed in [31]. In addition, we propose a model structure that determines the effective initial values of user embedding $\phi_{i}$ for a user $i$ based on small data samples $\mathcal{D}^{spt}_{i}$. Accordingly, as illustrated in Fig. 2, the entire structure of our model is composed of two parts: a prediction network that outputs user-specific predicted movements of the user, and an initialization network that outputs the effective initial point of user embedding before adaptation. The shared parameters $\theta$ contains all trainable parameters of the prediction network and the initialization network, except for the user embedding $\phi$ (see Fig. 3(c)).

In the prediction network, three types of inputs (i.e., $\mathbf{x}_{s},\mathbf{x}_{r},$ and $\mathbf{x}_{h}$) were fed into separate multilayer perceptron (MLP) blocks (with parameters $\theta_{s},\theta_{r}$, and $\theta_{h}$, which are subsets of $\theta$) that extract each feature. Subsequently, feature vectors passed through the integrating layers to produce $\hat{\mathbf{y}}$, which is the predicted user movement. The user embedding $\phi$ was concatenated with the input vector of the second integrating layer to allow user-specific prediction.

The initialization network received the entire $\mathcal{D}^{spt}_{i}$ consisting of $B$ pairs of $(\mathbf{x}^{spt},\mathbf{y}^{spt})_{i}$ as the input. To extract the features of the $\mathbf{x}^{spt}$, MLPs that shared the parameters $\theta_{s},\theta_{r},$ and $\theta_{h}$ from the prediction network were employed, and an MLP with parameter $\theta_{h}$ was again adopted to extract the features of $\mathbf{y}^{spt}$. Note that $\mathbf{x}_{h}$ and $\mathbf{y}$ are vectors of the same format that represent human actions. To obtain a representative $\phi_{i}$ corresponding to $B$ pairs of $(\mathbf{x}^{spt},\mathbf{y}^{spt})_{i}$, we first obtained the user embedding candidates and weight values, each corresponding to one $(\mathbf{x}^{spt},\mathbf{y}^{spt})_{i}$ pair, by feeding the feature vectors into a UE encoder and weight encoder, respectively. The weight value indicates the extent to which the corresponding pair expresses the user characteristics. The relative importance of the corresponding pair among the batches was determined by passing the weight value through the softmax function. Therefore, the representative $\phi_{i}$ ($1\times S$, where $S$ is the size of $\phi$) was acquired by matrix multiplication of the weight values ($1\times B$) and $B$ user embedding candidates ($B\times S$). We employed three-layered MLPs for each UE and weight encoder. For the UE encoder, because it is unnecessary to embed different users to have the same bias, we deleted the bias term of the last layer.

The meta-training process of the proposed model is realized in three steps: (1) UE initialization, (2) UE adaptation, and (3) meta-update, as summarized in Algorithm 1 and Fig. 3. User-specific parameters (i.e., $\phi$) are determined via the UE initialization and UE adaptation steps, and the parameters shared across various users (i.e., $\theta$) are learned via the meta-update step.

During the first step, UE initialization, $\phi_{i}$ is obtained by passing $\mathcal{D}^{spt}_{i}$ through the initialization network described in Section III-B. Subsequently, in the UE adaptation step, $\phi_{i}$ is updated to $\phi^{\prime}_{i}$ by backpropagating $\mathcal{L}(\theta,\phi_{i};\mathcal{D}^{spt}_{i})$, which is the regression loss of the prediction network conditioned by $\phi_{i}$ when using $\mathcal{D}^{spt}_{i}$. Either one or a few gradient steps relative to $\phi_{i}$ can be taken. For example, using one gradient step, $\phi^{\prime}_{i}$ can be computed as follows:

where $\alpha$ denotes the inner learning rate. In the case of adopting multiple gradient steps, decaying the inner learning rate by the decay rate $d$ for every update can benefit a delicate adaptation.

In the meta-update step, we update $\theta$ using $\mathcal{D}^{qry}_{i}$, while fixing the $\phi^{\prime}_{i}$ determined by $\mathcal{D}^{spt}_{i}$. The meta-update is performed by overseeing the user-specific prediction results and values of the adapted $\phi^{\prime}_{i}$ from multiple users. The meta-loss we propose consists of three terms expressed as:

where $K$ represents the number of users sampled for the one meta-update, whereas $\lambda_{1}$ and $\lambda_{2}$ are the weights of each loss term. The first term aims to update $\theta$ to reduce the regression loss of $\mathcal{D}^{qry}_{i}$ in the prediction network with adapted user-specific parameters $\phi^{\prime}_{i}$, which allows $\theta$ to infer general human movements across the users. The second term, which is inspired by [28], enables the initialization network to output an effective $\phi_{i}$ close to the adapted $\phi^{\prime}_{i}$. The $stopgrad(\phi^{\prime}_{i})$ denotes that we consider it as a constant, therefore the derivative of $stopgrad(\phi^{\prime}_{i})$ with respect to $\theta$ is zero. The last term encourages the average of multiple $\phi^{\prime}_{i}$, each adapted to a different user, to move toward a zero vector. The common nonzero bias from multiple $\phi^{\prime}_{i}$ indicates the general movement tendency of the users. Therefore, by forcibly reducing the bias, we induce a general tendency to learn by $\theta$. In addition, the $\phi^{\prime}_{i}$ of different users are induced to be disentangled around zero and eventually learned to address individual differences. Taken together, $\theta$ is updated with a gradient step of the meta-loss; therefore,

where $\beta$ indicates the outer learning rate. The prediction and initialization networks were gradually trained by repeatedly performing the three steps using the sampled batch for each iteration. During the evaluation phase, the meta-update step is omitted, and only the UE initialization and UE adaptation processes are performed to achieve user-adapted motion prediction.

To evaluate our meta-learning approach for the fast user adaptation problem, it is necessary to collect a multi-person motion dataset acquired in a situation wherein a human physically interacts with a robot. A representative pHRI situation occurs when a user performs a target task with haptic guidance from a robot. The haptic guidance system, which has been recognized as a promising human–machine interface [5], can be defined as a system in which the control input of the target task is determined by the interaction between the force exerted by the user and the guiding force of the robot [32]. Because users are free to decide how much guiding force they will accept every moment, different users exhibit different responses to the guiding force. Therefore, it is possible to obtain a wide variety of human motion data, which is suitable for evaluating fast user adaptation performance.

We utilized a dataset consisting of motion data from 20 participants performing a target task with haptic guidance, which was collected in our previous work [8]. In the experimental environment, as illustrated in Fig. 4(a), the participants were instructed to play a virtual air hockey game controlled with a haptic device. In the hockey environment, a user receives points by smashing the puck with their paddle and putting the puck into the opponent’s goalpost. Conversely, the user loses points if they fail to defend a puck heading to their goal. Through the haptic device, the participants consistently received a guiding force to assist them; however, they were allowed to choose whether to follow the guidance. The robotic guiding force dynamically changed according to the opponent’s play.

The entire dataset is composed of the following three data types suitable for our prediction model structure, as described in Section III-B. First, the human action data (corresponding to $\textbf{x}_{h}$ and y) consist of 2-D position vectors of the end-effector of the haptic device determined by the user, which was transmitted to the target task as the control input. Second, the robot action data (corresponding to $\textbf{x}_{r}$) consist of 2-D force vectors that the robot exerts on the user. Finally, the state data of the target task (corresponding to $\textbf{x}_{s}$) consist of the 2-D position and velocity vectors of the paddles and the puck in the virtual air hockey environment. Fig. 4(b) presents an example of the interaction process between a user and a robot that occurred on a haptic device. In the timestep $T$, the user action corresponds to the position vector $\mathbf{x}_{h}$ of the end-effector of the haptic device (marked as an orange dot). Simultaneously, the user receives a guiding force $\mathbf{x}_{r}$ (marked as a yellow arrow). The example illustrates the next user action (i.e., the position vector $\mathbf{y}$ at timestep $T+1$) when the user follows the guiding force. If the guiding force does not match the user’s intention, the user is allowed to control the end-effector in their desired direction by applying a force exceeding the guiding force.

A total of 1.52 M timesteps of data (i.e., 1.52 M pairs of $(\mathbf{x},\mathbf{y})$) were collected from 20 participants aged $25.58\pm 2.09$ (mean $\pm$ standard deviation across participants) years. Each participant performed $84.25\pm 3.52$ trials, and the length of data collected per participant (i.e., $N_{i}$) was $76.2\pm 12.7$ K timesteps. The total duration of the data collection per participant was approximately 1–1.5 h, and the details of the procedure are described in [8]. All collected data were normalized. The mean values of each participant’s action (i.e., $\mathbf{y}$) were $-0.056\pm 0.043$ (horizontal direction) and $-0.152\pm 0.053$ (vertical direction).

According to Article 15 (2) of the Bioethics and Safety Act and Article 13 of the Enforcement Rule of Bioethics and Safety Act in Korea, a research project “which utilizes a measurement equipment with simple physical contact that does not cause any physical change in the subject” (Korean to English translation by the authors) is exempted from the approval. The entire experimental procedure was designed to use only a haptic device and a monitor that did not cause any physical changes in the subject.

Evaluation: To evaluate the fast user adaptation performance, we measured the prediction performance for $\mathcal{M}^{test}$, which consisted of user datasets that were not used for training, that is, users in $\mathcal{M}^{test}$ were not included in $\mathcal{M}^{tr}$. We assumed a realistic sampling situation during the evaluation procedure. If $\mathcal{D}^{spt}_{i}$ (i.e., batch for adaptation) and $\mathcal{D}^{qry}_{i}$ (i.e., batch for prediction) were randomly sampled from the entire user dataset $\mathcal{U}_{i}$ as in the training procedure (Algorithm 1), there may be unrealistic cases of predicting the current human motion while utilizing the later motion data within the same episode for adaptation. To prevent this, for the evaluation, we utilized data from different episodes to adapt the model and to validate the prediction, that is, $\mathcal{D}^{spt}_{i}$ and $\mathcal{D}^{qry}_{i}$ consisted of data from separate episodes.

As a metric for prediction performance, we adopted the mean squared error (MSE) between the predicted value and the ground-truth value of the next user action. Five-fold cross-validation was conducted to reduce the effect of the discrepancy between the training and test datasets. Therefore, the 20-user dataset was divided into five sub-datasets consisting of four users each, and a total of five training-validation processes were performed using five $(\mathcal{M}^{tr},\mathcal{M}^{test})$ pairs. The averaged values of the resulting five prediction errors (MSE) were used to compare the learning methods.

Baselines: We set baseline methods, including non-meta-learning and meta-learning approaches. Ahead of both approaches, the zero-velocity baseline [33] was adopted, which assumed that the user maintained the previous action. This helps to understand the prediction performance of the other learning methods at an appropriate scale.

As a non-meta-learning approach, we trained the same structured prediction model using a conventional supervised learning method. For a fair comparison with the adaptation process of the meta-learning approach, we aimed to determine the extent to which performance is improved when the model trained with a non-meta-learning method goes through the parameter update process with $\mathcal{D}^{spt}$ (i.e., fine-tuned). Therefore, we implemented the non-meta-learning baselines in two ways: when the model parameters were fixed and when they were fine-tuned with a few gradient steps, similar to the adaptation steps of the meta-learning approach.

For the meta-learning approaches, we evaluated the most representative methods, MAML and Reptile. In addition, we tested the performance of integrating the model regression network (MRN) [24] into the adaptation process of MAML, which is equivalent to PAML [23]. Our approach to solve the fast user adaptation problem was to separate the adaptive parameters and enforce them to represent only user-specific movements. Therefore, we also considered two other meta-learning baselines, ANIL and CAVIA, that distinguish between adaptive and shared parameters. There is a difference in how each method divides the parameters. Whereas ANIL has the body layers be fixed and the last layer adaptive, CAVIA adopts the adaptive parameters that join as an auxiliary input for the body layers.

Implementation details: All learning-based baseline methods were implemented using our prediction network structure (Fig. 2), except for the user embedding $\phi$. Only the CAVIA method adopts additional adaptive parameters to the body layers; therefore, it can be implemented using the same structure as our prediction network. We set the hyper-parameters of our method and baseline methods to be as similar as possible. For the meta-learning approaches, including our method, the adaptation process was conducted through five gradient steps based on a stochastic gradient descent optimizer. CAVIA and the proposed method that updates $\phi$ (we set the size of $\phi$ to 32) adopted an inner learning rate of 0.05, and the other methods, which directly update the parameters of the network layers, employed an inner learning rate of 0.01, for both training and evaluation phases. Exceptionally, in the training phase of the MAML and ANIL algorithms, a small inner learning rate of 0.003 was adopted, because it exhibited more stable learning. Regarding the meta-update, an Adam optimizer with an outer learning rate of 0.001 was adopted for all methods, and each model was trained for 500 K steps. The batch sizes of $\mathcal{D}^{spt}$ and $\mathcal{D}^{qry}$ were set to 1 K samples. For the methods adopting multi-person data for one meta-update, such as Reptile or our method, the total number of data samples used for one meta-update was maintained at 1 K samples by adopting 200 samples each from five different users. Regarding the non-meta-learning approaches, an Adam optimizer with a learning rate of 0.001 was employed to train the model for 500 K steps, using batches consisting of 1 K samples. In the fine-tuned baseline case, five gradient steps with a learning rate of 0.01 were taken in the same manner as the adaptation process in the meta-learning approaches.