Feasibility-aware Imitation Learning from Observations
through a Hand-mounted Demonstration Interface

Imitation learning methods acquire a control policy using demonstration data, which are collected through a robot operation. Wong et al. proposed a system that enables users to control a mobile robot by a smartphone while viewing images from a camera mounted on the robot. This situation allows for the teaching of both mobility operations and robotic arm manipulation for imitating kitchen tasks [12]. Cuan et al. demonstrated that adding real-time haptic feedback to a teleoperation device improves the efficiency and quality of the data collection and imitated a door-opening task[13]. However, in these imitation learning methods, the learned policy is known to be vulnerable to modeling errors and noise, which can cause the robot’s behavior based on the learned policy to become unstable. This problem is called a covariate shift. Interactive approaches have been proposed to address this issue.

Interactive imitation learning alternates between demonstration and policy learning and acquires a stable policy by utilizing a learned policy for demonstration. Two representative methods are DAgger and DART. In the former approach, the robot moves according to a learned policy, and the operator takes action and collects data only when learning is insufficient [14, 15, 16, 17, 18]. The DART approach injects noise into human operations and prompts human operators to take action to deal with it [19, 20, 21, 22, 23].

These imitation learning methods assume that demonstration data are collected by directly operating a robot. However, operating it requires specialized skills. To enable robots to automate a broader range of tasks, methods are desirable that allow them to learn from demonstrations without requiring such specialized skills.

Imitation learning using human hand demonstrations allows human experts to perform intuitive motion demonstrations. However, the corresponding robot states and actions must be estimated from human demonstration data since the states and actions of humans and robots are different. Some methods use trial and error to estimate actions that imitate the estimated robot state sequences [7, 24, 25]. Smith et al. used a CycleGAN model to replace humans with robots in demonstration videos and learned corresponding robot actions through model-based reinforcement learning [24]. While those studies provide intuitive demonstration, this approach struggles with the physical differences between humans and robots.

Recent research has focused on unifying human and robot end-effectors to simplify imitation [2, 3, 4, 5]. A previous work developed a demonstration interface that unified human and robot end-effectors, enabling through behavior cloning the imitation of tasks like grasping and pushing [5]. Chi et al. also developed a similar demonstration interface that facilitated the learning of generalizable policies [2].

In contrast to those studies, we unify end-effectors using a hand-mounted demonstration interface and provide feasibility visual feedback for demonstrators and robot-executable demonstrations.

In the imitation learning framework, the differences between the demonstrator and robot dynamics are addressed by estimating feasibility, which is utilized for policy learning [26, 27, 28]. Since these methods evaluate feasibility after demonstrations, they may collect many demonstrations with low feasibility. Our method introduces feasibility into ILfO and proposes a method that incorporates it into feedback to the demonstrator and policy learning.

BCO assumes that the environment follows a Markov decision process (MDP). At time $t=1$, initial state $\mathbf{s}_{1}$ follows initial state distribution $p(\mathbf{s}_{1})$, and at time $t$, given state $\mathbf{s}_{t}$ and action $\mathbf{a}_{t}$, subsequent state $\mathbf{s}_{t+1}$ follows state transition probability $p(\mathbf{s}_{t+1}\mid\mathbf{s}_{t},\mathbf{a}_{t})$. The policies of the demonstrator and the agent are denoted as $\pi^{e}(\mathbf{a}_{t}\mid\mathbf{s}_{t})$ and $\pi^{r}_{\phi}(\mathbf{a}_{t}\mid\mathbf{s}_{t})$. While the demonstrator provides demonstrations based on $\pi^{e}(\mathbf{a}_{t}\mid\mathbf{s}_{t})$, during policy learning for the agent, only state sequence $\mathbf{S}^{e}\!=\!\{\mathbf{s}_{t}^{e}\}_{t=1}^{T}$ from the demonstrations is available.

BCO utilizes IDM to predict actions from demonstration state sequence $\mathbf{S}^{e}$. The IDM is trained using state sequence $\mathbf{S}^{r}\!=\!\{\mathbf{s}_{t}^{r}\}_{t=1}^{T}$ and action sequence $\mathbf{A}^{r}\!=\!\{\mathbf{a}_{t}^{r}\}_{t=1}^{T}$ collected when the agent follows arbitrary policy $\pi^{\mathrm{rand}}$ in the environment. IDM $f^{\mathrm{IDM}}_{\theta}$ predicts action $\mathbf{a}_{t}$ using states $\mathbf{s}_{t}$ and $\mathbf{s}_{t+1}$, and assuming that the agent’s state sequence $\mathbf{S}^{r}$ and action sequence $\mathbf{A}^{r}$ follow $p(\mathbf{S}^{r},\mathbf{A}^{r}\mid\pi^{\mathrm{rand}})$, IDM parameters $\theta$ are optimized:

Using optimized parameters $\theta^{*}$, action $\tilde{\mathbf{a}}^{e}_{t}=f^{\mathrm{IDM}}_{\theta^{*}}(\mathbf{s}^{e}_{t},\mathbf{s}^{e}_{t+1})$ is predicted for demonstration states $\mathbf{s}^{e}_{t}$ and $\mathbf{s}^{e}_{t+1}$.

Parameters $\phi$ of agent’s policy $\pi_{\phi}^{r}$ are learned using demonstration state sequence $\mathbf{S}^{e}$ and IDM. When demonstration state sequence $\mathbf{S}^{e}$ follows $p(\mathbf{S}^{e}\mid\pi^{\mathrm{e}})$, the parameters are learned:

To collect demonstration data with a shared hand structure between a robot and a human, we developed a hand-mounted demonstration interface that shares the two-finger opposing structure of the robot hand with the human hand (Fig. 2). This hand-mounted demonstration interface, which was developed with reference to Hamaya et al. [4], allows the demonstrator to grasp the handle and open/close the fingers. The distance between the fingers can be calculated from an encoder attached to the hand-mounted demonstration interface, and the interface’s pose is observed using a motion capture system.

Demonstration data $\mathbf{S}^{e}\!=\!\{\mathbf{s}^{e}_{t}\}_{t=1}^{T}$ from the task execution consists of state $\mathbf{s}^{e}_{t}$, which is composed of hand-mounted demonstration interface pose information $\mathbf{p}^{e}_{t}$ and environmental observation $\mathbf{o}^{e}_{t}$. This method focuses on capturing the robot’s dynamics and assumes that the robot’s actions can be estimated from the sequence of its poses $\mathbf{p}^{r}_{t}$ to acquire a generalizable model for the task. For IDM and FDM learning, the hand pose information from the demonstrations is used to predict the actions and the next hand pose. The training data for IDM and FDM consist of the robot’s pose series data $\mathbf{P}^{r}\!=\!\{\mathbf{p}^{r}_{t}\}_{t=1}^{T}$ and action series data $\mathbf{A}^{r}=\{\mathbf{a}^{r}_{t}\}_{t=1}^{T}$, collected as the robot follows randomly generated motion trajectories within the workspace using tracking policy $\pi^{\mathrm{track}}$. For policy learning, the actions are predicted from the pose transitions included in demonstration data $\mathbf{S}^{e}$, and the policy is learned based on these predictions.

IDM and FDM are trained using data collected when the robot moves randomly within the workspace. The training data for IDM and FDM are collected by generating random motion trajectories within the robot’s operational environment. The robot follows these trajectories using a tracking policy $\pi^{\mathrm{track}}$, collecting a series of robot pose data $\mathbf{P}^{r}$ and action data $\mathbf{A}^{r}$ during the trajectory tracking. Here robot’s pose series data $\mathbf{P}^{r}$ and action series data $\mathbf{A}^{r}$ are assumed to follow $p(\mathbf{P}^{r},\mathbf{A}^{r}\mid\pi^{\mathrm{track}})$.

IDM is a model, $f_{\theta}^{\mathrm{IDM}}(\mathbf{p}_{t},\mathbf{p}_{t+1})$, which predicts action $\mathbf{a}_{t}$ required to reproduce a change in the pose between $\mathbf{p}_{t}$ and $\mathbf{p}_{t+1}$. FDM is a model, $f_{\psi}^{\mathrm{FDM}}(\mathbf{p}_{t},\mathbf{a}_{t})$, which estimates pose $\mathbf{p}_{t+1}$ that results from pose $\mathbf{p}_{t}$ and action $\mathbf{a}_{t}$. Parameters $\theta$ and $\psi$ of IDM and FDM are optimized based on the following equations:

Using the optimized parameters, IDM and FDM predict the action that changes the robot’s pose, $\tilde{\mathbf{a}}_{t}=f_{\theta^{*}}^{\mathrm{IDM}}(\mathbf{p}_{t},\mathbf{p}_{t+1})$, and the robot’s pose at the next time step, $\tilde{\mathbf{p}}_{t+1}=f_{\psi^{*}}^{\mathrm{FDM}}(\mathbf{p}_{t},\mathbf{a}_{t})$.

The feasibility of demonstrated hand poses $\mathbf{P}^{e}\!=\!\{\mathbf{p}^{e}_{t}\}_{t=1}^{T}$ is calculated and fed back to the demonstrator to encourage demonstrations that are easier for the robot to imitate. The feasibility is calculated using action prediction $\tilde{\mathbf{a}}_{t}^{e}=f^{\mathrm{IDM}}_{\theta^{*}}(\mathbf{p}_{t}^{e},\mathbf{p}_{t+1}^{e})$ from the IDM and the predicted robot pose at the next time step using FDM. Feasibility $w_{t}$ at time step $t$ is calculated as follows:

Here $\sigma_{w}$ is a parameter that adjusts the scale of feasibility. Hand pose information $\mathbf{P}^{e}$ from the demonstrated motion is plotted on a graph, and feasibility $w_{t}$ is visualized by coloring the plot according to the values of $w_{t}$ (Fig. 3) to provide visual feedback to the demonstrator.

A policy is learned using demonstration data $\mathbf{S}^{e}$, a trained IDM, and feasibility $w_{t}$ of the demonstration data, where higher feasibility actions are weighted. Parameters $\phi$ of policy $\pi_{\phi}$ are learned using the following equation:

When using learned policy $\pi_{\phi^{*}}$, state $\mathbf{s}^{r}_{t}=\{\mathbf{p}^{r}_{t},\mathbf{o}_{t}\}$, which includes robot’s pose $\mathbf{p}^{r}_{t}$ and environmental information $\mathbf{o}_{t}$, is provided as input to the policy. FABCO’s learning process is shown in Algorithm 1.

The experimental setup for the pipette insertion task is shown in Fig. 4. Its goal is for the robot to insert a pipette, which it is gripping, into a vial. The pipette is 195 mm long, and the diameter of the vial’s opening is 20.0 mm. The robot begins the task while holding the pipette in the start area, and its initial pose is randomized within a range of $\pm$ 15.0 degrees for each of the xyz axes in the Euler angles. The state of this task ($\mathbf{s}_{t}$) is represented by 12-dimensional data consisting of the 6-dimensional position and the orientation data of the robot (or the interface) and the 6-dimensional position and the orientation data of the pipette. The robot’s action ($\mathbf{a}_{t}$) is a 6-dimensional velocity command applied to the position and the orientation of the robot’s end-effector.

In this experiment, we used UR5e from UNIVERSAL ROBOTS as the manipulator and a ROBOTIS HAND RH-P12-RN-UR from ROBOTIS as the robot hand. To measure the pose of the hand-mounted demonstration interface and the robot’s hand, we employed six Flex13 cameras from OptiTrack. The workspace in this experiment, limited by the robot’s range of motion, was 300 mm vertically, 500 mm horizontally, and 450 mm high. The angular range was set to ±30.0 degrees for rotation around the x-axis, $\pm$ 30.0 degrees for rotation around the y-axis, and $\pm$ 45.0 degrees for rotation around the z-axis.

To train IDM and FDM, we generated random motion trajectories within the workspace shown in Fig. 4 and collected training data. Five points were randomly sampled from the workspace, and the robot followed a trajectory that connected them. Each trajectory consists of 50 position and orientation points; the robot followed one trajectory to collect a series of 50 state-action data points. The training data were collected by running the robot for approximately 2.5 hours, obtaining data from 2500 trajectories.

The IDM takes hand poses ${\mathbf{p}_{t-1},\mathbf{p}_{t},\mathbf{p}_{t+1}}$ as input and outputs action $\mathbf{a}_{t}$. The FDM takes hand pose $\mathbf{p}_{t}$ and action $\mathbf{a}_{t}$ as input and outputs the hand pose at the next time step, $\mathbf{p}_{t+1}$. Both IDM and FDM are modeled using neural networks with five hidden layers, consisting of 256, 1024, 2048, 1024, and 256 nodes. ReLU is used as an activation function for the hidden layers; Sigmoid is used for the output layer. The optimizer is Adam, the loss function is L1 loss, and the batch size is set to 256. Out of the 2500 trajectories, 2000 were used as training data and 500 as validation data. The model was trained for 2000 epochs, and we selected the model with the highest prediction performance on the validation data at the end of each epoch.

We collected pipette insertion demonstrations using the hand-mounted demonstration interface . The process of inserting the pipette into the vial is shown in Fig. 6. The demonstrators were informed in advance about the robot’s operational range and speed and were instructed to demonstrate the task in such a way to complete it as quickly as possible within the robot’s allowable range of motion and speed. The initial states of the demonstrations were selected to ensure diversity within the experimental setup. In this experiment, we collected 50 trajectories each for demonstrations with FABCO visual feasibility feedback and demonstrations without visual feedback. After the two types of demonstrations, the participants completed the NASA-TLX[11] to evaluate their workload during the experiment.

The policy model used for all the methods is neural networks with two hidden layers, with 256 and 128 nodes in the first and second hidden layers. ReLU was used as the activation function for both the hidden layers and the output layer. In this experiment, parameter $\sigma_{w}$, which adjusts the scale of feasibility, was set to $0.15$. The optimizer is Adam; the loss function is L1 loss, with a batch size of 256. The model was trained for 2000 epochs, and the model was selected with the highest prediction performance on the validation data at the end of each epoch.

Figure. 6 shows the robot performing a pipette insertion using the policy learned with FABCO, based on demonstrations provided by Subject 1. It successfully imitated the pipette insertion task (Fig. 6).

Table I presents the success rates of the pipette insertion task, tested 30 times for the four methods (FABCO, FABCO w/o weighting, FABCO w/o FB, and BCO) that were trained using demonstrations from four participants. When comparing the success rates, the policy performance clearly improved with the visual feasibility feedback and weighting based on feasibility during the demonstrations. Notably, the visual feasibility feedback significantly improved the performance. Subject 2 tends to achieve a high task success rate regardless of the presence/absence of feedback.

Fig. 7 shows the demonstration trajectory of Subject 4 and the trajectory generated by the learned policy. The demonstrated motion clearly generated a trajectory from the initial position to the pipette insertion position regardless of the presence of feedback. However, the policy learned from the demonstration data collected without visual feedback failed to imitate the demonstrated motion due to error compounding.

Fig. 8 shows the feasibility of the demonstration trajectories with/without visual feasibility feedback, indicating that using visual feasibility feedback helps demonstrate executable motion by the robot. Additionally, a t-test confirmed a significant difference in feasibility between the demonstrations with/without visual feedback.

Fig. 9 illustrates the transition of feasibility over 50 demonstrations by the four participants. Demonstrations without visual feedback show low feasibility and high variance throughout; demonstrations with visual feedback start with low feasibility but improve as the participants adjust their motions based on the feedback, leading to demonstrations with higher feasibility.

The results of the NASA-TLX survey completed by the participants after the experiment are shown in Table II. The NASA-TLX results indicated that while visual feedback slightly increased the demonstrators’ mental demand and effort, it allowed them to perform the demonstrations with a greater sense of task accomplishment. This suggests that although the feedback slightly increased the number of factors the demonstrators needed to consider to collect data that the robot could easily imitate, the sense of contributing valuable data to the robot’s policy learning increased their sense of accomplishment.