Imitation Learning for Variable Speed Contact Motion for Operation up to Control Bandwidth

In this study, we used two Geomagic Touch haptic devices manufactured by 3D systems (Rockhill, SC, USA) as manipulators (Fig. 1, respectively. Sensors, a motor driver, and a microcomputer were built into the robot, and the robot could be controlled by connecting a USB cable to a personal computer. Detailed specifications can be found at https://www.3dsystems.com/haptics-devices/touch/specifications. Two robots were used during the data collection phase, and an autonomous operation phase using an NN model was executed using a single robot. The robot’s joints and Cartesian coordinates are defined as shown in Fig. 1. The model of the robots was assumed to be the same as that in [27]. However, the physical parameters of the robot were different and were identified on the basis of [33]. Note that in this model, we neglected the interference among the axes and treated them as three single-DOF axes. Therefore, inertia, damping, and gravity coefficients were identified as a single-DOF system.

Table I lists the physical parameter values used in this study. The parameters $J$, $D$, and $G$ are the inertia, friction compensation coefficient, and gravity compensation coefficient, respectively. The parameters with subscripts 1, 2, and 3 represent those of the first, second, and third joints, respectively.

This robot can measure the joint angles of the first to third joints and calculate the angular velocity and torque response using pseudo-differentiation and a reaction force observer (RFOB) [34], respectively. Acceleration control was realized using a disturbance observer (DOB) [35], and robustness against modeling errors was guaranteed. A position controller and a force controller were implemented in the robot; these two controllers were composed of a proportional and differential position controller and a proportional force controller, respectively. Here, $\theta$, $\dot{\theta}$, and $\tau$ represent the joint angle, angular velocity, and torque, respectively, and the superscripts $cmd$, $res$, and $ref$ indicate the command, response, and reference values, respectively. The torque reference of the slave controller $\mbox{\boldmath$\tau$}_{s}^{ref}$ is given as follows:

where $\mbox{\boldmath$\theta$}_{s}$ and $\mbox{\boldmath$\tau$}_{s}$ are the slave variables, defined as follows:

In addition, $\mbox{\boldmath$J$}=\textrm{diag}[{J_{1},J_{2},J_{3}}]$. Here, $s$ is the Laplace operator. Additionally, the parameters $K_{p}$, $K_{d}$, and $K_{f}$ are scalar, and represent the proportional position gain, derivative position gain, and proportional force gain, respectively. We set the same control parameters in each axis. The gains were determined by trial and error so that the control bandwidth covered the motion frequencies of a human. Note that we adjusted the proportional-derivative control of the position to have the characteristics of critical damping. Table I shows the control parameters.

Bilateral control is a remote operation technology between two robots. The operator first operates the master robot and then operates the slave robot directly through the master robot [21] [22]. The operation and reaction forces can be independently measured by the master and slave. This controller was implemented to imitate human object manipulation skills. A four-channel bilateral controller was implemented similar to that in [27].

A block diagram of the four-channel bilateral controller in the demonstration (data collection phase) is shown on the left side of Fig. 2. The command values of the slave robot in the four-channel bilateral control are given as follows:

where $\mbox{\boldmath$\theta$}_{m}$ and $\mbox{\boldmath$\tau$}_{m}$ are the master variables defined as follows:

Fig. 3 shows the data collection phase. The two robots were used for data collection based on a four-channel bilateral control, as described in Section II-A. The objective was to generate motion to quickly or slowly erase a line written in pencil. This task simulated contact-rich motions, such as polishing [36], grinding [37], and wiping [38]. Therefore, the operator of the master robot erased the lines using seven different frequencies, that is, 0.61, 0.85, 1.10, 1.22, 1.47, 1.59, and 1.83 Hz. Frequency adjustments were performed using a metronome. The motions at 0.61 and 1.83 Hz were the slowest and fastest motions that the human was able to manipulate the robot, respectively, and for the other frequencies, the human was instructed to divide these frequencies into seven equal parts using a metronome. It was not exactly divided into seven equal parts, as the value was calculated based on the peak frequency of human motion. This trial was conducted three times at paper heights of 3.5, 5.6, and 6.6 cm from the surface of the desk. The height was adjusted by piling up books under a plate for placing paper. A total of 21 trials were conducted. The saved motion data points were acquired over 15 s in each case, and the joint angle, angular velocity, and torque of the master and slave data were stored at 1 kHz. Training data were obtained by augmenting the collected data 20 times by down-sampling at 50 Hz using the technique described in [39].

Additionally, Figs. 4 and 5 show some of the training data of $\theta_{s1}^{res}$ and $\tau_{s1}^{res}$ for a height of 5.6 cm, respectively. From these figures, when the operating speed changes, it can be confirmed that the required motion and force adjustment differ, although the trajectory is similar. When the operation is the fastest, the torque is the greatest because the inertial force is the highest, whereas the torque decreases with a decrease in the operating frequency. However, when the operation was the slowest, the torque was slightly larger to compensate for the nonlinearity of the frictional force. This is a major problem, which makes it difficult to achieve motion generation at variable speeds.

In this study, we use a network consisting of a recurrent NN (RNN). An RNN, which has a recursive structure, is a network that holds time-series information. This network has contributed significantly to the fields of natural language processing and speech processing [40][41] and has recently been widely applied to robot motion planning [42]. However, RNNs are hindered by the vanishing gradient problem, which makes it difficult to learn long-term data. Long short-term memory (LSTM) refers to an NN that can learn long-term inferences [43]. This approach was improved based on the results of numerous studies and was adopted in this study. To extract the feature values from the response variables that do not depend on time-series information, we implemented a convolutional NN (CNN) prior to the LSTM. We expected that the CNN would extract time-independent transformations, such as Jacobian matrices.

The network inputs are $\mbox{\boldmath$\theta$}_{s}^{res}$, $\dot{\mbox{\boldmath$\theta$}}_{s}^{res}$, $\mbox{\boldmath$\tau$}_{s}^{res}$, and the frequency command of the first joint, whereas the outputs are $\hat{\mbox{\boldmath$\theta$}}_{m}^{res}$, $\hat{\dot{\mbox{\boldmath$\theta$}}}_{m}^{res}$, and $\hat{\mbox{\boldmath$\tau$}}_{m}^{res}$ of each joint in the next step. The variables with $\hat{\bigcirc}$ are estimates given by the NN. The frequency command was designed based on the peak frequency values of the first joint angle of the robot, which was calculated using the FFT. Here, the next step indicates a point 20 ms later than the slave data. Autonomous operation was realized by considering the network calculation time required to generate online motion. Thus, the data had 315,000 ($15,000[\rm{ms}]/20[\rm{ms}]\times 21[\rm{trials}]\times 20[\rm{augmentation}]$) input-output samples. Additionally, the weights were optimized using the mean square error between the normalized master value and network output.

Fig. 6 shows the network. The responses $\mbox{\boldmath$\theta$}^{res}_{s}$, $\dot{\mbox{\boldmath$\theta$}}_{s}^{res}$, and $\mbox{\boldmath$\tau$}_{s}^{res}$ of each joint of the slave robot are reshaped into other channels. The reshape was designed to predict the effect of batch normalization (BN) for each unit dimension. Additionally, the mini-batch consisted of 100 random sets of 300 time-sequential samples corresponding to 6 s. The frequency command was manually provided using a keyboard and normalized using max-min normalization. Max-min denormalization was set at the output of the network. In this study, the computer used for training and autonomous operation comprised an Intel Core i7-8700K CPU, 32 GB of memory, and an nVIDIA GTX 1080 Ti GPU.

The right part of Fig. 2 shows a block diagram of the slave robot conducting autonomous execution using the trained NN. In the autonomous operation phase, the demonstrator, master robot, and master controllers are substituted by the trained NN. In this case, the command values are not the true response values of the master, but the estimated values provided by the NN. Fig. 7 shows its detailed schematic. The NNs predict responses of the master in the next step (20 ms after) from the responses of the slave and frequency commands. Then, the master’s responses are commanded to the slave.

It is worth noting that the control system in the autonomous execution phase is the same as that in the data collection phase. Although the input interval into the network was 20 ms, the control period of the position and force controller was 1 ms. Therefore, the control input was updated during every control period.

First, the control bandwidth of the robot was tested. Fig. 8 shows a Bode diagram of the angle response of the first joint ($\theta_{s1}^{res}/\theta_{s1}^{cmd}$) in free motion with zero torque commands. As can be observed from this figure, the control bandwidth where the gain is -3 dB was approximately 2.3 Hz. It should be noted that operations over the control bandwidth are extremely difficult, and motion over the control bandwidth is not a subject of interest in the research field of control engineering.

Prior to performing the main experiment, one of the three conditions described in the introduction was tested. The validity of the predicting command value of the slave, that is, the predicting master response, was evaluated. In this case, the environment was not changed from the data collection phase, the NN was not used, and motion at 2.17 Hz was simply replayed as a command value. The following two command values for the next step were given for comparison:

1. 1.

   the master’s response during the training data (our approach); $\mbox{\boldmath$\theta$}^{cmd}_{s}=\mbox{\boldmath$\theta$}_{m}^{tr},\mbox{\boldmath$\tau$}^{cmd}_{s}=-\mbox{\boldmath$\tau$}_{m}^{tr}$, and

2. 2.

   the slave’s response during the training data; $\mbox{\boldmath$\theta$}^{cmd}_{s}=\mbox{\boldmath$\theta$}_{s}^{tr},\mbox{\boldmath$\tau$}^{cmd}_{s}=\mbox{\boldmath$\tau$}_{s}^{tr}$.

The variables with superscript $tr$ indicate the training data. Figs. 9 and 10 show the experimental results for $\theta_{s1}^{res}$ and $\tau_{s1}^{res}$, where the blue lines represent the original slave responses in the training data. The orange and gray lines represent the responses reproduced using the master and slave responses in the training data, respectively. The orange lines indicate that the response was almost identical to that of the data collection when the master’s next response was used as a command value. However, based on the gray lines, when the next response of the slave was given as a command value, the shape of the response differed significantly from that in the training data. It is evident that the amplitude was smaller than that in the original slave response, and a large phase delay occurred. Given that the motion was rapid, and it was very close to the control bandwidth, the transfer function from the command to the response cannot be 1.

These results clearly show that predicting the master’s response is important for reproducing fast motion. It should be noted that kinesthetic teaching cannot satisfy this condition, nor does conventional imitation learning using bilateral control [23]. As such, temporal reproducibility at high speeds can only be achieved using our approach. Hence, variable-speed imitation learning with precise reproducibility has been made possible for the first time.

The results of the experiment conducted to change the operating speed based on the training data were compared with the results of a motion copying system [28]. In the latter, the data collected at a frequency of 1.22 Hz and height of 5.6 cm were used to reproduce the operation. Given that the motion copying system simply rescales the time axis, it only requires one time series of data for reproduction. To convert the operating speed, the original data were rescaled to fit the target speed data. The training data were rescaled along the time axis of the data using linear interpolation with a zero-order hold.

First, nine and 16 convoluted channels were compared for the implementation of the CNN. The variable-speed range for the 16 channels was wider than that for the nine channels. Hence, the proposed method was implemented using 16 channels. Fig. 12 shows the training loss versus epochs for the 16-channel network. We then selected the model of the 1500 epochs because the loss almost settled down. The learning time was approximately 40 min.

The success rate of the operation was then evaluated. Fig. 11 shows the working area. Given that this robot was not equipped with a camera, it was not possible to completely erase the entire area. In contrast, it is easy to erase the entire area by combining the proposed method with conventional methods using a camera. However, when several methods are combined, it is difficult to evaluate the effectiveness of the proposed method. Therefore, we investigated whether we could erase the arc-shaped area through which the end-effector of the robot passed. When the robot erased more than 90 % of the area inside the red lines, this was defined as successful, and it was inferred by a human. Note that because it is a cyclic motion, it can gradually erase all the lines as time goes by if it could keep operating with stable contact.

Evaluation was conducted using 15 frequency commands: 0.49, 0.61, 0.73, 0.85, 0.98, 1.10, 1.16, 1.22, 1.34, 1.47, 1.53, 1.59, 1.71, 1.83, and 1.95 Hz, for five heights of 3.5, 4.7, 5.6, 6.9, and 7.6 cm from the surface of the desk. Three trials were conducted for each condition, for a total of 225 trials (15 [frequencies] $\times$ 5 [heights] $\times$ 3 [trials]). It should be noted that height information from the desk surface was not given to the robot. Given that the robot was not equipped with a camera, it needed to adapt to the perturbation of the height using only the angle, angular velocity, and torque information. The experiments can be viewed using the following link to a video: (https://youtu.be/2XjbauSGu0s).

Fig. 13 shows the success rate for each height. The blue lines show the success rates of the motion copying system, whereas the orange lines represent the rates of the proposed method. As shown in the figure, the motion copying system performs its task under limited frequencies and heights, whereas the proposed method can adapt to variations in both the speed and height. The success rate was the same or higher than that of the motion-copying system under all conditions. Particularly, given that the motion copying system does not have an adaptation mechanism against a height perturbation, it was significantly less effective at heights of 6.6 and 7.6 cm. Figs. 14-17 show the angular responses of $\theta_{s1}^{res}$ and the torque responses of $\tau_{s1}^{res}$, $\tau_{s2}^{res}$, and $\tau_{s3}^{res}$ for a height of 5.6 cm. The blue lines represent the responses of the motion-copying system, whereas the orange lines show the responses of the proposed method. The red lines indicate the working area. In the motion copying system, within a high-speed range, the amplitude of $\theta_{s1}^{res}$ was too small to meet the conditions shown in Fig. 11. In the low-speed range, the amplitude was too large to remain within the desk. In contrast, in the proposed method, the angular response was almost constant in amplitude and independent of frequency, indicating that it could operate properly within the working range. However, the amplitudes of the torque varied adaptively for different frequencies in the proposed method. These figures clearly demonstrate that the proposed method could achieve almost the same angular trajectory regardless of the frequency, whereas the motion copying system exhibited a strong dependency on the frequency; simultaneously, the proposed method could give adaptive force command values with dependencies on the frequency. Thus, the proposed method can effectively handle frequency-dependent physical phenomena, such as inertial force and friction. It is also worth noting that the reason why the response of the proposed method and the motion copying system differed at 1.22Hz is that the proposed method generated trajectories in real time, then the motion changed according to subtle changes in the environments and the states of the robot. For example, a typical example is that the contact force changed according to the change of the shape of the eraser between the data collection and autonomous operation. The overall success rate of the proposed method was 98.2 %.

Fig. 18 shows the reproducibility of the frequency at a height of 5.6 cm. The horizontal axis shows the frequency command, whereas the vertical axis shows the peak frequency measured using the FFT. Given that the proposed method was 100 % successful, all the peak frequencies of $\theta_{s1}^{res}$ are plotted. Moreover, four additional experiments were conducted to further evaluate the extrapolation performance of the proposed method. In contrast, given that the conventional method had few successful samples, the behaviors that did not meet the conditions in Fig. 11 are plotted. The blue, orange, and green plots show the peak frequencies of the motion copying system, the proposed method, and the proposed method applied during the additional experiment, respectively. The solid line indicates the identity mapping. When the plots are along the line, the reproducibility of the frequency is ideal. In the motion copying system, the operating frequency was adjusted by the designer, and the reproducibility of the operating frequency was consequently high. However, the proposed method could also operate at the command frequency, although there were more variations compared to the motion copying system. When extrapolation was far from the training data, the reproducibility was reduced, although the peak frequency tended to increase with an increase in the frequency command. It is worth noting that the operation at 2.08 Hz was achieved using a 2.69 Hz command, indicating that the operation was faster than the fastest training data at 1.83 Hz. Thus, the proposed method could not only change the operating frequency, but it could also perform the task faster than a human. Additionally, the control bandwidth of the robot was approximately 2.3 Hz, and it was very difficult to operate at frequencies beyond the control bandwidth. It is also worth noting that the proposed method can achieve behaviors close to such a limit even when using imitation learning, which has no explicit dynamic model. To the best of the authors’ knowledge, there is no other imitation learning that can operate at a frequency almost at the limit of the control bandwidth.

In general imitation learning, the objective is to reproduce the teacher data, thus, the evaluation is mostly based on the success rate of the task, and whether a specific physical quantity can be controlled is not the main issue. However, the proposed method not only imitates the motions but also has a frequency command, thus, the actual frequency can be compared with the frequency command during autonomous operation. Therefore, by labeling the actual frequency as a frequency command and storing it with the motion data, the autonomous motion data can be regarded as new teacher data. If online learning, such as Bayesian optimization, is applied to the new teacher data, the accuracy of motion generation can be further improved iteratively. In addition, although it is not the subject of this paper, it is expected that generalization performance against modeling errors of the robot’s parameters can also be treated as the perturbation of environments.

To further demonstrate the generalization performance of the proposed method, a task of slicing a cucumber was conducted using the same NN model and controller. No parameter was changed from the previous experiments. The height of the surface of the slicer was 5.6 cm from the desk surface. A cucumber was fixed at the tip of the robot, and the cucumber was sliced using a slicer. Fig. 19 shows a snapshot of the experiment. As shown in the figure, the robot continued to slice the cucumber. The success rates versus frequency commands are shown in Fig. 20. The evaluation was conducted using 15 frequency commands: 0.49, 0.61, 0.73, 0.85, 0.98, 1.10, 1.16, 1.22, 1.34, 1.47, 1.53, 1.59, 1.71, 1.83, and 1.95 Hz, and three trials were performed for each command. This result shows that fast motions were required to slice the cucumber because it was rigid and required a lot of force for slicing. In other words, the robot had to move quickly and exploit its own inertial force to slice them. In conventional force control theories, the inertial forces of robots are often treated as disturbances. However, using the proposed method, it was possible to realize motions that exploited inertial forces. Particularly, it is very effective to use the inertial force of a robot with a small maximum output torque, such as the robot used in this experiment. Additionally, to make the most effective use of the inertial force, a high-frequency motion is required for high acceleration. It would have been difficult to slice a cucumber without using our method, which can operate the robot to the limit of the control bandwidth.