Velocity Regulation of 3D Bipedal Walking Robots with Uncertain Dynamics Through Adaptive Neural Network Controller

The HZD framework provides conditions for the existence of provably stable limit walking cycles by enforcing virtual constraints that are invariant through impact. This technique allows synthesizing feedback controllers that realize stable and dynamic locomotion in 2D/3D robots [13, 14].

The mismatch between the mathematical model and the real model of 3D bipedal robots leads to the failure of the designed controllers when applied to the real hardware. To prevent from falling, researchers proposed to use heuristically-designed feedback regulators on top of the HZD-based designed controllers to realize asymptotic stability of the walking cycle  [16, 17].

In practice, two regulators are mostly used to stabilize the walking motion of a 3D bipedal robot: foot placement and torso regulators [2, 11, 17]. These regulators often use decoupled structures, relating certain joints with a specific desired feature—such as hip velocity or torso inclination—of the robot motion. For example, the foot placement regulator adds an offset to the desired swing hip pitch and swing hip roll joints to regulate the longitudinal and lateral walking speeds, respectively, and prevent the robot from falling. These offsets are determined by

where $v_{x}[k]$ and $v_{y}[k]$ are the average longitudinal and lateral speeds of the robot at the middle of step $k$, $v^{d}_{x}$, $v^{d}_{y}$ are the reference speeds, and $K_{p_{x}},K_{d_{x}},K_{p_{y}},K_{d_{y}}$ are manually tuned gains. The readers can refer to  [2, 11, 17] for additional information about the other regulators.

Motivated by the work of [18, 19, 20, 21] in the application of neural networks for the control of nonlinear systems e.g., robotics manipulators, we propose a framework of adaptive neural network-based controllers that compensate the unknown dynamics of the system while tracking the desired velocity. In addition, we show that the proposed network could be seen as a generalization of the heuristic regulators described in Section II-B, and its extension to the case of unknown dynamics. By this, we aim to develop a new general framework for the development of adaptive feedback controllers that render stable limit walking cycles, even when the dynamic properties of the robot change. To validate the proposed method in simulation, we use as our testbed the Cassie-series bipedal robot described in Section II-A.

Typical adaptive controllers rely on approximating the unknown dynamics of the dynamical system as the linear combination of unknown parameters of the system. Then, a control feedback law is computed using the approximation of the unknown dynamics based on the estimation of the unknown parameters, which are updated using a close form update rule. In particular, [22] showed that the unknown dynamics of a robotic manipulator could be approximated as

where $\tilde{\tau}$ is the approximated unknown dynamics, $\mathbf{Y}(\mathbf{q})$ is a matrix of functions dependent on the state of the robot, and $\mathbf{\omega}$ is a vector of parameters. Then, an update law of the form

is used to estimate the unknown parameters online, where $\mathbf{K}$ is a symmetric positive definite matrix, and $\mathbf{s}$ is the filtered tracking error. However, these controllers are restricted to the prior knowledge of the dynamics structure and extensive system modeling and preliminary analysis are required to compute the regression matrix.

With the emergence of learning methods and the ability of neural networks to approximate complex, nonlinear functions, new neural network-based adaptive control algorithms were proposed [21, 19, 20, 18]. The main advantage of neural network-based controllers is that they can virtually approximate any smooth functions, including the unknown dynamics in a robotic system, without the need for computing a regression matrix. Analogous to equation (5), the unknown dynamics of more complex systems, for which a linear parameterization is not accurate enough, can be effectively approximated using neural networks. Then, the nonlinear parameterization of the unknown dynamics can be obtained through neural networks as

where $\hat{{\mathbf{W}}}$, $\hat{{\mathbf{V}}}$ are estimates of the ideal neural network weights provided by some on-line weight tuning algorithms.

In this section, we propose an adaptive feedback regulator for virtual constraints to achieve improved speed tracking under model uncertainties. The proposed regulator will have the following form:

where $\delta\mathbf{y}^{d}_{2}$ represent the modification in the original trajectory to render a stable walking limit cycle, which can be obtained either from offline optimization [14, 17], or offline training of a neural network policy [7, 6].

We will then use an adaptive neural network based feedback controller to determine $\delta\mathbf{y}^{d}_{2}$, as shown in Fig. 2. Inspired by the results of [8], we focus our analysis on three specific outputs: i) the virtual constraints related to the robot’s joints that control the step length in the longitudinal plane (swing hip pitch angle, swing knee), ii) the virtual constraints related to the joints that control the step length in the frontal plane (swing hip roll angle, stance hip roll angle), and iii) the virtual constraints related with the joints that control torso inclination (stance hip pitch angle, stance knee, stance hip roll angle). Let $v_{x}[k]$ and $v_{y}[k]$ be the average longitudinal and lateral speeds of the robot in the middle of step $k$, $v^{d}_{x}$, $v^{d}_{x}$ be the reference speeds, $\phi,\phi^{d},\dot{\phi},\dot{\phi}^{d}$ be the actual and desired torso inclination and angular velocity, we define

where ${\delta\mathbf{y}}^{d}_{2,x}$, ${\delta\mathbf{y}}^{d}_{2,y}$, ${\delta\mathbf{y}}^{d}_{2,\phi}$ represent the modification that compensates the unknown dynamics corresponding to each of the three decoupled systems respectively. A detailed structure for the decoupled controllers is presented in Fig. 3, each of which resembles a feedback PD controller and a feed-forward neural network term to track desired behaviors under uncertainties. This adaptive controller structure enhances the robustness of the controller since it allows the PD term to keep the system stable while the network is learning to compensate changes in the dynamic properties of the robot or the environment. The details of the structure and update rule of the neural networks that compute $\Psi_{x},\Psi_{y},\Psi_{\phi}$ will be discussed in the following section.

Finally, we denote that the structure chosen for the adaptive controllers proposed in (9) allows us to generalize the use of the additional regulators to stabilize the walking limit cycle described in Section II. In particular, when the output of the neural network is zero, the adaptive controller renders the structure of the traditional regulators. However, the generalized structure proposed by this adaptive controller does not restrict the regulation of the joint trajectories to only the swing hip pitch angle (as in the traditional approach) but allows the controller to learn which trajectories should be modified in order to achieve the successful regulation of the desired longitudinal and lateral velocity. Section IV illustrates this point in detail through simulation results on the bipedal robot Cassie under various scenarios.

As shown in Fig. 3, the inputs of the neural networks are the actual and desired values of the longitudinal and lateral velocity, torso inclination, and torso angular velocity. The outputs are the feedforward terms compensating for the unknown dynamics of the decoupled systems. Each network only has one hidden layer with one thousand neurons. Notice that we can think of the hidden layer neurons as creating a random set of basis functions, and the task of the neural network is to learn the weights on those basis functions that provide the desired compensation as a function of the inputs to the matrix. This is a variant of the adaptive controller proposed in [22], and can be given as a simple delta-rule that only applies to the output weights:

where $w_{i,j}$ is the weight from the $i$th hidden neuron to the $j$th output, $E_{j}$ is the error signal for the $j$th output (the difference between the desired and current velocity), $h_{i}$ is the output of the $i$th hidden neuron, and $\gamma$ is the learning rate chosen as $\gamma=1e^{-4}$. This is structurally the same as (6).

It is important to mention that the neural network does not require training a priori since the learning process is performed online. The output weights are initialized to zero, and the input weights are randomly initialized. We use the encoder initialization scheme from [23] to generate input weights with a broad distribution, ensuring a collection of basis functions that covers the space. This general approach has been used as a model of biological adaptive arm control [24] and as a simple benchmark task controlling an inverted pendulum [25], but here we apply it in a very different context and with a different source of the error signal. Previous applications had always been in the domain of torque control and used the output of a PD controller to generate the error signal.

First, we tested the response of the controller when the mass of the torso is increased by approximately $50\%$, $100\%$ and $130\%$. The original mass of the robot’s torso is $10.33\>kg$; then, after the increments in mass the total mass of the torso corresponds to $15kg$, $20kg$, and $23kg$ respectively. Fig. 4 (a) shows the response for these three cases. Interestingly, we can see that the tracking error of the average speed does not converge to zero immediately, but it takes some time until the controller learns to compensate for the unknown dynamics, which illustrates the on-line learning process of the adaptive controller.

We also tested the controller with changes in different dynamic properties like the position of the center of mass of the pelvis, by adding an offset of $+0.05m$, $+0.1m$, and $-0.1m$ in the longitudinal direction. The responses of the adaptive controller for the 3 cases are shown in Fig. 4 (b), where we can see that the controller performs well even under large parameter uncertainties of the robot’s dynamics. Similarly as in the previous test, we can see the learning curve of the controller while the actual walking velocity converges to the desired velocity.

To illustrate the significant contribution of the adaptive controller to the speed tracking and stability of the system in the face of model uncertainties, we compared its performance against two HZD-based controllers. Fig. 5 and Fig. 6 show the velocity tracking performance of the adaptive controller when compared with the classic HZD-based controller for tracking fixed desired velocity [17], and with an HZD-based Reinforcement Learning controller for tracking varying velocities [7]. The robot’s pelvis mass is increased by $5kg$, and an offset of $+0.1m$ is added to the pelvis’ COM in the longitudinal direction. As shown in Fig. 5 and Fig. 6, the existing two HZD-based controllers yield significantly large steady-state tracking error when the dynamics properties of the robot change relative to the model used in the design of these controllers. With the adaptive controller, however, the actual walking velocities converge to a wide range of desired velocities under these changes. More importantly, the convergence is achieved through online learning of the model and does not require a priori training.

To further demonstrate the effectiveness of the adaptive controller to compensate for the unknown dynamics when tracking desired walking speeds in different directions we tested the controller for different walking speeds in diagonal directions. Fig. 7 shows the performance of the controller when tracking diagonal speed with $v_{x}=0.4,v_{y}=-0.2$, and $v_{x}=0.6,v_{y}=0.1$,

Two tests were performed to evaluate the robustness of the adaptive controller, i) external disturbance rejections, and ii) walking on uneven terrain. To evaluate external disturbance rejection, an adversarial force is applied directly at the robot’s torso in both forward and backward directions 2.5 seconds after the test started. Fig. 8, shows the response of the adaptive controller when an adversarial force of $30N$ and a force of $25N$ are applied during 0.1 seconds in the forward and backward directions, respectively. The controller can handle both external forces successfully without falling and, more importantly, recovering the speed tracking performance quickly after the disturbance is applied.

Fig. 9 illustrates the response of the adaptive controller when the robot is walking on uneven terrain. The terrain presents significant irregularities with slopes up to $20$ degrees with a maximum height of $0.4m$. The controller adapts successfully to the different terrain changes and keeps close tracking of the desired velocity throughout the whole test.

In this subsection, we demonstrate the adaptability of the proposed adaptive controller framework to adapt to different operating conditions taking advantage of the generalized structure of the controller. In particular, Fig. 10 (a) shows the speed tracking performance of the adaptive controller for two cases: 1) both hip and knee joints use the compensation provided by the neural network to regulate the walking speed, and 2) the output of the neural network is forced to zero to test the adaptability of the controller to unknown scenarios. The results of this test are shown in Fig. 10 (b), where we see that for case 1, both stance hip and stance knee joints contribute equally to the compensation of the unknown dynamics while tracking the desired speed of the robot. For case 2, since we force the neural network output corresponding to the hip joint to zero, the adaptive controller learns a different way to compensate for the unknown dynamics of the system by only using the compensation available in the knee joint, which demonstrate the adaptation capabilities of the proposed controller.