Hybrid LMC: Hybrid Learning and Model-based Control for Wheeled Humanoid Robot via Ensemble Deep Reinforcement Learning

where $x_{w}$ denotes the traversed position of SATYRR calculated as an average of two wheel encoders, $\theta$ is the pitch angle. $\dot{x}_{w}$ and $\dot{\theta}$ represent their corresponding derivatives. The mass of the body and wheel indicates $m_{0}$ and $m_{w}$, respectively. $u$ is the control input and torque applied to the wheel, $r$ is the radius of the wheel, and $g$ is gravity. The length between the center of the wheel and the center of mass (CoM) of the body is denoted by $L$, $I_{w}$ is the inertia of the wheel, and $I_{0}$ is the inertia of the body.

where $\tau_{LQR}(\bm{s})$ and $\tau_{\phi}(\bm{s})$ are the output action (torque) from the LQR and the hybrid policy $\phi(a|\bm{s})$ at given state $\bm{s}$, respectively. As seen in [15, 16], using prior knowledge of the system (e.g., its model and conventional controller) can aid the RL network in operating within safer bounds as well as increase its sampling efficiency. Conversely, RL policies can assist conventional controller in adapting to various environmental changes by interacting with the world. The proposed Hybrid LMC follows this outline but differs from previously explored residual RL frameworks - instead of a deterministic policy, a stochastic approach with distributional actions is utilized. We assume that a stochastic approach promotes the search - through randomly sampled behaviors - of the nearby action-space for more optimal torques. This ultimately result in better tracking of the desired states and in reduction of residual error, $\Delta\bm{q}=\bm{q}^{des}-\bm{q}$ created by unexpected disturbance, and environmental changes. The detailed procedure of Hybrid LMC is decribed in Algorithm 1. Also we note that our strategy builds its action as the sum of a stochastic policy and the conventional feedback controller (i.e the LQR), unlike BCF where the action is only sampled from the hybrid policy [17].

where $\mu_{H}(\bm{s})$ denotes the mean of action from LQR and $\sigma^{2}_{H}$ is its variance. To acquire a distributional action from a conventional controller, we empirically assume the variance $\sigma^{2}_{H}(=0.4)$ for the LQR. The mean $\mu_{H}(\bm{s})$ is the same with an original action from LQR. We believe that the LQR, $H(a|\bm{s})$, can guide $\phi(a|\bm{s})$ in exploring more realistic torques during the early stages of training as the feedback controller is able to leverage prior information about the model and dynamics.

where $\mu_{\pi_{m}}(\bm{s})$ and $\sigma^{2}_{\pi_{m}}(\bm{s})$ denote the mean and variance of a single RL policy $\pi(a|\bm{s})$. Each RL policy $\pi(a|\bm{s})$ is trained with the use of the SAC algorithm [18] that has achieved state-of-the-art (SOTA) performance in simulated robotic systems by addressing the continuous action problem. SAC was determined suitable here because it is a stochastic policy that chooses an action by sampling from a Gaussian distribution. This enables exploration of a larger state-space and action-space area.

Simulation Setup: All experiments for validating the Hybrid LMC were conducted using MuJoCo [22] simulation which is widely used to evaluate many learning-based methods. We modeled a wheeled humanoid robot, SATYRR using a Unified Robot Description Format (URDF) that has the same physical parameters (Table. I) as a real hardware platform. (Fig. 3)

Experimental Variation: We tested Hybrid LMC on a diverse set of model parameters through changing of mass, friction, gear ratio, and CoM position values. The ranges of each parameter are described in Table. II.

Baselines: Hybrid LMC is compared to the following baselines:
1) LQR controller: LQR controller derived using a WIP model (a conventional feedback controller).
2) Model-free RL algorithms: SAC (stochastic approach) and Deep Deterministic Policy Gradient (DDPG) [23] (deterministic approach) have shown promising results in the continuous action space.
3) Residual RL: Residual RL framework [15] consisting of the sum of a deterministic residual policy and a feedback controller. To benchmark this framework’s perfromance for comparision, we tested DDPG with LQR (DDPG+LQR) and SAC with LQR (SAC+LQR) in our experiments.
4) Bayesian controller fusion (BCF): A hybrid control strategy combining a model-free RL and a conventional controller [17] that motivates a basic structure of Hybrid LMC.

Evaluation Metric: To compare the performance of Hybrid LMC against baselines, we use root mean square error (RMSE) between the errors of respective desired states and position of $x_{w}$, velocity $\dot{x}_{w}$, and pitch angle $\theta$.

Defining State-Action Space and Reward Function: The state $\bm{s}$ at time $k$ is defined as $\bm{s}^{k}=\langle\bm{q}^{k},\bm{\tau}^{k-1},\bm{C}^{k},\bm{\Theta}^{k}\rangle$ and each components are defined as follows:

• •

  State-space vector: $\bm{q}^{k}$ = $\langle x^{k}_{w},\theta^{k},\dot{x}^{k}_{w},\dot{\theta}^{k}\rangle$

• •

  Applied torque vector at the previous time step $t-1$: $\bm{\tau}^{k-1}=\langle\tau^{k-1}_{LQR},\tau^{k-1}_{\phi},\tau^{k-1}_{c}\rangle$

• •

  The desired position and pitch angle:
  $\bm{C}^{k}=\langle C^{k}_{x},C^{k}_{\theta}\rangle\in\bm{q}^{des}$

• •

  History of position error and velocity: $\bm{\Theta}^{k}=\langle\bm{\Theta}^{k}_{x_{e}},\bm{\Theta}^{k}_{\dot{x}}\rangle$ $\bm{\Theta}^{k}_{x_{e}}=\langle\Delta x_{w}^{t-2},\Delta x_{w}^{t-1},\Delta x_{w}^{t}\rangle$, $\bm{\Theta}^{k}_{\dot{x}}=\langle\dot{x}^{t-2}_{w},\dot{x}^{t-1}_{w},\dot{x}^{t}_{w}\rangle$

where the symbol $\Delta$ indicates the error between the desired value and the actual value. The usage of $\bm{\tau}^{k-1}$ and $\bm{\Theta}^{k}_{x_{e}}$ is motivated by previous works [25, 12]. The key to generating an end-to-end (state-to-torque) policy was found in the inclusion and usage of both $\bm{\tau}^{k-1}$ and $\bm{\Theta}^{k}_{x_{e}}$ within the residual RL framework.

where $err(x)_{t}=x^{des}_{w}(t)-x_{w}(t)$ and $err(\theta)_{t}=0-\theta(t)$. Indicating the change pattern of the error denotes $err^{\prime}(y)_{t-1}=$ $y^{des}(t)-y(t-1)$. The scaling factor $K=[0.1,0.1]$ and $\bm{e}^{s}_{t}=[err(x)_{t},err(\theta)_{t}]^{T}$.

Learning the Ensemble Deep Reinforcement Learning: In this work, we use 10 single SAC policy $\pi(a|\bm{s})$ ($M=10$). A single RL policy $\pi(a|\bm{s})$ is a 3-layer multi-layer perceptron (MLP), with input $\bm{s}\in\mathbb{R}^{15}$, and output $u\in\mathbb{R}^{1}$ that represents wheel torque for stabilization. We trained for total of 5,00 epochs with each epoch consisting of a maximum of 4,000 steps. The policy was updated at every 1,000 steps. The size of our replay buffer was $1e^{6}$. The learning rate was set to $1e^{-3}$. The discount factor was set to 0.99. We chose $\rho$, for updating our target network value, to be 0.995. The entropy regularization coefficient value is chosen using an auto temperature adjustment [26]. The batch size is 64 and the Adam optimizer was utilized for all our implementations. The structure of SAC was implemented by referring the OPENAI open source [27]. A single desktop with 1 GPU (RTX3060ti) was used for training RL algorithms. Training takes roughly half day on the desktop machine.

Performance comparison of key components: An ablation study is conducted to investigate and analyze how crucial components of the network affect the performance of Hybrid LMC. Based on our results we believe that feeding previous torque commands $\tau^{k-1}_{LQR},\tau^{k-1}_{\phi},\tau^{k-1}_{c}$ and a history of the states $\Theta_{x_{e}},\Theta_{\dot{x}}$ as an input to Hybrid LMC is critical for achieving better performance, as seen in Table III. Here, Hybrid LMC is trained with the velocity trajectory, Fig. 5(c). Each component has a considerable impact on the convergence of the network and in achieving better performance. We hypothesize that leveraging the history state and the previous torque brings benefits to end-to-end (state-to-torque) learning in the broad residual RL framework, also utilized in Hybrid LMC.

Verifying Hybrid LMC with varied LQR gains: Here we test the dependency of Hybrid LMC on the LQR. The Hybrid LMC was trained using the original gains (described in section IV) but tested using varied LQR gains. From row 1 of Table IV, we see that increasing LQR gains results in similar performance and superiority of the Hybrid LMC. However, decreasing the gains generally resulted in an increase in RMSE error and lead to worse performance. This suggests that high performance of Hybrid LMC is dependant on the tuning and response of the feedback controller during training. In other words, switching the gains or feedback controller in deployment is not recommended. We believe that varying the gains randomly during training may help address this issue [13].