Back-stepping Experience Replay with Application to Model-free Reinforcement Learning for a Soft Snake Robot

The above classical off-policy RL algorithms often face challenges with systems characterized by sparse rewards or challenging tasks with rewards hard to reshape. In such scenarios, RL agents rarely achieve informative standard forward explorations due to a low success rate in reaching goals in complex problems without precise guidance [13]. To address these challenges, we propose a novel Back-stepping Experience Replay (BER) algorithm for tasks with different goals (Alg. 1), designed to enhance the learning efficiency of off-policy RL algorithms. This is achieved by incorporating exploration methods in both forward and backward directions.

The BER algorithm requires at least an approximate reversibility of the system. This means that from a standard transition $(\bm{s}_{t},\bm{a}_{t},\bm{s}_{t+1})$, a back-stepping transition $(\bm{s}_{t+1},\widetilde{\bm{a}_{t}},\bm{s}_{t})$ can be constructed, which is similar to a real transition $(\bm{s}_{t+1},\widetilde{\bm{a}_{t}},\bm{s}_{b,t})$ in the environment, i.e., $\bm{s}_{b,t}\approx\bm{s}_{t}$. The action in the back-stepping transition is calculated as $\widetilde{\bm{a}_{t}}=f(\bm{s}_{t},\bm{a}_{t},\bm{s}_{t+1})$, where function $f$ is dependent on the environment. The approximate reversibility is evaluated by a small upper bound $K$ for all transitions during back-stepping calculation:

There exists a perfect reversibility when $K=0$ with a probably complex function $f$, while an approximate reversibility might be achieved with a slightly larger $K$ and a simpler and solvable function $f$.

The idea of BER is simple yet effective: instead of solely relying on forward explorations (navy blue solid line in Fig. 1) from initial states to goals, which depend heavily on the randomness of forward trajectories to reach these goals, RL agents also navigate backward from the goals to the initial states in the tasks (sky blue solid line in Fig. 1). The standard transitions are sampled from the standard forward and backward exploration trajectories (solid lines in Fig. 1), where the initial states of themselves are included. Then, the back-stepping transitions are calculated based on the standard transitions to constitute the reversed trajectories (dashed lines in Fig. 1), where the virtual goals are set to be the original initial state in their corresponding standard trajectories, such that the reversed trajectories are guaranteed to reach their virtual goals and contribute to the learning efficiency.

During the explorations, the standard and the back-stepping transitions are collected and stored in separate replay buffers for training. A strategy $\mathbb{S}_{t}$ is used to sample the transitions from the standard replay $R_{f}$ with a probability $P_{t,f}$ and from the back-stepping replay $R_{b}$ with a probability $P_{t,b}$, where $P_{t,f}+P_{t,b}=1$. For a system with imperfect reversibility, $P_{t,b}$ gradually drops to zero to distill the transition set for training because of the inaccurate back-stepping transition. The details of BER are shown in Alg. 1. It should be noticed that the operator $\odot$ between the states and the goals also indicates the modification of the sequential data (e.g., the history data) when the back-stepping transitions are constructed.

The BER accelerates the estimation of Q-functions of the RL agent by using the reversed successful trajectories to bootstrap the networks. One interpretation of BER is a bi-directional search method for standard off-policy RL approaches, with a higher convergence rate and learning efficiency. The distillation strategy of the transitions for training needs to be carefully tuned and might be combined with other exploration tricks, to reach an accurate policy learning in the end and avoid the limitations brought by the bi-directional search method, e.g., non-trivial sub-optimum.

In the practical learning tasks, the accuracy and the complexity of the function $f:\widetilde{\bm{a}_{t}}=f(\bm{s}_{t},\bm{a}_{t},\bm{s}_{t+1})$, which calculates the actions $\widetilde{\bm{a}_{t}}$ in the back-stepping transitions $(\bm{s}_{t+1},\widetilde{\bm{a}_{t}},\bm{s}_{t})$, need to be balanced. An accurate $f$ yields better reversibility (with smaller $K$ in Eq. (2)) with more accurate back-stepping transitions and brings less bias and noise, while $f$ itself could be computationally expensive or even unsolvable. On the other hand, a moderate relaxation of the accuracy of $f$ might boost the efficiency of the calculation of back-stepping transitions, when the larger bias and the noises brought by the approximate reversibility (with larger $K$) are managed by the distillation mechanism in BER.

Compared with soft snake robots where each air chamber was controlled independently [20], in this paper, a more compact soft snake robot with snake skins [18] is considered. There are only four independent air paths to generate the traveling-wave deformation of the robot, which enables the robot to traverse complex environments more easily by reducing the number of pneumatic tubing. The body of the robot consists of six bending actuators and each actuator is divided into four air chambers (Figs. 3A, 3D) that connect to four air paths (Fig. 3B). Four sinusoidal waves with 90-degree phase differences and the same amplitude can be used as references of pressures in air paths to generate traveling-wave deformation (Fig. 3C), when the biases of waves induce unbalance actuation for steering of the robot.

Serpentine locomotion is adopted for the movement of the soft snake robot, where the anisotropic friction between the snake skins and the ground propels the robot during the traveling-wave deformation [21]. The artificial snake skins are designed with a soft substrate and embedded rigid scales (Fig. 3E); see [19] for more details.

To describe the serpentine locomotion of the robot, the dynamic model in [21] is adopted, where the body of the robot is modeled as an inextensible curve in a 2D plane with a total length $L$ and a constant density $\rho$ per unit length. The position of each point on the robot at time $t$ is defined as:

where $s$ is the curve length measured from the tail of the robot.

By utilizing a mean-zero anti-derivative $I_{0}$ [22] ($I_{0}[f](s,t)$ $=\int_{0}^{s}f(s^{\prime},t)\differential{s}^{\prime}-\frac{1}{L}\int_{0}^{L}\differential{s}\int_{0}^{s}\differential{s^{\prime}}f(s^{\prime},t)$), the position $\bm{X}(s,t)$ and the orientation $\theta(s,t)$ (the angle between the local tangent direction and the X-axis of the inertial frame) of each point are described as a function of the position $\overline{\bm{X}}(t)$ and orientation $\overline{\theta}(t)$ (Fig. 4) of the center of mass (COM) of the robot:

where $\bm{X}_{s}=(\cos{\theta},\sin{\theta})$ and $\kappa(s,t)$ is the local curvature. $\overline{\bm{X}}(t)=\frac{1}{L}\int_{0}^{L}\bm{X}(s,t)\differential{s}$, $\overline{\theta}(t)=\frac{1}{L}\int_{0}^{L}\theta(s,t)\differential{s}$. The curvature $\kappa(s,t)$ is related to the local pneumatic pressure via:

where $K_{b}$ is the proportional constant and $\Delta p(s,t)$ is the pressure difference between the two air chambers at point $s$.

The anisotropic friction $\bm{f}_{fric}$ between the snake skins and the ground is described as a weighted average of the independent components in different local directions (forward $\hat{\bm{f}}$, backward $\hat{\bm{b}}$, transverse $\hat{\bm{t}}$):

where $\hat{\bm{u}}$ represents the direction of the local velocity, $\mu_{f}$, $\mu_{b}$, and $\mu_{t}$ are the friction coefficients of the snakeskin in $\hat{\bm{f}}$, backward $\hat{\bm{b}}$, and $\hat{\bm{t}}$ directions, respectively. $H(x)=(1+sgn(x))/2$, where $sgn$ is the signum function.

The dynamics of each point of the snake robot is determined by Newton’s second law:

where $f_{inte}$ is the internal force in the robot body, which includes internal air pressure, bending elastic force, et al, with observations: $\int_{0}^{L}\bm{f}_{inte}=0$ and $\int_{0}^{L}(\bm{X}(s,t)-\overline{\bm{X}}(t))\times\bm{f}_{inte}=0$.

Finally, the dynamics for the COM of the robot are derived using the equation (3)-(8) with the observations of $f_{inte}$; see [22] for more details.

Based on the dynamic model of the robot, which simplifies a dynamic system for all points of the robot to a single dynamic system for the COM of the robot, a simulator is designed with proper discretizations and numerical techniques for RL training. The simulation results matched the experimental results [19] of the soft snake robot when different pressure biases were applied for the robot’s steering (Fig. 3F), where the wavy trajectories in the experiments were attributed to the limited number (25) of the tracking markers in the tests.

In this paper, the locomotion and navigation of the soft snake robot is formulated as a Markov Decision Process (MDP) $\mathcal{M}$ and solved with a model-free RL. The $\mathcal{M}$ is defined as a tuple $\mathcal{M}=(\mathcal{A},\mathcal{S},\mathcal{R},\mathcal{T},\gamma)$:

1. 1.

   Action space: Compared with a random Central Pattern Generator (CPG) [7], more constrained sinusoidal waves are used to generate a smoother traveling-wave deformation of the robot for better locomotion efficiency. Besides, the learned controller of the robot is limited to avoid high-frequency pressure change, i.e., the RL agent is only able to generate an action to change the parameters of the waveform at the beginning of each actuation period $[0,T]$ that is same as the period of the sinusoidal waves, and one episode consists of multiple connected actuation periods. The sinusoidal pressure $p_{i}$ for $i$-th channel of the robot is designed as:

   $\begin{split}p_{i}=p_{m}\sin{(c\cdot\frac{2\pi}{T}t_{r}+\frac{(i-1)\cdot\pi}{2})}+b_{i,pre}+(b_{i}-b_{i,pre})\frac{t_{r}}{T}\end{split}$

   (9)

   where $t_{r}\in[0,T]$ is the relative time in one actuation period. $p_{m}$ and $b_{i}\in[0,b_{m}]$ are the fixed magnitude and bias of the sinusoidal waves for the $i$-th channel, respectively, $i\in\{1,2,3,4\}$. $b_{i,pre}$ is a one-step history of the wave bias $b_{i}$ for the $i$-th channel with $b_{i,pre}=0$ at the initial state. $c\in\{-1,1\}$ is a variable to control the propagation direction of the traveling-wave deformation and thus can change the movement direction of the robot.

   The action space $\mathcal{A}$ of the RL agent for locomotion and navigation of the robot is designed as:

   $$\bm{a}=\{b_{a,1},b_{a,2},c\}\in\mathcal{A}$$

   (10)

   where $b_{i}$’s are constructed by $b_{a,1}\in[-b_{m},b_{m}]$ and $b_{a,2}\in[-b_{m},b_{m}]$:

   $$\begin{cases}b_{1},b_{3}=\max(0,b_{a,1}),-\min(0,b_{a,1})\\ b_{2},b_{4}=\max(0,b_{a,2}),-\min(0,b_{a,2})\end{cases}$$

   (11)

   At the beginning of each actuation period, based on the current policy, the RL agent observes the state and generates an action, which specifies the waveform of the pressures in that period to propel the snake robot. The wave design guarantees the continuity of the pressures across different actuation periods to avoid impractical sudden changes in the pressures and the robot’s body shape.

2. 2.

   State space: A goal-conditioned state is used for the learning of the RL agent for adapting to different random targets. Specifically, a relative representation of the snake robot’s position and orientation with respect to the target is used as part of the state (Fig. 4):

   $$\bm{s}=\{\Delta X,\Delta Y,\Delta\theta,\bm{b}_{a,1,pre},\bm{b}_{a,2,pre}\}\in\mathcal{S}$$

   (12)

   where $\Delta X=x_{g}-\overline{X}$, $\Delta Y=y_{g}-\overline{Y}$ denote the relative position of the target to the COM of the snake robot, $\Delta\theta=\theta_{g}-\overline{\theta}\in(-\pi,\pi]$ represents the relative direction of the target to the main direction of the robot, and $\theta_{g}=\arctan(\Delta Y/\Delta X)$ is the angle between the line from the COM of the robot to the target and the X-axis, $\bm{b}_{a,1,pre}$ and $\bm{b}_{a,2,pre}$ are two-step histories of the action $b_{a,1}$ and $b_{a,2}$, respectively, with an initial setting of $\{0,0\}$.

   The velocities of COM of the robot are not included as part of the state because the value of the Froude number $Fr$ [22] in serpentine locomotion of the snake robot is small, indicating that the frictional and gravitational effects dominate the inertial effect. Two-step histories (longer than one step) are introduced to compensate for the omission of the velocity state.

3. 3.

   Reward function: The reward function $r$ is pivotal for the RL agent to learn the desired behaviors. The training objective in this work is to drive the COM of the snake robot to reach a random target as soon as possible, with a preference for serpentine locomotion where the robot approaches the target along its main direction. Therefore, the reward assigned to the agent at time $t$ is designed as:

   $$r_{t}=\begin{cases}\begin{split}&w_{1}\frac{\Delta L_{t}}{\Delta L_{0}}+w_{2}\frac{2\Delta\theta_{r,t}}{\pi}+R_{g},&\Delta L_{t}\leq\epsilon\\ &w_{1}\frac{\Delta L_{t}}{\Delta L_{0}}+w_{2}\frac{2\Delta\theta_{r,t}}{\pi},&\text{else}\end{split}\end{cases}$$

   (13)

   where $w_{1}$ and $w_{2}$ are non-positive coefficients, $R_{g}$ is a large sparse positive success reward once the COM of the robot enters a neighborhood of the target with a radius of $\epsilon$. $\Delta L_{t}=\sqrt{\Delta X_{t}^{2}+\Delta Y_{t}^{2}}$ is the distance between the COM of the robot and target at time $t$, and $\Delta L_{t}=\Delta L_{0}$ when $t=0$. The deflection $\Delta\theta_{r,t}\in[0,\pi/2]$ is used in the reward to allow the robot to approach the target in a backward direction as well:

   $$\Delta\theta_{r,t}=\begin{cases}\begin{split}&|\Delta\theta_{t}|,&-\pi/2\leq\Delta\theta_{t}\leq\pi/2\\ &\pi-|\Delta\theta_{t}|,&\text{else}\end{split}\end{cases}$$

   (14)

4. 4.

   Transition probabilities: The transition probability, $\mathcal{T}(\bm{s}^{\prime}|\bm{s},a)$, characterizes the underlying dynamics of the robot system in the environment. In this study, we do not assume any detailed knowledge of this transition probability while developing our RL algorithm.