PIE: Parkour with Implicit-Explicit Learning Framework for Legged Robots

Vision-guided locomotion plays a pivotal role in enhancing the autonomy and adaptability of legged robots. Traditional approaches typically decouple this problem into two components: perception and controller. The perception component translates visual inputs into elevation maps or traversability maps to guide the robot’s locomotion. Meanwhile, the controller component employs either model-based methods [13, 14] or RL methods [8, 15]. Some approaches bypass explicit map construction, wherein both perception and controller utilize learning-based techniques to generate hierarchical RL methods [16, 17]. Most of these methods are primarily used for robot navigation and obstacle avoidance, rather than flexible adaptation to complex terrains, as the decoupling process ultimately results in information loss and system delays. Recently, vision-guided RL methods have generally employed end-to-end control systems, showing great promise in traversing complex terrains. For example, Agarwal et al. [10] designed a two-stage learning framework where a student policy, guided by a teacher policy with privileged information inputs, directly predicts joint angles based on depth camera inputs and proprioceptive feedback. Yang et al. [11] proposed a coupled training framework utilizing a transformer structure to integrate both proprioception and visual observations. Leveraging a self-attention mechanism, it fuses these inputs, enabling autonomous navigation in indoor and outdoor environments with varying obstacles. Yang et al. [12] utilized a 3D voxel representation with $SE(3)$ equivariance as the extracted feature from visual inputs, achieving precise understanding of terrain.

In the context of parkour, constructing precise environmental understanding through exteroceptive and proprioceptive sensors in highly dynamic situations to achieve agile, nimble and robust movement is paramount. Hoeller et al. [1] described a hierarchical pipeline for navigation in parkour terrains. However, when encountering unstructured terrain, occupancy voxels obtained from an encoder-decoder architecture often imply incorrectness and mistrust, leading to inappropriate responses from navigation and locomotion modules, along with high training costs and low scalability. Zhuang et al. [2] proposed a multi-stage method leveraging soft/hard constraints to accelerate the training process, enabling the robot to learn traversing various terrains directly from depth images. However, its privileged physics information is strongly associated with the geometric properties of obstacles in simulation. This makes it challenging for training the robot to cope with terrain that cannot be described solely by geometric properties. Cheng et al. [3] adopted a framework akin to [10]. Differently, it introduces waypoints into the teacher policy’s privileged inputs to guide the student policy to autonomously learn heading, but it is the manual specification of waypoints based on terrain that imposes considerable limitations on this method. Futhermore, the two-stage training paradigm used by all aforementioned parkour works results in information loss and performance degradation of the deployed student policy. Meanwhile, these works primarily focus on explicit terrain estimation distilled from the teacher policy, and lack the implicit estimation of the surroundings, thereby hindering the robot’s maximal performance capabilities.

As shown in Fig. 2, the PIE framework comprises three sub-networks: actor, critic and estimator which will be elaborated upon in Section III-B. An asymmetric actor-critic architecture is adopted to simplify the two-stage training paradigm into a single stage [5]. The actor network receives inputs solely from proprioception and exteroception data obtainable during deployment, whereas the critic network can incorporate additional privileged information. The optimization process consists of two concurrent parts: the optimization of the actor-critic and the optimization of the estimator. In this work, the actor-critic is optimized using the proximal policy optimization (PPO) algorithm [18].

The vectors to be estimate, $\hat{\mathbf{v}}_{t}$, $\hat{\mathbf{h}}_{t}^{f}$, $\mathbf{z}_{t}^{m}$ and $\mathbf{z}_{t}$, can be divided into implicit and explicit estimations on two levels.

On the level of understanding the robot’s state and surroundings, a blind robot can implicitly infer its state and surrounding terrain solely through robust onboard sensors by estimating the successor proprioceptive state [5, 21]. However, it must interact with the environment and then adjust its inference based on immediate feedback. In the parkour scenario, the robot must anticipate the terrain ahead. For example, in high or long jumps, the robot needs to gather momentum in advance to execute liftoff successfully, which can only be achievable through exteroceptive sensors providing additional information. However, solely explicitly estimating terrain using laggy and noisy depth cameras is not sufficiently reliable, especially near edges where the risk of stepping off is high [2, 3]. Hence, we propose a multi-head auto-encoder mechanism to integrate implicit and explicit estimation of the robot’s state and surroundings.

The encoder module has two inputs: temporal depth images $\mathbf{d}_{t}^{H_{2}}$ and temporal proprioceptive observations $\mathbf{o}_{t}^{H_{1}}$. The depth observations in the near past are stacked in the channel dimension to form $\mathbf{d}_{t}^{H_{2}}$, while the proprioceptive observations in the near past are concatenated to form $\mathbf{o}_{t}^{H_{1}}$. In this work, we set $H_{1}=10$ and $H_{2}=2$. $\mathbf{d}_{t}^{H_{2}}$ is processed by a CNN encoder to extract depth feature maps, while $\mathbf{o}_{t}^{H_{1}}$ is processed by an MLP encoder to extract proprioceptive features. To facilitate cross-modal reasoning between visual tokens from the 2D depth feature map and proprioceptive features, we employ a shared transformer encoder to further integrate the two modalities. Since the robot is equipped with only one egocentric depth camera, it cannot directly perceive terrain beneath or behind its body visually. Therefore, the outputs of the transformer, which are the depth and proprioceptive features after cross-modal attention processing, are concatenated and fed into a GRU to generate memories of the state and terrain. The outputs of the GRU are then served as the policy network’s input vectors. One of these vectors is a encoded height map estimation vector $\mathbf{z}_{t}^{m}$, which is decoded into $\hat{\mathbf{m}}_{t}$ to reconstruct the high-dimensional height map $\mathbf{m}_{t}$, serving as the explicit estimation of the surrounding terrain. Another vector is a purely latent vector $\mathbf{z}_{t}$. Instead of using a pure auto-encoder, following prior work [5], we adopt a VAE structure to extract features for $\mathbf{z}_{t}$. Namely, we employ the KL divergence as the latent loss. When decoded alongside other vectors with explicit meanings, it reconstructs $\hat{\mathbf{o}}_{t+1}$ to represent the robot’s successor state $\mathbf{o}_{t+1}$, encapsulating implicit information about the robot’s state and surroundings. We use mean-squared-error (MSE) loss between the decoded reconstruction and the ground truth for both explicit and implicit estimations. This constitutes the first level of implicit-explicit estimation.

The second level concerns whether the vector is an encoded latent vector or an explicit physical quantity. $\mathbf{z}_{t}^{m}$ and $\mathbf{z}_{t}$ are encoded latent vectors followed by a decoder for reconstruction. Through this compression and dimension reduction, noise influence is reduced, resulting in greater robustness. For the explicit physical quantities, $\hat{\mathbf{v}}_{t}$ and $\hat{\mathbf{h}}_{t}^{f}$ are explicitly estimated to prioritize velocity tracking during training and provide relevant foot clearance information crucial for understanding terrain in parkour scenarios. The loss for this explicit physical quantities estimation part is also mean-squared-error (MSE) loss. This constitutes the second level of implicit-explicit estimation.

The overall training loss for our estimator is defined as

where $q(\mathbf{z}_{t}|\mathbf{o}^{{H}_{1}}_{t},\mathbf{d}^{{H}_{2}}_{t})$ is the posterior distribution of the $\mathbf{z}_{t}$, given $\mathbf{o}^{{H}_{1}}_{t}$ and $\mathbf{d}^{{H}_{2}}_{t}$. $p(\mathbf{z}_{t})$ is the prior distribution of $\mathbf{z}_{t}$ parameterized by a standard normal distribution.

We deployed PIE on a DEEP Robotics Lite3, which is a low-cost quadruped robot. As illustrated in Fig. 2, when standing, the thigh joint height is 25cm, and the distance between the two thigh joints measures 34cm. The Lite3 weighs 12.7kg, with a peak knee joint torque of 30.5Nm [22, 23]. Similarly, the Unitree A1, used by Zhuang et al. [2] and Cheng et al. [3], has a thigh joint height of 26cm and a distance of 40cm between the thigh joints [24]. It weighs 12kg, with a peak knee joint torque of 33.5Nm. Thus, it is reasonable to use their results as baselines in our comparative experiments. Raw depth images are received at a frequency of 10Hz using its onboard Intel RealSense D435i camera. Joint actions are outputted at a frequency of 50Hz. Image post-processing and network inference are performed using its onboard Rockchip RK3588 processor.

We designed five sets of ablations to compare against ours, aiming to verify the effectiveness of PIE’s dual-level implicit-explicit estimation:

• •

  PIE w/o reconstructing $\hat{\mathbf{o}}_{t+1}$: This method can only estimate the robot’s state and surroundings explicitly without reconstructing $\mathbf{o}_{t+1}$.

• •

  PIE w/o reconstructing $\hat{\mathbf{m}}_{t}$: This method utilizes both proprioception and exteroception but only reconstructs $\mathbf{o}_{t+1}$ as implicit estimation of surrounding, lacking explicit estimation of terrain.

• •

  PIE w/o estimating $\hat{\mathbf{v}}_{t}$: This method is trained without estimating $\hat{\mathbf{v}}_{t}$.

• •

  PIE w/o estimating $\hat{\mathbf{h}}_{t}^{f}$: This method is trained without estimating $\hat{\mathbf{h}}_{t}^{f}$.

• •

  PIE using predicted $\mathbf{o}_{t+1}$: This method allows the policy network to use the predicted $\mathbf{o}_{t+1}$ as input, rather than the pure latent vector $\mathbf{z}_{t}$ used for reconstruction.

In simulation, we implemented an environment similar to the training setup for metric comparison, wherein we evaluated the performance of PIE against other ablations on different terrains. To precisely assess the policy’s traversal capability across terrains, including gap, stairs and step (hurdle was not specifically tested in simulation as the randomized step lengths covered the hurdle characteristics), we configured curriculum-like terrains with ten levels of increasing difficulty per column. We measured average level of terrain difficulty that robots can reach by letting robots start from the easiest level and gradually traversing the rest of their column until termination due to falls or collisions. For each terrain type, we created forty sets of environments, with one hundred robots tested simultaneously to ensure randomness in terrains and robots.

As shown in Table III, our PIE framework utilizing both implicit and explicit estimation demonstrates superior performance across all terrains. Although PIE w/o $\hat{\mathbf{o}}_{t+1}$ exhibits commendable performance in simulation, its reliance solely on explicit terrain estimation leads to a lack of comprehensive terrain understanding, making it susceptible to falls due to minor deviations in foothold positioning when encountering the most challenging terrains. The performance of PIE w/o $\hat{\mathbf{h}}_{t}^{f}$ significantly declines compared to the former two methods because directly estimating $\hat{\mathbf{h}}_{t}^{f}$ remains crucial for the robot’s direct comprehension of the terrain beneath its feet, lacking which can hinder the robot’s ability to execute extreme parkour maneuvers by stepping along the terrain edges. Without estimating the robot’s velocity, the PIE w/o $\hat{\mathbf{v}}_{t}$ evidently introduces significant biases in velocity tracking, consequently diminishing the robot’s maneuverability on high-difficulty terrains. The absence of regression between predicted $\hat{\mathbf{m}}_{t}$ and the ground truth in PIE w/o $\hat{\mathbf{m}}_{t}$ makes it difficult to extract useful terrain information directly from the input depth image, resulting in poor performance on highly challenging parkour terrains. PIE using predicted $\mathbf{o}_{t+1}$ struggles on step and stairs terrain because unlike $\hat{\mathbf{z}}_{t}$, which has a prior distribution parameterized by a standard normal distribution, the distribution of $\mathbf{o}_{t+1}$ is more complex, making it more challenging for the policy network to extract useful information from it.

Additionally, we evaluated the performance of both PIE and PIE w/o $\hat{\mathbf{o}}_{t+1}$ across all terrains in the presence of various camera input errors by employing significantly higher noise levels to the camera, beyond those used in standard randomization. These errors normally cause discrepancies between visual inputs and the actual environment, which often occurs under conditions where the unreliable depth camera is subject to various interference. We employed these metrics to assess the effectiveness of learning implicit-explicit estimation in such scenarios as shown in Fig. 3. Overall, PIE outperforms PIE w/o $\hat{\mathbf{o}}_{t+1}$ in both metrics, particularly when considerable noise is added to the camera, resulting in significant performance disparities between the two. In contrast, within the normal domain randomization range, their performances are comparable. This suggests that not only can our learning framework better acquire robot state estimation and terrain understanding, but also, through the estimation of $\hat{\mathbf{o}}_{t+1}$, the policy can make correct decisions when vision contradicts proprioception, placing more trust in proprioception to achieve better performance.

We compared the success rates of PIE with those of the ablations in the real world. Each method was tested for ten trails on each terrain of each difficulty level, including step, gap and stairs. Additionally, although we did not train specifically on ramp terrains, we still conducted tests on ramp terrains to evaluate the generalization performance.

As shown in Fig. 4 and the supplementary video, owing to the implicit-explicit estimation, our framework shows outstanding performance across all skills, surpassing all other ablations and previous related works. We find that our PIE framework enables the robot to climb obstacles as high as 0.75m (3x robot height), leap over gaps as large as 1m (3x robot length) and climb stairs as high as 0.25m (1x robot height). This represents a significant performance improvement of at least 50% compared to the state-of-the-art robot parkour frameworks as shown in Table IV, which indicates the relative size of traversable obstacles with respect to quadruped’s height and length respectively. Notably, PIE exhibits consistent success rates compared to in simulation, demonstrating remarkable sim-to-real transferability. It is worth mentioning that, despite the lack of specific training for ramp terrains in simulation, our framework still demonstrates better generalization performance.

Among the ablations, PIE w/o $\hat{\mathbf{o}}_{t+1}$ performs relatively better, but due to the larger delay and noise in perception and actuation in the real world, its performance noticeably decreases compared to in simulation. PIE w/o $\hat{\mathbf{h}}_{t}^{f}$ struggles to learn to handle terrain edges correctly due to the lack of intuitive estimation of foot clearance, resulting in lower success rates. As for PIE w/o $\hat{\mathbf{v}}_{t}$, when the terrain difficulty increases, the estimation of base velocity deteriorates, leading to a noticeable decline in success rates when following velocity commands. PIE w/o $\hat{\mathbf{m}}_{t}$ faces challenges extracting useful terrain information from more complicated real-world depth image compared to in simulation, making external perception interference rather than assistance, thus resulting in near-zero success rates across all terrains.

Due to the significant disturbances the depth camera faces in outdoor settings, the sim-to-real gap becomes more pronounced. To fully assess the robustness and generalization abilities of the PIE framework in such environments, as shown in Fig. 5, we conducted a long-distance hike from the ZJU Yuquan campus to the top of Laohe Mountain and back. The round trip covered 2km, with an elevation gain of 153m. Along this trail, the robot was required to traverse long-range continuous curved stairs of varying heights and widths, irregularly shaped steps and hurdles, ramps with steep inclinations, as well as deformable, slippery ground and rocky surfaces. The robot completed the entire hiking trail in just 40 minutes with no stops, except for moments when the operator and photographer struggled to keep up due to the robot’s rapid ascent on inconsistently high and wide stairs. Additionally, as shown in Fig. 6, we conducted tests in dim outdoor conditions at night, where the robot continuously jumped over high steps and irregular rocks, and climbed up and down slopes and stairs. Please refer to the supplementary video for the demonstration of the hike and night tests.