OGMP: Oracle Guided Multi-mode Policies for
Agile and Versatile Robot Control

We apply the proposed framework to two bipedal control tasks: parkour and dive, with varying objectives and extent of multi-modality as shown in table I. Note that the choice of task-vital multi-mode is user-defined, and table I simply reflects our choice for the same. Evident from table I, $J_{\mathcal{T}}$ is task-dependent. Recent attempts in quadruped parkour [2, 3], and locomotion [6, 1] show some well-shaped candidates for $J_{\mathcal{T}}$, albeit case-specific. In general, a reasonable $J_{\mathcal{T}}$ could be hard to design (for instance, the dive task); a compelling unified alternative would be to “track” the oracle’s $\epsilon$-neighbourhood reference to the optimal solution. Hence, we propose minimizing the task-independent surrogate objective: $\tilde{J_{\mathcal{T}}}:=\sum_{t=0}^{\infty}\|x^{\pi}_{t}-x^{\Xi}_{t}\|$. $\tilde{J_{\mathcal{T}}}$’s applicability is studied in Sec. IV-B and a reward is proposed in Sec III-C for an equivalent maximization objective.

For any locomotion task, a simple heuristic for an oracle would be linearly interpolating the relevant state variables from the initial to the desired goal states. In parkour, to advance along the track, the heading position can be linearly interpolated along the track while adapting to the terrain height, as shown in Fig. 4 (left). For dive, the oracle can linearly interpolate the base height and corresponding rotational DoF from $0^{\degree}$ to $360^{\degree}$. Naming this heuristic oracle as $\Xi_{\text{li}}$ (Fig. 4), its high $\epsilon$ is obvious as the generated ansatz do not consider the system’s inertia and gravity. Hence oracles that capture the dominant dynamics of the hybrid system are required. To this end, we have used a modified version of the simplified Single Rigid Body (SRB) model[25] whose dynamics in the world coordinates are given by:

where $\ddot{p}$, $\omega$ is the robot COM acceleration and the angular velocity, $r_{i},\,f_{i},\,\tau_{i}$ are the position, force and moment vectors from the $i^{th}$ contact point and $m,\,I,\,g$ are the mass, moment of inertia and gravity. Typically, $r_{i}$ is from a predefined contact schedule, leading to time-varying dynamics. Since, by definition, an oracle need not provide realistic control, we define an auxiliary control $\bar{\tau}_{i}$, encompassing the overall moment, making the dynamics time-invariant. Additionally, the rotation and rotation rate matrices are made constant by considering the average reference orientation over a horizon. Upon these approximations to Eq. 6 and discretization leads to a linear time-invariant (LTI) system over the current horizon, where $x_{t}\in\mathbb{R}^{13},\,y_{t},\,u_{t}\in\mathbb{R}^{12}$ are the gravity-augmented state, relevant output, and control vectors. Thus, oracles can be constructed considering two distinct phases: flight and contact. In flight, $u_{t}=0$ as there are no contacts, and during contact, an optimizer of choice can be used to compute the optimal control for a given objective, $u_{t}=u^{*}_{t}$. The reference state trajectory is obtained by applying the corresponding control and forward simulating Eq. 7. Using $\Xi_{\text{li}}$ as the reference for a quadratic tracking objective on the LTI system, optimizing with preview control [29] and LQR results in oracles, $\Xi_{\text{prev}}$ and $\Xi_{\text{lqr}}$ respectively, having a smaller $\epsilon$ than $\Xi_{\text{li}}$ as shown in Fig 4

Mode Encoder: We train a mode encoder, $\xi$, on diverse locomotion modes to obtain a compact conditioning space ideal for commanding our policy. Similar to [28], the encoder, $z=\xi(x[t,\,t+\Delta t])$, maps the trajectory space to a latent space ($\dim(z)=2$). Uniformly sampling from a mode parameter set,$\Psi_{\mathbb{M}}$ and a set of initial states, $\mathbb{X}_{0}$, we generate a rich and balanced modal dataset by querying the oracle, $\Xi$ as shown in Fig. 3.c. Minimizing the reconstruction loss for a $32$ neurons single hidden layer LSTM auto-encoder on the custom dataset generates a set of latent mode points with a structured clustering as visualized in Fig. 3.c.

Mode Conditioned Policy: Our choice of action space for a stationary policy is from [28]. Given the inherent partial observability of the system, for the observation space $o_{t}\in\mathcal{O}$ we choose $o_{t}=[\tilde{x}_{t},z_{t},\,c_{t},h_{t}]$, where $\tilde{x}_{t}$ is the robot’s proprioceptive feedback, $z_{t}$ is the latent mode command, $c_{t}$ is a clock signal [28] and $h_{t}$ is optional task-based feedback (like terrain scan for parkour). The per-step reward for a task-agnostic surrogate tracking objective is defined as

where $er_{p},er_{o}$ are the errors in base position and orientations. Thus, $r_{\text{track}}$ minimizes an error in the task space, and $r_{\text{regulation}}$ regularizes the motion for enhancing sim-to-real. Note for the diverse modes across both the tasks (parkour and dive) the reward weights remain the same, hinting a sense of algorithmic robustness that arises from guided learning. The proposed permissible state constraint (Sec. II) is programmed as a termination condition, terminate episode:= $\mathds{1}{(\|x^{\pi}_{t}-x^{\Xi}_{t}\|_{W}>\rho)}$. $W_{11}$ and $W_{33}$ are set to $1$ for the parkour and dive tasks, respectively, with the remaining entries as zero. For solving the resulting random horizon POMDP, we use off-the-shelf PPO to train a policy: $128$ nodes per layer, $2$ layer LSTM network, where each episode is an arbitrary variant of the task $\psi_{\mathcal{T}}$ uniformly sampled from $\Psi_{\mathcal{T}}$.

To achieve extreme agility and mode versatility, a single multi-mode policy is trained per task: $\pi_{\text{parkour}}$ for parkour and $\pi_{\text{dive}}$ for diving. The supplementary video and Fig 1 show $\pi_{\text{parkour}}$ successfully navigating challenging parkour tracks with blocks and gaps placed randomly, demonstrating versatile agility over leap lengths and jump heights. $\pi_{\text{dive}}$ performs omnidirectional flips from different heights and transitions smoothly to landing. Despite lacking a reference for the actuated DoFs, $\pi_{\text{dive}}$ learns an emergent behavior to curl and extend its legs for flips and landings, modulating the torso angular velocity and landing impact. As seen in the video, $\pi_{\text{parkour}}$ significantly deviates from the oracle’s reference to find the optimal behavior, yet results in regularized motion due to the oracle bound. Sim-to-real transfer of $\pi_{\text{parkour}}$’s modes and transitions can be seen in Fig 1,5 and supplementary video.

Finally, we analyze our choice of surrogate $\tilde{J}_{\mathcal{T}}$ and design choices that impact performance (measured via undiscounted returns). For parkour, a potential true objective, $J_{\mathcal{T}}$, is the displacement along the track. Thus the validity of using $\tilde{J}_{\mathcal{T}}$ can be quantified through its disparity with $J_{\mathcal{T}}$.

Choice of observation space $(o_{t})$: Ablation of observation space components (excluding $\tilde{x}_{t}$) shows that variants with $c_{t}$ consistently outperform their counterparts (Fig. 6i.a). Variants without $h_{t}$ are myopic and aggressive, with higher accelerations (table II) as they purely rely on compressed $z_{t}$ for terrain feedback, making them sub-optimal compared to terrain-aware variants. From Fig. 6i.a, we observe that latent conditioning does not improve performance (see [$h_{t},c_{t}$] and [$z_{t},h_{t},c_{t}$]), hence is purely for analysis and reusability. However, a conditioned $\pi_{\text{parkour}}$ can use the oracle as a closed-loop reactive planner during inference, driving the system to the commanded mode. From Fig.6i.a and b, a similar trend of $J_{\mathcal{T}}$ and $\tilde{J}_{\mathcal{T}}$ shows no disparity caused by observation space variations.

Choice of oracle $(\Xi)$: The three oracles, $\Xi_{\text{li}}$, $\Xi_{\text{lqr}}$, and $\Xi_{\text{prev}}$, have a non-increasing trend in maximum deviation bound, $\epsilon$. As the optimal exploration bound ($\rho^{*}$) depends on $\epsilon$, we vary $\rho$ from 0.1 to 0.8 for each oracle. For parkour, different oracles show no significant performance difference (see video results). However, for dive, $\Xi_{\text{prev}}$ variants perform significantly better (Fig. 6i.c and 6iii.c).A tighter bound ($\rho=0.1$) works best for $\Xi_{\text{prev}}$, as it has the lowest $\epsilon$. Conversely, higher exploratory deviation ($\rho=0.4$) performs best for $\Xi_{\text{lqr}}$ and $\Xi_{\text{li}}$. Thus, $\epsilon_{\text{li}}\geq\epsilon_{\text{lqr}}\geq\epsilon_{\text{prev}}\implies\rho^{*}_{\text{li}}\geq\rho^{*}_{\text{lqr}}>\rho^{*}_{\text{prev}}$, affirming Eq 5 .

Choice of oracle’s horizon $(\Delta t)$: We evaluate policy performance across different horizons: $\Delta t=7dt$, $15dt$, and $30dt$ for parkour. The shortest horizon, $\Delta t=7dt$, leads to a myopic behavior, maintaining a high $\tilde{J}_{\mathcal{T}}$ but low $J_{\mathcal{T}}$ as it remains stationary by exploiting the high replanning frequency (Fig. 6iii.d and 6i.d, e) as also observed by [13]. Although advancing forward, $\Delta t=15dt$ fails to anticipate farther terrain, resulting in quasi-static maneuvers (Fig. 6iii.e). Conversely, $\Delta t=30dt$ enables the robot to leap efficiently from block to block, demonstrating agility and achieving the optimal outcome (Fig. 6iii.f). Increasing $\Delta t$ aligns $\tilde{J}_{\mathcal{T}}$ with the true task objective, $J_{\mathcal{T}}$ (Fig. 6i.d, e).

Choice of permissible state bound $(\rho)$: For $\Xi_{\text{prev}}$ in parkour, we varied $\rho$ from $0.05$ to $0.8$ (Fig. 6i.f and g). We found an optimal $\rho^{*}=0.5$ with performance decreasing away from this value. For $\rho<\rho^{*}$, the optimal solution may lie outside the $\rho$-neighborhood of $x^{\Xi}$ (Fig. 6iii.g and Fig. 6iv). Conversely, $\rho>\rho^{*}$ increases local optima within $\epsilon+\rho$, leading to sub-optimal solutions (high $\tilde{J}_{\mathcal{T}}$), seen in Fig. 6iii.i for $\rho=0.8$, where the policy stagnates without advancing (low $J_{\mathcal{T}}$). Training curves in Fig. 6iv show that $0.1\leq\rho\leq 0.6$ converges to the global optima, while the rest settle at local optima. Note that $\rho\rightarrow\infty$ is standard PPO as it optimizes $\tilde{J}_{\mathcal{T}}$ unguided by $\Xi$. Vanilla unguided PPO ($\rho=10^{10}$) falls into the same local optima as $\rho=0.8$. Thus, oracle-guided optimization improves standard PPO by escaping local optima with the right choice of $\rho$.