PEAR: Primitive enabled Adaptive
Relabeling for boosting Hierarchical
Reinforcement Learning

PEAR performs adaptive relabeling on expert demonstration trajectories $D$ to generate efficient higher level subgoal transition datatset $D_{g}$, by employing the current lower primitive’s action value function $Q_{\pi^{L}}(s,s_{i}^{e},a_{i})$. In a typical goal-conditioned RL setting, $Q_{\pi^{L}}(s,s_{i}^{e},a_{i})$ describes the expected cumulative reward where the start state and subgoal are $s$ and $s_{i}^{e}$, and the lower primitive takes action $a_{i}$ while following policy $\pi^{L}$. While parsing $D$, we consecutively pass the expert demonstrations states $s^{e}_{i}$ as subgoals, and $Q_{\pi^{L}}(s,s^{e}_{i},a_{i})$ computes the expected cumulative reward when the start state is $s$, subgoal is $s_{i}^{e}$ and the next primitive action is $a_{i}$. Intuitively, a high value of $Q_{\pi^{L}}(s,s^{e}_{i},a_{i})$ implies that the current lower primitive considers $s^{e}_{i}$ to be a good (highly rewarding and achievable) subgoal from current state $s$, since it expects to achieve a high intrinsic reward for this subgoal from the higher policy. Hence, $Q_{\pi^{L}}(s,s_{i}^{e},a_{i})$ considers goal achieving capability of current lower primitive for populating $D_{g}$. We depict a single pass of adaptive relabeling in Figure 2 and explain the procedure in detail below.

Adaptive Relabeling Consider the demonstration dataset $D=\{e^{j}\}_{i=1}^{N}$, where each trajectory $e^{j}=(s^{e}_{0},s^{e}_{1},\ldots,s^{e}_{T-1})$. Let the initial state be $s^{e}_{0}$. In the adaptive relabeling procedure, we incrementally provide demonstration states $s^{e}_{i}$ for $i=1$ to $T-1$ as subgoals to lower primitive’s action value function $Q_{\pi^{L}}(s^{e}_{0},s^{e}_{i},a_{i})$, where $a_{i}=\pi^{L}(s=s^{e}_{i-1},g=s^{e}_{i})$. At every step, we compare $Q_{\pi^{L}}(s^{e}_{0},s^{e}_{i},a_{i})$ to a threshold $Q_{thresh}$ ($Q_{thresh}=0$ works consistently for all experiments). If $Q_{\pi^{L}}(s^{e}_{0},s^{e}_{i},a_{i})>=Q_{thresh}$, we move on to next expert demonstration state $s^{e}_{i+1}$. Otherwise, we consider $s^{e}_{i-1}$ to be a good subgoal (since it was the last subgoal with $Q_{\pi^{L}}(s^{e}_{0},s^{e}_{i-1},a_{i})>=Q_{thresh}$), and use it to compute subgoal transitions for populating $D_{g}$. Subsequently, we repeat the same procedure with $s^{e}_{i-1}$ as the new initial state, until the episode terminates. This is depicted in Figure 2 and Algorithm 1.

Periodic re-population of higher level subgoal dataset HRL approaches suffer from non-stationarity due to unstable higher level station transition and reward functions. In off-policy HRL, this occurs since previously collected experience is rendered obsolete due to continuously changing lower primitive. We propose to mitigate this non-stationarity by periodically re-populating subgoal transition dataset $D_{g}$ after every $p$ timesteps according to the goal achieving capability of the current lower primitive. Since the lower primitive continuously improves with training and gets better at achieving harder subgoals, $Q_{\pi^{L}}$ always picks reachable subgoals of appropriate difficulty, according to the current lower primitive. This generates a natural curriculum of achievable subgoals for the lower primitive. Intuitively, $D_{g}$ always contains achievable subgoals for the current lower primitive, which mitigates non-stationarity in HRL. The pseudocode for PEAR is given in Algorithm 2. Figure 2 shows the qualitative evolution of subgoals with training in our experiments.

Dealing with out-of-distribution states Our adaptive relabeling procedure uses $Q_{\pi^{L}}(s^{e}_{0},s^{e}_{i},a_{i})$ to select efficient subgoals when the expert state $s^{e}_{i}$ is within the training distribution of states used to train the lower primitive. However, if the expert states are outside the training distribution, $Q_{\pi_{L}}$ might erroneously over-estimate the values on such states, which might result in poor subgoal selection. In order to address this over-estimation issue, we employ an additional margin classification objective(Piot et al., 2014), where along with the standard $Q_{SAC}$ objective, we also use an additional margin classification objective to yield the following optimization objective $\bar{Q}_{\pi^{L}}=Q_{SAC}+$
$\arg\underset{Q_{\pi_{L}}}{\min}\>\underset{\pi^{L}}{\max}(\mathbb{E}_{(s_{0}^{e},\cdot,\cdot)\sim D_{g},s_{i}^{e}\sim\pi^{H},a_{i}\sim\pi^{L}}[Q_{\pi^{L}}(s_{0}^{e},s_{i}^{e},a_{i})]-\mathbb{E}_{(s_{0}^{e},s_{i}^{e},\cdot)\sim D_{g},a_{i}\sim\pi^{L}}[Q_{\pi^{L}}(s_{0}^{e},s_{i}^{e},a_{i})])$
This surrogate objective prevents over-estimation of $\bar{Q}_{\pi^{L}}$ by penalizing states that are out of the expert state distribution. We found this objective to improve performance and stabilize learning. In the next section, we explain how we use adaptive relabeling to yield our joint optimization objective.

Here, we explain our joint optimization objective that comprises of off-policy RL objective with IL based regularization, using $D_{g}$ generated using primitive enabled adaptive relabeling. We consider both behavior cloning (BC) and inverse reinforcement learning (IRL) regularization. Henceforth, PEAR-IRL will represent PEAR with IRL regularization and PEAR-BC will represent PEAR with BC regularization. We first explain BC regularization objective, and then explain IRL regularization objectives for both hierarchical levels.

For the BC objective, let $(s^{e},s^{e}_{g},s^{e}_{next})\sim D_{g}$ represent a higher level subgoal transition from $D_{g}$ where $s^{e}$ is current state, $s^{e}_{next}$ is next state, $g^{e}$ is final goal and $s^{e}_{g}$ is subgoal supervision. Let $s_{g}$ be the subgoal predicted by the high level policy $\pi_{\theta_{H}}^{H}(\cdot|s^{e},g^{e})$ with parameters $\theta_{H}$. The BC regularization objective for higher level is as follows:

Similarly, let $(s^{f},a^{f},s^{f}_{next})\sim D_{g}^{L}$ represent lower level expert transition where $s^{f}$ is current state, $s^{f}_{next}$ is next state, $g^{f}$ is goal and $a$ is the primitive action predicted by $\pi_{\theta_{L}}^{L}(\cdot|s^{f},s^{e}_{g})$ with parameters $\theta_{L}$. The lower level BC regularization objective is as follows:

We now consider the IRL objective, which is implemented as a GAIL (Ho and Ermon, 2016) objective implemented using LSGAN (Mao et al., 2016). Let $\mathbb{D}_{\epsilon}^{H}$ be the higher level discriminator with parameters $\epsilon_{H}$. Let $J_{D}^{H}$ represent higher level IRL objective, which depends on parameters $(\theta_{H},\epsilon_{H})$. The higher level IRL regularization objective is as follows:

Similarly, for lower level primitive, let $\mathbb{D}_{\epsilon_{L}}^{L}$ be the lower level discriminator with parameters $\epsilon_{L}$. Let $J_{D}^{L}$ represent lower level IRL objective, which depends on parameters $(\theta_{L},\epsilon_{L})$. The lower level IRL regularization objective is as follows:

Finally, we describe our joint optimization objective for hierarchical policies. Let the off-policy RL objective be $J^{H}_{\theta_{H}}$ and $J^{L}_{\theta_{L}}$ for higher and lower policies. The joint optimization objectives using BC regularization for higher and lower policies are provided in Equations 5 and  6 respectively.

The joint optimization objectives using IRL regularization for higher and lower policies are provided in Equations 7 and  8 respectively.

Here, $\psi$ is regularization weight hyper-parameter. We describe ablations to choose $\psi$ in Section 5.

Now, we define the suboptimality of policy $\pi$ with respect to optimal policy $\pi^{*}$ as:

Similarly, the suboptimality of lower primitive $\pi_{\theta_{L}}^{L}$ can be bounded as:

where $\lambda_{L}=\frac{2}{(1-\gamma)^{2}}R_{max}\|\frac{d_{c}^{\pi^{**}}}{\kappa}\|_{\infty}$

The proofs for Equations 10 and 11 are provided in Appendix A.1. We next discuss the effect of training on the two terms in RHS of Equation 10, which bound the suboptimality of $\pi_{\theta_{H}}^{H}$.

Effect of adaptive relabeling on sub-optimality bounds We firstly focus on the first term which is dependent on $\phi_{D}$. Since we represent the generated subgoal dataset as $D_{g}$, we replace $\phi_{D}$ with $\phi_{D_{g}}$. In Theorem 1, we assume the optimal policy $\pi^{*}$ to be $\phi_{D_{g}}$ common in $\Pi_{D}^{H}$. Since $\phi_{D_{g}}$ denotes the upper bound of the expected TV divergence between $\pi^{*}$ and $\pi_{D}^{H}$, $\phi_{D_{g}}$ provides a quality measure of the subgoal dataset $D_{g}$ populated using adaptive relabeling. Intuitively, a lower value of $\phi_{D_{g}}$ implies that the optimal policy $\pi^{*}$ is closely represented by $D_{g}$, or alternatively, the samples from $D_{g}$ are near optimal. As the lower primitive improves with training and is able to achieve harder subgoals, and since $D_{g}$ is re-populated using the improved lower primitive after every $p$ timesteps, $\pi_{D_{g}}$ continually gets closer to $\pi^{*}$, which results in reduced value of $\phi_{D}$. This implies that due to decreasing first term, the suboptimality bound in Equation 10 gets tighter, and consequently $J(\pi_{\theta_{H}}^{H})$ gets closer to optimal $J(\pi^{*})$ objective. Hence, our periodic re-population based approach generates a natural curriculum of achievable subgoals for the lower primitive, which continuously improves the performance by tightening the upper bound.

Effect of IL regularization on sub-optimality bounds Now, we focus on the second term in Equation 10, which is TV divergence between $\pi_{D}^{H}(\tau|s,g)$ and $\pi_{\theta_{H}}^{H}(\tau|s,g)$ with expectation over $s\sim\kappa,\pi_{D}^{H}\sim\Pi_{D}^{H},g\sim G$. As before, $D$ is replaced by dataset $D_{g}$. This term can be viewed as imitation learning (IL) objective between expert demonstration policy $\pi_{D_{g}}^{H}$ and current policy $\pi_{\theta_{H}}^{H}$, where TV divergence is the distance measure. Due to this IL regularization objective, as policy $\pi_{\theta_{H}}^{H}$ gets closer to expert distribution policy $\pi_{D_{g}}^{H}$ with training, the LHS sub-optimality bounds get tighter. Thus, our proposed periodic IL regularization using $D_{g}$ tightens the sub-optimality bounds in Equation 10 with training, thereby improving performance.

Generalized framework We now derive our generalized framework for the joint optimization objective, where we can plug in off-the-shelf RL and IL methods to yield a generally applicable practical HRL algorithm. Considering sub-optimality is positive (Equation  9), we can use Equation 10 to derive the following objective:

where (considering $\pi_{D}^{H}(\tau|s,g)$ as $\pi_{A}$, and $\pi_{\theta_{H}}^{H}(\tau|s,g)$ as $\pi_{B}$), $d(\pi_{A}||\pi_{B})=D_{TV}(\pi_{A}||\pi_{B})$.

Notably, the second term $\lambda_{H}*\phi_{D}$ in RHS of Equation 12 is constant for a given dataset $D_{g}$. Equation 12 can be perceived as a minorize maximize algorithm which intuitively means: the overall objective can be optimized by $(i)$ maximizing the objective $J(\pi_{\theta_{H}}^{H})$ via RL, and $(ii)$ minimizing the distance measure $d$ between $\pi_{D}^{H}$ and $\pi_{\theta_{H}}^{H}$ (IL regularization). This formulation serves as a framework where we can plug in RL algorithm of choice for off-policy RL objective $J(\pi_{\theta_{H}}^{H})$, and distance function $d$ of choice for IL regularization, to yield various joint optimization objectives.

In our setup, we plug in entropy regularized Soft Actor Critic (Haarnoja et al., 2018a) to maximize $J(\pi_{\theta_{H}}^{H})$. Notably, different parameterizations of $d$ yield different imitation learning regularizers. When $d$ is formulated as Kullback–Leibler divergence, the IL regularizer takes the form of behavior cloning (BC) objective (Nair et al., 2018) (which results in PEAR-BC), and when $d$ is formulated as Jensen-Shannon divergence, the imitation learning objective takes the form of inverse reinforcement learning (IRL) objective (which results in PEAR-IRL). We consider both these objectives in Section 5 and provide empirical performance results.

In Figure 3, we depict the success rate performance of PEAR and compare it with other baselines averaged over $5$ seeds. The primary goal of these comparisons is to verify that the proposed approach indeed mitigates non-stationarity and demonstrates improved performance and training stability.

Comparing with fixed window based approach

RPL: In order to demonstrate the efficacy of adaptive relabeling, we compare our method with Relay Policy Learning (RPL) baseline. RPL (Gupta et al., 2019) uses supervised pre-training followed by relay fine tuning. In order to ascertain fair comparisons, we use an ablation of RPL which does not use supervised pre-training. PEAR outperforms this baseline, which demonstrates that adaptive relabeling outperforms fixed window based relabeling and is crucial for mitigating non-stationarity. Since PEAR and RPL both employ jointly optimizing RL and IL based learning and only differ in adaptive relabeling, it is evident that adaptive relabeling is crucial for generating feasible subgoals.

Comparing with hierarchical baselines

RAPS: RAPS (Dalal et al., 2021) uses hand designed action primitives at the lower level, where the goal of the upper level is to pick the optimal sequence of action primitives. The performance of such approaches significantly depends on the quality of action primitives, which require substantial effort to hand-design. We found that except maze navigation task, RAPS is unable to perform well on other tasks, which we believe is because selecting appropriate primitive sequences is hard on other harder tasks. Notably, hand designing action primitives is exceptionally complex in environments like rope manipulation. Hence, we do not evaluate RAPS in rope environment.

HAC: Hierarchical actor critic (HAC) (Levy et al., 2018) deals with non-stationarity by relabeling transitions while assuming an optimal lower primitive. Although HAC shows good performance in maze navigation task, PEAR consistently outperforms HAC on all other tasks.

HIER-NEG and HIER: We also compare PEAR with two hierarchical baselines: HIER and HIER-NEG, which are hierarchical baselines that do not leverage expert demonstrations. HIER-NEG is a hierarchical baseline where the upper level is negatively rewarded if the lower primitive fails to achieve the subgoal. Since HIER, HIER-NEG and PEAR all are hierarchical approaches, we use these baseline comparisons to motivate that the performance improvement is not just due to use of hierarchical abstraction, but instead due to adaptive relabeling and primitive-enabled regularization. This is clearly evidenced by superior performance of PEAR.

Comparing with non-hierarchical baselines

Additionally, we consider single-level Discriminator Actor Critic (DAC) (Kostrikov et al., 2019) that leverages expert demos, single-level  SAC (FLAT) baseline, and Behavior Cloning (BC) baselines. However, they fail to perform well in any of the tasks.

Here, we perform ablation analysis to elucidate the significance of our design choices. We choose the hyper-parameter values after extensive experiments, and keep them consistent across all baselines.

We assess whether PEAR mitigates non-stationarity in HRL by comparing it with the vanilla HIER baseline, as shown in Figure 4. We compute the average distance between subgoals predicted by the higher-level policy and those achieved by the lower-level primitive throughout training. A lower average distance suggests that PEAR generates subgoals achievable by the lower primitive, inducing lower primitive behavior to be optimal. Our findings reveal that PEAR consistently maintains low average distances, validating its effectiveness in reducing non-stationarity. Additionally, as seen in Figure 4, post-training results show that PEAR achieves significantly lower distance values than the HIER baseline, highlighting its ability to generate feasible subgoals through primitive regularization.

Additional Ablations: First, we verify the importance of adaptive relabeling by replacing it in PEAR-IRL by fixed window relabeling (as in RPL (Gupta et al., 2019)). As seen in Figure 6, this ablation (PEAR-IRL) consistently outperforms PEAR-RPL on all tasks, which demonstrates the benefit of adaptive relabeling. Further, we compare PEAR-IRL and PEAR-BC (with margin classification objectives), with PEAR-IRL-No-Margin and PEAR-BC-No-Margin (without margin objectives) in Figure 7. PEAR-IRL and PEAR-BC outperform their No-Margin counterparts, which shows that this objective efficiently deals with the issue of out-of-distribution states, and induces training stability.

Further, we analyse how varying $Q_{thresh}$ affects performance in Appendix A.4 Figure 8. We next study the impact of varying $p$. Intuitively, if $p$ is too large, it impedes generation of a good curriculum of subgoals (Appendix A.4 Figure 9). Also, a low value of $p$ may lead to frequent subgoal dataset re-population and may impede stable learning. We also choose optimal window size $k$ for RPL experiments, as shown in Appendix A.4 Figure 10. We also evaluate learning rate $\psi$ in Appendix A.4 Figure 11. If $\psi$ is too small, PEAR is unable to utilize IL regularization, whereas conversely if $\psi$ is too large, the learned policy might overfit. We also deduce the optimal number of expert demonstrations required in Appendix A.4 Figure 12. Next, we compare the performance of PEAR-IRL with HER-BC, which is a single-level implementation of HER with expert demonstrations. As seen in Appendix A.4 Figure 13, PEAR significantly outperforms this baseline, which demonstrates the advantage of our hierarchical formulation. We also provide qualitative visualizations in Appendix A.5.

Real world experiments: We perform experiments on real world robotic pick and place, bin and rope environments (Fig 6). We use Realsense D435 depth camera to extract the robotic arm position, block, bin, and rope cylinder positions. Computing accurate linear and angular velocities is hard in real tasks, so we assign them small hard-coded values, which shows good performance. We performed $5$ sets of experiments with $10$ trial each, and report the average success rates. PEAR-IRL achieves accuracy of $0.8$, $0.6$, and $0.3$, whereas PEAR-BC achieves accuracy of $0.8$, $0$, $0.3$ on pick and place, bin and rope environments. We also evaluate the performance of next best performing RPL baseline, but it fails to achieve success in any of the tasks.