Proposing Hierarchical Goal-Conditioned Policy Planning
in Multi-Goal Reinforcement Learning ^††thanks: This is the preprint version of the paper accepted at ICAART 2025.

Reinforcement learning (RL) is based on the Markov decision process (MDP). An MDP is described as a tuple $\langle S,A,T,R,\gamma\rangle$, where $S$ is a set of states, $A$ is a set of (primitive) actions, $T$ is a transition function (the probability of reaching a state from a state via an action), $R:S\times A\times S\mapsto\mathbb{R}$ and $\gamma$ is the discount factor. The value of a state $s$ is the expected total discounted reward an agent will get from $s$ onward, that is,

where $s_{t+1}$ is the state reached by executing action $a$ in state $s_{t}$. Similarly, the value of executing $a$ in $s$ is defined by

We call it the Q function and its value is a q-value. The aim in MDPs and RL is to maximize $V(s)$ for all reachable states $s$. A policy $\pi:S\mapsto A$ tells the agent what to do: execute $a=\pi(s)$ when in $s$. A policy can be defined in terms of a Q function:

It is thus desirable to find ‘good’ q-values (see later).

Goal-conditioned reinforcement learning (GCRL) (Schaul et al.,, 2015; Liu et al.,, 2022) is based on the GCMDP, defined as the tuple $\langle S,A,T,R,G,\gamma,\rangle$, where $S$, $A$, $T$ and $\gamma$ are as before, $G$ is a set of (desired) goals and $R:S\times A\times S\times G\mapsto\mathbb{R}$ is goal-conditioned reward function. In GCRL, the value of a state and the Q function are conditioned on a goal $g\in G$: $V(s,g)$, respectively, $Q(a,s,g)$. A goal-conditioned policy is

such that $\pi(s,g)$ is the action to execute in state $s$ towards achieving $g$.

Traditionally, hierarchical RL (HRL) has been a divide-and-conquer approach, that is, determine the end-goal, divide it into a set of subgoals, then select or learn the best way to reach the subgoals, and finally, achieve each subgoal in an order appropriate to reach the end-goal.

“Hierarchical Reinforcement Learning (HRL) rests on finding good re-usable temporally extended actions that may also provide opportunities for state abstraction. Methods for reinforcement learning can be extended to work with abstract states and actions over a hierarchy of subtasks that decompose the original problem, potentially reducing its computational complexity.” (Hengst,, 2012)

“The hierarchical approach has three challenges [619, 435]: find subgoals, find a meta-policy over these subgoals, and find subpolicies for these subgoals.” (Plaat,, 2023)

The divide-and-conquer approach can be thought of as a top-down approach. The approach that our framework takes is bottom-up: the agent learns ‘skills’ that could be employed for achieving different end-goals, then memorizes sets of connected skills as more complex skills. Even more complex skills may be memorized based on less complex skills, and so on. Higher-level skills are always based on already-learned skills. In this work, we call a skill (of any complexity) a high-level action.

The version Monte Carlo tree search (MCTS) we are interested in is applicable to single agents based on MDPs (Kocsis and Szepesvari,, 2006).

An MCTS-based agent in state $s$ that wants to select its next action loops thru four phases to generate a search tree rooted at a node representing $s$. Once the agent’s planning budget is depleted, it selects the action extending from the root that was most visited (see below) or has the highest q-value. While there still is planning budget, the algorithm loops thru the following phases (Browne et al.,, 2012).

Selection A route from the root until a leaf node is chosen.¹¹1For discrete problems with not ‘too many’ actions, a node is a leaf if not all actions have been tried at least once in that node. Let $UCBX:Nodes\times A\mapsto\mathbb{R}$ denote any variant of the Upper Confidence Bound, like UCB1 (Kocsis and Szepesvari,, 2006). For every non-leaf node $n$ in the tree, follow the path via action $a^{*}=\operatorname*{arg\,max}_{a\in A}UCBX(n,a)$.

Expansion When a leaf node $n$ is encountered, select an action $a$ not yet tried in $n$ and generate a new node representing $s^{\prime}$ as child of $n$.

Rollout To estimate a value for $s^{\prime}$ (or $n^{\prime}$ representing it), simulate one or more Monte Carlo trajectories (rollouts) and use them to calculate a reward-to-go from $n^{\prime}$. If the rollout value for $s^{\prime}$ is $V(s^{\prime})$, the $Q(a,s)$ can be calculated.

Backpropagation If $n^{\prime}$ has just been created and the q-value associated with the action leading to it has been determined, then all action branches on the path from the root till $n$ must be updated. That is, the change in value at the end of a path must be propagated bach up the tree.

In this paper, when we write $f(n)$ (where $n$ might have indices and $f$ is some function), we generally mean $f(n.s)$ and $f(s^{\prime})$, where $n.s=s^{\prime}$ is the state represented by node $n$.

We make a distinction between multi-objective RL (MORL) and multi-goal RL (MGRL). MORL (Liu et al.,, 2015) attempts to achieve all goals simultaneously. There is often a weight or priority assigned to goals. MGRL (Kaelbling,, 1993; Sutton et al.,, 2011) does not weight any goal as more or less important and each goal is assumed to eventually be pursued individually. There is no question about which goal is more important; the agent simply pursues the goal it is commanded to. In both cases, the agent can learn all goals simultaneously (broadly speaking). When MORL goals are prioritized and MGRL has a curriculum, the two frameworks become very similar.

Our framework follows the MGRL approach, not the MORL approach.

If the agent decides to explore and thus to generate a new GCP, what should the end-point of the new GCP be? That is, if the agent is at $s$, how does the agent select $s^{\prime}$ to learn $\pi[s,s^{\prime}]$? The ‘quality’ of a behavioral goal $s^{\prime}$ to be achieved from current exploration point $s$ is likely to be based on curiosity (Schmidhuber,, 2006, 1991) and/or novelty-seeking (Lehman and Stanley,, 2011; Conti et al.,, 2017) and/or coverage (Vassilvitskii and Arthur,, 2006; Huang et al.,, 2019) and/or reachability (Castanet et al.,, 2023).

We require a function that takes as argument the exploration point $s$ and maps to behavioral goal $s^{\prime}$ that is similar enough to $s$ that it has good reachability (not too easy or too hard to achieve it, aka a GOID - goal of intermediate difficulty (Castanet et al.,, 2023)). Moreover, of all $s^{\prime\prime}$ with approximately equal similarity to $s$, $s^{\prime}$ must be the $s^{\prime\prime}$ most dissimilar to all children of $s$ on average. The latter property promotes novelty and (local) coverage.

We denote the function that selects the ‘appropriate’ behavioral goal when at exploration point $s$ as $ChooseBevGoal(s)$. Some algorithms have been proposed in which variational autoencoders are trained to represent $ChooseBevGoal(s)$ with the desired properties (Khazatsky et al.,, 2021; Fang et al.,, 2022; Li et al.,, 2022): An encoder represents states in a latent space, and a decoder generates a state with similarity proportional the (hyper)parameter. They sample from the latent space and select the most applicable subgoals for the planning phase (Khazatsky et al.,, 2021; Fang et al.,, 2022) or perturb the subgoal to promote novelty (Li et al.,, 2022). Another candidate for representing $ChooseBevGoal(s)$ is the method proposed by Castanet et al., (2023): use a set of particles updated via Stein Variational Gradient Descent “to fit GOIDs, modeled as goals whose success is the most unpredictable.” There are several other works that train a neural network that can then be used to sample a good goal from any exploration point (Shin and Kim,, 2023; Wang et al., 2024b, , e.g.).

In HGCPP, each time the node representing $s$ is expanded, a new behavioral goal $g_{b}$ is generated. Each new $g_{b}$ to be achieved from $s$ must be as novel as possible given the behavioral goal already associated with (i.e. connecting the children of) $s$. We thus propose the following. Sample a small batch $B$ of candidate behavioral goals using a pre-trained variational autoencoder and select $g_{b}\in B$ that is most different to all existing behavioral goals already associated with $s$. The choice of measure of difference/similarity between two goals is left up to the person implementing the algorithm.

For every iteration thru the MCTS tree, for every internal node on the search path, a decision must be made whether to follow one of the generated HLAs or to explore and thus expand the tree with a new HLA.

Let $\eta(s,\delta)$ be an estimate for the number candidate behavioral goals $g_{b}$ around $s$, with $\delta$ being a hyper-parameter proportional to the distance of $g_{b}$ from $s$. We propose the following exploration strategy, based on the logistic function. Expand node $n$ with a probability equal to

where $0<k\leq 1$ and $x$ is the number of GCPs starting in $n$. If we do not expand, then we select an HLA from $HLAs(n)$ that maximizes $UCBX$.

Suppose that we are learning policy $\pi[s_{s},s_{e}]$. Let $\sigma[s_{s},s_{e}]$ be a sequence $s_{1},a_{1},s_{2},a_{2},\ldots,a_{j},s_{j+1}$ of states and primitive actions such that $s_{1}=s_{s}$ and $s_{j+1}=s_{e}$. Let $\mathit{traj}(s_{s},s_{e})$ be all such trajectories (sequences) $\sigma[s_{s},s_{e}]$ experienced by the agent in its lifetime. Then we define the value $R^{\pi[s_{s},s_{e}]}_{g}$ of $\pi[s_{s},s_{e}]$ with respect to goal $g$ as

In words, $R^{\pi[s_{s},s_{e}]}_{g}$ is the average sum of rewards experienced of actions executes in trajectories from $s_{s}$ to $s_{e}$ for pursuing $g$. At the moment, we do not specify whether $R_{g}(a_{i},s_{i})$ is given or learned.

Only GCPs are directly involved in backpropagation. In other words, we backpropagate values from node to node up the plan-tree as usual in MCTS while treating the GCPs as primitive actions in regular MCTS. The way the hierarchy is constructed guarantees that there is a sequence of linked GCPs between any node reached and the root node.

Every time a GCP $\pi[n.s,n^{\prime}.s]$ and corresponding node $n^{\prime}$ are added to the plan-tree, the value of $\pi$ determined just after learning $\pi$ (i.e. $R^{\pi}_{g}$) is propagated back up the tree, for every desired goal. In general, for every GCP $\pi[n.s,\cdot]$ (and every non-leaf node $n$) on the path from the root until $n^{\prime}$, for every goal $g\in G$, we maintain a goal-conditioned value $Q(\pi,n,g)$ representing the estimated reward-to-go from $n.s$ onwards.

As a procedure, backpropagation is defined by Algorithm 1. Note that $q$ and $fv$ are arrays indexed by goals in $G$.

Every non-GCP HLA has a q-value. They are maintained as follows. Let $\pi_{i}\to\ldots\to\pi_{j}$ be the sequence of GCPs that constitutes non-GCP HLA $h[n_{i},n_{j}]$. Every non-GCP HLA either ends at a leaf or it does not. That is, either $n_{j}=n^{\prime}$ or not.

If $n_{j}$ is a leaf, then

where $m$ is the number of GCPs constituting $h$. Else if $n_{j}$ is not a leaf, then

where $n_{j+1}$ is the node at the end of $\pi^{j}$.

Suppose that $\pi[n,n^{\prime}]$ has just been generated and its q-value propagated back. Let the path from the root till leaf node $n^{\prime}$ be described by the sequence of linked GCPs

Notice that some non-GCP HLAs will be completely or partially constituted by GCPs that are a subsequence of (2). Only these HLAs’ q-values need to be updated.

An idea is to use a progress-based strategy: Fix the order of the goals in $G$ arbitrarily: $g_{1},g_{2},\ldots,g_{n}$. Let $Prog(g,c,t,w,\theta)$ be true iff: the average reward with respect to $g$ experienced over $w$ GCP-steps $c$ is at least $\theta$ or the number of steps is less than $t$. Parameter $t$ is the minimum time the agent should spend learning how to achieve a goal, per attempt. If $Prog(g_{i},c,t,w,\theta)$ becomes false while pursuing $g_{i}$, then set $c$ to zero, and start pursuing $g_{i+1}$ if $i\neq n$ or start pursuing $g_{1}$ if $i=n$. Colas et al., (2022) discuss this issue in a section called “How to Prioritize Goal Selection?”.

Once training is finished, the robot can use the generated plan-tree to achieve any particular goal $g\in G$. The execution process is depicted in Figure 4.

Note that $\pi_{1}|\pi_{2}$ means that GCPs $\pi_{1}[s_{s}^{1},s_{e}^{1}]$ and $\pi_{2}[s_{s}^{2},s_{e}^{2}]$ are concatenated to form $\pi[s_{s}^{1},s_{e}^{2}]$; concatenation is defined if and only $s_{e}^{1}=s_{s}^{2}$.

Almost any RL algorithm can be used to learn GCPs, once generated. They might need to be slightly modified to fit the GCRL setting. For instance, Kulkarni et al., (2016); Zadem et al., (2023) use DQN (Mnih et al.,, 2015), Ecoffet et al., (2021) uses PPO (Schulman et al.,, 2017), Yang et al., (2022) uses DDPG (Lillicrap et al.,, 2016) and Shin and Kim, (2023); Castanet et al., (2023) use SAC (Haarnoja et al.,, 2018).

The three main functions that have to be represented are the contextual goal-conditioned policies (GCPs), GCP values and goal-conditioned Q functions. We propose to approximate these functions with neural networks.

In the following, we assume that every state $s$ is described by a set of features, that is, a feature vector $\mathcal{F}_{s}$. Recall that a GCP $\pi[s_{s},s_{e}]$ has context state $s_{s}$ and behavioral goal state $s_{e}$. Hence, every GCP can be identified by its context and behavioral goal.

There could be one policy network for all GCPs $\pi[\cdot,\cdot]$ that takes as input three feature vectors: a pair of vectors to identify the GCP and one vector to identify the state for which an action recommendation is sought. The output is a probability distribution over actions $A$; the output layer thus has $|A|$ nodes. The action to execute is sampled from this distribution.

There could be one policy-value network for all $R_{g}^{\pi}$ that takes as input three feature vectors: a pair of vectors to identify the GCP and one vector to identify the state representing the desired goal in $G$. The output is a single node for a real number.

There could be one q-value network for $Q(h,s,g)$ that takes as input four feature vectors: a pair of vectors to identify the HLA $h$, one vector to identify the state $s$ and one vector to identify the state representing the desired goal $g\in G$. The output is a single node for a real number.

We could also look at universal value function approximators (UVFAs) for inspiration (Schaul et al.,, 2015).

“Experience replay was first introduced by Lin, (1992) and applied in the Deep-Q-Network (DQN) algorithm later on by Mnih et al. (2015).” (Zhang and Sutton,, 2017)

Archiving or memorizing a selection of trajectories experienced in a replay buffer is employed in most of the algorithms cited in this paper. Maintaining such a memory buffer in HGCPP would be useful for periodically updating $ChooseBevGoal(\cdot)$ and the GCP policy network. We leave the details and integration into the high-level HGCPP algorithm for future work.

Also on the topic of memory, we could generalize the application of generated (remembered) HLAs: Instead of associating each HLA with a plan-tree node, we associate them with a state. In this way, HLAs can be reused from different nodes representing equal or similar states.

Suppose we are attempting to learn $\pi[s,s^{\prime}]$. If no trajectory starting in $s$ reached $s^{\prime}$ after a given learning-budget, then instead of placing $\pi[s,s^{\prime}]$ in $HLAs(s)$, $\pi[s,s^{\prime\prime\prime}]$ is placed in $HLAs(s)$, where $s^{\prime\prime\prime}$ was reached in at least one experienced trajectory (starting in $s$) and is the best (in terms of $ChooseBevGoal(\cdot)$) of all states reached while attempting to learn $\pi[s,s^{\prime}]$. In the formal algorithm (line 3.d.) $s^{\prime\prime}$ is either $s^{\prime}$ or $s^{\prime\prime\prime}$. This idea of relabeling failed attempts as successful is inspired by the Hindsight Experience Replay algorithm (Andrychowicz et al.,, 2017).