Active Exploration for Learning
Symbolic Representations

We assume that the agent’s environment can be described by a semi-Markov decision process (SMDP), given by a tuple $D=(S,O,R,P,\gamma)$, where $S\subseteq\mathbb{R}^{d}$ is a $d$-dimensional continuous state space, $O(s)$ returns a set of temporally extended actions, or options [21] available in state $s\in S$, $R(s^{\prime},t,s,o)$ and $P(s^{\prime},t\mid s,o)$ are the reward received and probability of termination in state $s^{\prime}\in S$ after $t$ time steps following the execution of option $o\in O(s)$ in state $s\in S$, and $\gamma\in(0,1]$ is a discount factor. In this paper, we are not concerned with the time taken to execute $o$, so we use $P(s^{\prime}\mid s,o)=\int P(s^{\prime},t\mid s,o)\mathrm{d}t$.

An option $o$ is given by three components: $\pi_{o}$, the option policy that is executed when the option is invoked, $I_{o}$, the initiation set consisting of the states where the option can be executed from, and $\beta_{o}(s)\rightarrow[0,1]$, the termination condition, which returns the probability that the option will terminate upon reaching state $s$. Learning models for the initiation set, rewards, and transitions for each option, allows the agent to reason about the effect of its actions in the environment. To learn these option models, the agent has the ability to collect observations of the forms $(s,O(s))$ when entering a state $s$ and $(s,o,s^{\prime},r,t)$ upon executing option $o$ from $s$.

We are specifically interested in learning option models which allow the agent to easily evaluate the success probability of plans. A plan is a sequence of options to be executed from some starting state, and it succeeds if and only if it is able to be run to completion (regardless of the reward). Thus, a plan $\{o_{1},o_{2},...,o_{n}\}$ with starting state $s$ succeeds if and only if $s\in I_{o_{1}}$ and the termination state of each option (except for the last) lies in the initiation set of the following option, i.e. $s^{\prime}\sim P(s^{\prime}\mid s,o_{1})\in I_{o_{2}}$, $s^{\prime\prime}\sim P(s^{\prime\prime}\mid s^{\prime},o_{2})\in I_{o_{3}}$, and so on.

Recent work [12, 13] has shown how to automatically generate a symbolic representation that supports such queries, and is therefore suitable for planning. This work is based on the idea of a probabilistic symbol, a compact representation of a distribution over infinitely many continuous, low-level states. For example, a probabilistic symbol could be used to classify whether or not the agent is currently in front of a door, or one could be used to represent the state that the agent would find itself in after executing its ‘open the door’ option. In both cases, using probabilistic symbols also allows the agent to be uncertain about its state.

The following two probabilistic symbols are provably sufficient for evaluating the success probability of any plan [13]; the probabilistic precondition: $\mathrm{Pre}(o)=P(s\in I_{o})$, which expresses the probability that an option $o$ can be executed from each state $s\in S$, and the probabilistic image operator:

which represents the distribution over termination states if an option $o$ is executed from a distribution over starting states $Z$. These symbols can be used to compute the probability that each successive option in the plan can be executed, and these probabilities can then be multiplied to compute the overall success probability of the plan; see Figure 1 for a visual demonstration of a plan of length 2.

Subgoal Options  Unfortunately, it is difficult to model $\mathrm{Im}(o,Z)$ for arbitrary options, so we focus on restricted types of options. A subgoal option [19] is a special class of option where the distribution over termination states (referred to as the subgoal) is independent of the distribution over starting states that it was executed from, e.g. if you make the decision to walk to your kitchen, the end result will be the same regardless of where you started from.

For subgoal options, the image operator can be replaced with the effects distribution: $\mathrm{Eff}(o)=\mathrm{Im}(o,Z),\forall Z(S)$, the resulting distribution over states after executing $o$ from any start distribution $Z(S)$. Planning with a set of subgoal options is simple because for each ordered pair of options $o_{i}$ and $o_{j}$, it is possible to compute and store $G(o_{i},o_{j})$, the probability that $o_{j}$ can be executed immediately after executing $o_{i}$: $G(o_{i},o_{j})=\int_{S}\mathrm{Pre}(o_{j},s)\mathrm{Eff}(o_{i})(s)\mathrm{d}s.$

We use the following two generalizations of subgoal options: abstract subgoal options model the more general case where executing an option leads to a subgoal for a subset of the state variables (called the mask), leaving the rest unchanged. For example, walking to the kitchen leaves the amount of gas in your car unchanged. More formally, the state vector can be partitioned into two parts $s=[a,b]$, such that executing $o$ leaves the agent in state $s^{\prime}=[a,b^{\prime}]$, where $P(b^{\prime})$ is independent of the distribution over starting states. The second generalization is the (abstract) partitioned subgoal option, which can be partitioned into a finite number of (abstract) subgoal options. For instance, an option for opening doors is not a subgoal option because there are many doors in the world, however it can be partitioned into a set of subgoal options, with one for every door.

The subgoal (and abstract subgoal) assumptions propose that the exact state from which option execution starts does not really affect the options that can be executed next. This is somewhat restrictive and often does not hold for options as given, but can hold for options once they have been partitioned. Additionally, the assumptions need only hold approximately in practice.

Now we show how to construct a more general model than $G$ that can be used for planning with abstract partitioned subgoal options. The advantages of our approach versus previous methods are that our algorithm is much faster, and the resulting model is Bayesian, both of which are necessary for the active exploration algorithm introduced in the next section.

Recall that the agent can collect observations of the forms $(s,o,s^{\prime})$ upon executing option $o$ from $s$, and $(s,O(s))$ when entering a state $s$, where $O(s)$ is the set of available options in state $s$. Given a sequence of observations of this form, the first step of our approach is to find the factors [13], partitions of state variables that always change together in the observed data. For example, consider a robot which has options for moving to the nearest table and picking up a glass on an adjacent table. Moving to a table changes the $x$ and $y$ coordinates of the robot without changing the joint angles of the robot’s arms, while picking up a glass does the opposite. Thus, the $x$ and $y$ coordinates and the arm joint angles of the robot belong to different factors. Splitting the state space into factors reduces the number of potential masks (see end of Section 2.2) because we assume that if state variables $i$ and $j$ always change together in the observations, then this will always occur, e.g. we assume that moving to the table will never move the robot’s arms.¹¹1The factors assumption is not strictly necessary as we can assign each state variable to its own factor. However, using this uncompressed representation can lead to an exponential increase in the size of the symbolic state space and a corresponding increase in the sample complexity of learning the symbolic models.

Finding the Factors   Compute the set of observed masks $M$ from the $(s,o,s^{\prime})$ observations: each observation’s mask is the subset of state variables that differ substantially between $s$ and $s^{\prime}$. Since we work in continuous, stochastic domains, we must detect the difference between minor random noise (independent of the action) and a substantial change in a state variable caused by action execution. In principle this requires modeling action-independent and action-dependent differences, and distinguishing between them, but this is difficult to implement. Fortunately we have found that in practice allowing some noise and having a simple threshold is often effective, even in more noisy and complex domains. For each state variable $i$, let $M_{i}\subseteq M$ be the subset of the observed masks that contain $i$. Two state variables $i$ and $j$ belong to the same factor $f\in F$ if and only if $M_{i}=M_{j}$. Each factor $f$ is given by a set of state variables and thus corresponds to a subspace $S_{f}$. The factors are updated after every new observation.

Let $S^{*}$ be the set of states that the agent has observed and let $S^{*}_{f}$ be the projection of $S^{*}$ onto the subspace $S_{f}$ for some factor $f$, e.g. in the previous example there is a $S^{*}_{f}$ which consists of the set of observed robot $(x,y)$ coordinates. It is important to note that the agent’s observations come only from executing partitioned abstract subgoal options. This means that $S^{*}_{f}$ consists only of abstract subgoals, because for each $s\in S^{*}$, $s_{f}$ was either unchanged from the previous state, or changed to another abstract subgoal. In the robot example, all $(x,y)$ observations must be adjacent to a table because the robot can only execute an option that terminates with it adjacent to a table or one that does not change its $(x,y)$ coordinates. Thus, the states in $S^{*}$ can be imagined as a collection of abstract subgoals for each of the factors. Our next step is to build a set of symbols for each factor to represent its abstract subgoals, which we do using unsupervised clustering.

Finding the Symbols  For each factor $f\in F$, we find the set of symbols $Z^{f}$ by clustering $S^{*}_{f}$. Let $Z^{f}(s_{f})$ be the corresponding symbol for state $s$ and factor $f$. We then map the observed states $s\in S^{*}$ to their corresponding symbolic states $s^{d}=\{Z^{f}(s_{f}),\forall f\in F\}$, and the observations $(s,O(s))$ and $(s,o,s^{\prime})$ to $(s^{d},O(s))$ and $(s^{d},o,s^{\prime d})$, respectively.

In the robot example, the $(x,y)$ observations would be clustered around tables that the robot could travel to, so there would be a symbol corresponding to each table.

We want to build our models within the symbolic state space $S^{d}$. Thus we define the symbolic precondition, $\mathit{Pre}(o,s^{d})$, which returns the probability that the agent can execute an option from some symbolic state, and the symbolic effects distribution for a subgoal option $o$, $\mathit{Eff}(o)$, maps to a subgoal distribution over symbolic states. For example, the robot’s ‘move to the nearest table’ option maps the robot’s current $(x,y)$ symbol to the one which corresponds to the nearest table.

The next step is to partition the options into abstract subgoal options (in the symbolic state space), e.g. we want to partition the ‘move to the nearest table’ option in the symbolic state space so that the symbolic states in each partition have the same nearest table.

Partitioning the Options  For each option $o$, we initialize the partitioning $P^{o}$ so that each symbolic state starts in its own partition. We use independent Bayesian sparse Dirichlet-categorical models [20] for the symbolic effects distribution of each option partition.²²2We use sparse Dirichlet-categorical models because there are a combinatorial number of possible symbolic state transitions, but we expect that each partition has non-zero probability for only a small number of them. We then perform Bayesian Hierarchical Clustering [9] to merge partitions which have similar symbolic effects distributions.³³3We use the closed form solutions for Dirichlet-multinomial models provided by the paper.

There is a special case where the agent has observed that an option $o$ was available in some symbolic states $S^{d}_{a}$, but has yet to actually execute it from any $s^{d}\in S^{d}_{a}$. These are not included in the Bayesian Hierarchical Clustering, instead we have a special prior for the partition of $o$ that they belong to. After completing the merge step, the agent has a partitioning $P^{o}$ for each option $o$. Our prior is that with probability $q_{o}$,⁴⁴4This is a user specified parameter. each $s^{d}\in S^{d}_{a}$ belongs to the partition $p^{o}\in P^{o}$ which contains the symbolic states most similar to $s^{d}$, and with probability $1-q_{o}$ each $s^{d}$ belongs to its own partition. To determine the partition which is most similar to some symbolic state, we first find $A^{o}$, the smallest subset of factors which can still be used to correctly classify $P^{o}$. We then map each $s^{d}\in S^{d}_{a}$ to the most similar partition by trying to match $s^{d}$ masked by $A^{o}$ with a masked symbolic state already in one of the partitions. If there is no match, $s^{d}$ is placed in its own partition.

Our final consideration is how to model the symbolic preconditions. The main concern is that many factors are often irrelevant for determining if some option can be executed. For example, whether or not you have keys in your pocket does not affect whether you can put on your shoe.

Modeling the Symbolic Preconditions  Given an option $o$ and subset of factors $F^{o}$, let $S^{d}_{F^{o}}$ be the symbolic state space projected onto $F^{o}$. We use independent Bayesian Beta-Bernoulli models for the symbolic precondition of $o$ in each masked symbolic state $s^{d}_{F^{o}}\in S^{d}_{F^{o}}$. For each option $o$, we use Bayesian model selection to find the the subset of factors $F_{*}^{o}$ which maximizes the likelihood of the symbolic precondition models.

The final result is a distribution over symbolic option models $H$, which consists of the combined sets of independent symbolic precondition models $\{\mathit{Pre}(o,s^{d}_{F_{*}^{o}});\forall o\in O,\forall s^{d}_{F_{*}^{o}}\in S^{d}_{F_{*}^{o}}\}$ and independent symbolic effects distribution models $\{\mathit{Eff}(o,p^{o});\forall o\in O,\forall p^{o}\in P^{o}\}.$

The complete procedure is given in Algorithm 1. A symbolic option model $h\sim H$ can be sampled by drawing parameters for each of the Bernoulli and categorical distributions from the corresponding Beta and sparse Dirichlet distributions, and drawing outcomes for each $q_{o}$. It is also possible to consider distributions over other parts of the model such as the symbolic state space and/or a more complicated one for the option partitionings, which we leave for future work.

In the previous section we have shown how to efficiently compute a distribution over symbolic option models $H$. Recall that the ultimate purpose of $H$ is to compute the success probabilities of plans (see Section 2.2). Thus, the quality of $H$ is determined by the accuracy of its predicted plan success probabilities, and efficiently learning $H$ corresponds to selecting the sequence of observations which maximizes the expected accuracy of $H$. However, it is difficult to calculate the expected accuracy of $H$ over all possible plans, so we define a proxy measure to optimize which is intended to represent the amount of uncertainty in $H$. In this section, we introduce our proxy measure, followed by an algorithm for finding the exploration policy which optimizes it. The algorithm operates in an online manner, building $H$ from the data collected so far, using $H$ to select an option to execute, updating $H$ with the new observation, and so on.

First we define the standard deviation $\sigma_{H}$, the quantity we use to represent the amount of uncertainty in $H$. To define the standard deviation, we need to also define the distance and mean.

We define the distance $K$ from $h_{2}\in H$ to $h_{1}\in H$, to be the sum of the Kullback-Leibler (KL) divergences⁵⁵5The KL divergence has previously been used in other active exploration scenarios [17, 15]. between their individual symbolic effect distributions plus the sum of the KL divergences between their individual symbolic precondition distributions:⁶⁶6Similarly to other active exploration papers, we define the distance to depend only on the transition models and not the reward models.

We define the mean, $\mathbb{E}[H]$, to be the symbolic option model such that each Bernoulli symbolic precondition and categorical symbolic effects distribution is equal to the mean of the corresponding Beta or sparse Dirichlet distribution:

The standard deviation $\sigma_{H}$ is then simply: $\sigma_{H}=\mathbb{E}_{h\sim H}[K(h,\mathbb{E}[H])]$. This represents the expected amount of information which is lost if $\mathbb{E}[H]$ is used to approximate $H$.

Now we define the optimal exploration policy for the agent, which aims to maximize the expected reduction in $\sigma_{H}$ after $H$ is updated with new observations. Let $H(w)$ be the posterior distribution over symbolic models when $H$ is updated with symbolic observations $w$ (the partitioning is not updated, only the symbolic effects distribution and symbolic precondition models), and let $W(H,i,\pi)$ be the distribution over symbolic observations drawn from the posterior of $H$ if the agent follows policy $\pi$ for $i$ steps. We define the optimal exploration policy $\pi^{*}$ as:

For the convenience of our algorithm, we rewrite the second term by switching the order of the expectations: $\mathbb{E}_{w\sim W(H,i,\pi)}[\mathbb{E}_{h\sim H(w)}[K(h,\mathbb{E}[H(w)])]]=\mathbb{E}_{w\sim W(h,i,\pi)}[K(h,\mathbb{E}[H(w)])]].$

Note that the objective function is non-Markovian because $H$ is continuously updated with the agent’s new observations, which changes $\sigma_{H}$. This means that $\pi^{*}$ is non-stationary, so Algorithm 2 approximates $\pi^{*}$ in an online manner using Monte-Carlo tree search (MCTS) [3] with the UCT tree policy [11]. $\pi_{T}$ is the combined tree and rollout policy for MCTS, given tree $T$.

There is a special case when the agent simulates the observation of a previously unobserved transition, which can occur under the sparse Dirichlet-categorical model. In this case, the amount of information gained is very large, and furthermore, the agent is likely to transition to a novel symbolic state. Rather than modeling the unexplored state space, instead, if an unobserved transition is encountered during an MCTS update, it immediately terminates with a large bonus to the score, a similar approach to that of the R-max algorithm [2]. The form of the bonus is -$zg$, where $g$ is the depth that the update terminated and $z$ is a constant. The bonus reflects the opportunity cost of not experiencing something novel as quickly as possible, and in practice it tends to dominate (as it should).