Intrinsic Exploration as Multi-Objective RL

A Markov decision process (MDP) is defined by the tuple $<S,A,T,R,\gamma>$. $\mathcal{S}$ and $\mathcal{A}$ are spaces of states $s$ and actions $a$ respectively. The transition function $T:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ encodes the probability to transition to state $s^{\prime}$ when executing action $a$ in state $s$, i.e. $T(s,a,s^{\prime})=p(s^{\prime}|s,a)$. The reward distribution $R$ of support $\mathcal{S}\times\mathcal{A}\times\mathcal{S}$ defines the reward $r$ associated with transition $(s,a,s^{\prime})$. In the simplest case, goal-only rewards are deterministic and unit rewards are given for absorbing goal states, potential negative unit rewards are given for penalized absorbing states, and zero-reward is given elsewhere. $\gamma\in[0,1)$ is a discount factor. Solving a MDP is equivalent to finding the optimal policy $\pi^{*}$ starting from $s_{0}$:

with $a_{i}\sim\pi(s_{i})$, $s_{i+1}\sim T(s_{i},a_{i},\cdot)$, and $r_{i}\sim R(s_{i},a_{i},s_{i+1})$. Model-free RL learns an action-value function $Q$, which encodes the expected long-term discounted value of a state-action pair

Equation 2 can be rewritten recursively, also known as the Bellman equation

$s^{\prime}\sim p(s^{\prime}|s,a)$, $a^{\prime}\sim\pi(s^{\prime})$, which is used to iteratively refine models of $Q$ based on transition data.

While classic RL typically carries out exploration by adding randomness at a policy level (eg. random action, posterior sampling), intrinsic RL focuses on augmenting rewards with an exploration bonus. This approach was presented in Chentanez et al. (2005), in which agents aim to maximize a total reward $r_{total}$ for transition $(s,a,r,s^{\prime})$:

where $r^{e}$ is the exploration bonus and $\xi$ a user-defined parameter weighting exploration. The second term encourages agents to select state-action pairs for which they previously received high exploration bonuses. The definition of $r^{e}$ has been the focus of much recent theoretical and applied work; examples include model prediction error Stadie et al. (2015) or information gain Little and Sommer (2013).

While this formulation enables exploration in well behaved scenarios, it suffers from multiple limitations:

• •

  Exploration bonuses are designed to reflect the information gain at a given time of the learning process. They are initially high, and typically decrease after more transitions are experienced, making it a non-stationary target. Updating $Q$ with non-stationary targets results in higher data requirements, especially when environment rewards are stationary.

• •

  The exploration bonus given for reaching new areas of the state-action space persists in the estimate of $Q$. As a consequence, agents tend to over-explore and may be stuck oscillating between neighbouring states.

• •

  There is no dynamic control over the exploration-exploitation balance, as changing parameter $\xi$ only affects future total rewards. Furthermore, it would be desirable to control generating trajectories for pure exploration or pure exploitation, as these two quantities may conflict.

This work presents a framework for enhancing intrinsic exploration, which does not suffer from the previously stated limitations.

Multi-objective RL seeks to learn policies solving multiple competing objectives by learning how to solve for each objective individually Roijers et al. (2013). In multi-objective RL, the reward function describes a vector of $n$ rewards instead of a scalar. The value function also becomes a vector $\bm{Q}$ defined as

where $\bm{r}_{i}$ is the vector of rewards at step $i$ in which each coordinate corresponds to one objective. For simplicity, the overall objective is often expressed as the sum of all individual objectives; $\bm{Q}$ can be converted to a scalar state-action value function with a linear scalarization function: $Q_{\bm{\omega}}(s,a)=\bm{\omega}^{T}\bm{Q}(s,a)$, where $\bm{\omega}$ are weights governing the relative importance of each objective.

The advantage of the multi-objective RL formulation is to allow learning policies for all combinations of $\bm{\omega}$, even if the balance between each objective is not explicitly defined prior to learning. Moreover, if $\bm{\omega}$ is a function of time, policies for new values of $\bm{\omega}$ are available without additional computation. Conversely, with traditional RL methods, a pass through the whole dataset of transitions would be required.

Enhancing exploration with additional rewards can be traced back to the work of Storck et al. (1995) and Meuleau and Bourgine (1999), in which information acquisition is dealt with in an active manner. This type of exploration was later termed intrinsic motivation and studied in Chentanez et al. (2005). This field has recently received much attention, especially in the context of very sparse or goal-only rewards Reinke et al. (2017); Morere and Ramos (2018) where traditional reward functions give too little guidance to RL algorithms.

Extensive intrinsic motivation RL work has focused on domains with simple or discrete spaces, proposing various definitions for exploration bonuses. Starting from reviewing intrinsic motivation in psychology, the work of Oudeyer and Kaplan (2008) presents a definition based on information theory. Maximizing predicted information gain from taking specific actions is the focus of Little and Sommer (2013), applied to learning in the absence of external reward feedback. Using approximate value function variance as an exploration bonus was proposed in Osband et al. (2016). In the context of model-based RL, exploration based on model learning progress Lopes et al. (2012), and model prediction error Stadie et al. (2015); Pathak et al. (2017) were proposed. State visitation counts have been widely investigated, in which an additional model counting previous state-action pair occurrences guides agents towards less visited regions. Recent successes include Bellemare et al. (2016); Fox et al. (2018). An attempt to generalizing counter-based exploration to continuous state spaces was made in Nouri and Littman (2009), by using regression trees to achieve multi-resolution coverage of the state space. Another pursuit for scaling visitation counters to large and continuous state spaces was made in Bellemare et al. (2016) by using density models.

Little work attempted to extend intrinsic exploration to continuous action spaces. A policy gradient RL method was presented in Houthooft et al. (2016). Generalization of visitation counters is proposed in Fox et al. (2018), and interpreted as exploration values. Exploration values are also presented as an alternative to additive rewards in Szita and Lőrincz (2008), where exploration balance at a policy level is mentioned.

Most of these methods typically suffer from high data requirements. One of the reasons for such requirements is that exploration is treated as an ad-hoc problem instead of being the focus of the optimization method. More principled ways to deal with exploration can be found in other related fields. In bandits, the balance between exploration and exploitation is central to the formulation Kuleshov and Precup (2014). For example with upper confidence bound Auer et al. (2002), actions are selected based on the balance between action values and a visitation term measuring the variance in the estimate of the action value. In the bandits setting, the balance is defined at a policy level, and the exploration term is not incorporated into action values like in intrinsic RL.

Similarly to bandits, Bayesian Optimization Jones et al. (1998) (BO) brings exploration at the core of its framework, extending the problem to continuous action spaces. BO provides a data-efficient approach for finding the optimum of an unknown objective. Exploration is achieved by building a probabilistic model of the objective from samples, and exploiting its posterior variance information. An acquisition function such as UCB Cox and John (1992) balances exploration and exploitation, and is at the core of the optimization problem. BO was successfully applied to direct policy search Brochu et al. (2010); Wilson et al. (2014) by searching over the space of policy parameters, casting RL into a supervised learning problem. Searching the space of policy parameters is however not data-efficient as recently acquired step information is not used to improve exploration. Furthermore, using BO as global search over policy parameters greatly restricts parameter dimensionality, hence typically imposes using few expressive and hand-crafted features.

In both bandits and BO formulations, exploration is brought to a policy level where it is a central goal of the optimization process. In this work, we treat exploration and exploitation as two distinct objectives to be optimized. Multi-objective RL Roijers et al. (2013) provides tools which we utilize for defining these two distinct objectives, and balancing them at a policy level. Multi-objective RL allows for making exploration central to the optimization process. While there exist Multi-objective RL methods to find several viable objective weightings such as finding Pareto fronts Perny and Weng (2010), our work focuses on two well defined objectives whose weighting changes during learning. As such, we are mostly interested in the ability to change the relative importance of objectives without requiring training.

Modelling state-action values using a probabilistic model enables reasoning about the whole distribution instead of just its expectation, giving opportunities for better exploration strategies. Bayesian Q-learning Dearden et al. (1998) was first proposed to provide value function posterior information in the tabular case, then extended to more complicated domains by using Gaussian processes to model the state-action function Engel et al. (2005). In this work, authors also discuss decomposition of returns into several terms separating intrinsic and extrinsic uncertainty, which could later be used for exploration. Distribution over returns were proposed to design risk-sensitive algorithms Morimura et al. (2010), and approximated to enhance RL stability in Bellemare et al. (2017). In recent work, Bayesian linear regression is combined to a deep network to provide a posterior on Q-values Azizzadenesheli et al. (2018). Thomson sampling is then used for action selection, but can only guarantee local exploration. Indeed, if all action were experienced in a given state, the uncertainty of Q in this state is not sufficient to drive the agent towards unexplored regions.

To the best of our knowledge, there exists no model-free RL framework treating exploration as a core objective. We present such framework, building on theory from multi-objective RL, bandits and BO. We also present EMU-Q, a solution to exploration based on the proposed framework in fully continuous goal-only domains, relying on reducing the posterior variance of value functions.

This paper extends our earlier work Morere and Ramos (2018). It formalizes a new framework for treating exploration and exploitation as two objectives, provides strategies for online exploration control and new experimental results.

Multi-objective RL extends the classic RL framework by allowing value functions or policies to be learned for individual objectives. Exploitation and exploration are two distinct objectives for RL agents, for which separate value functions $Q$ and $U$ (respectively) can be learned. Policies then need to make use of information from two separate models for $Q$ and $U$. While exploitation value function $Q$ is learned from external rewards, exploration value function $U$ is modelled using exploration rewards.

Aiming to define policies which combine exploration and exploitation, we draw inspiration from Bayesian Optimization Brochu et al. (2010), which seeks to find the maximum of an expensive function using very few samples. It relies on an acquisition function to determine the most promising locations to sample next, based on model posterior mean and variance. The Upper-Confidence Bounds (UCB) acquisition function Cox and John (1992) is popular for its explicit balance between exploitation and exploration controlled by parameter $\kappa\in[0,\infty)$. Adapting UCB to our framework leads to policies balancing $Q$ and $U$. Contrary to most intrinsic RL approaches, our formulation keeps the exploration-exploitation trade-off at a policy level, as in traditional RL. This allows for adapting the exploration-exploitation balance during the learning process without sacrificing data-efficiency, as would be the case with a balance at a reward level. Furthermore, policy level balance can be used to design methods to control the agent’s learning process, e.g. stop exploration after a budget is reached, or encourage more exploration if the agent converged to a sub-optimal solution; see Section 4. Lastly, generating trajectories resulting only from exploration or exploration grants experimenters insight over learning status.

We propose to redefine the objective optimized by RL methods to incorporate both exploration and exploitation at its core. To do so, we consider the following expected balanced return for policy $\pi$:

where we introduced exploration rewards $r^{e}_{i}\sim R^{e}(s_{i},a_{i},s_{i+1})$ and parameter $\kappa\in[0,\infty]$ governing exploration-exploitation balance. Note that we recover Equation 1 by setting $\kappa$ to $0$, hence disabling exploration.

Equation 6 can be further decomposed into

where we have defined the exploration state-action value function $U$, akin to $Q$. Exploration behaviour is achieved by maximizing the expected discounted exploration return $U$. Note that, if $r^{e}$ depends on $Q$, then $U$ is a function of $Q$. For clarity, we omit this potential dependency in notations.

Bellman-type updates for both $Q$ and $U$ can be derived by unrolling the first term in both sums:

By identification we recover the update for $Q$ given by Equation 3 and the following update for $U$:

which is similar to that of $Q$. Learning both $U$ and $Q$ can be seen as combining two agents to solve separate MPDs for goal reaching and exploration, as shown in Figure 1 (right). This formulation is general in that any reinforcement learning algorithm can be used to learn $Q$ and $U$, combined with any exploration bonus. Both state-action value functions can be learned from transition data using existing RL algorithms.

The presented formulation is independent from the choice of exploration rewards, hence many reward definitions from the intrinsic RL literature can directly be applied here.

Note that in the special case $R^{e}=0$ for all states and actions, we recover exploration values from DORA Fox et al. (2018), and if state and action spaces are discrete, we recover visitation counters Bellemare et al. (2016).

Another approach to define $R^{e}$ consists in considering the amount of exploration left at a given state. The exploration reward $r^{e}$ for a transition is then defined as the amount of exploration in the resulting state of the transition, to favour transitions that result in discovery. It can be computed by taking an expectation over all actions:

We defined a function $\sigma$ accounting for the uncertainty associated with a state-action pair. This formulation favours transitions that arrive at states of higher uncertainty. An obvious choice for $\sigma$ is the variance of $Q$-values, to guide exploration towards parts of the state-action space where $Q$ values are uncertain. This formulation is discussed in Section 5. Another choice for $\sigma$ is to use visitation count or its continuous equivalent. Compared to classic visitation counts, this formulation focuses on visitations of the resulting transition state $s^{\prime}$ instead of on the state-action pair of origin $(s,a)$.

Exploration rewards are often constrained to negative values so that by combining an optimistic model for $U$ to negative rewards, optimism in the face of uncertainty guarantees efficient exploration Kearns and Singh (2002). The resulting model creates a gradient of $U$ values; trajectories generated by following this gradient reach unexplored areas of the state-action space. With continuous actions, Equation 12 might not have closed form solution and the expectation can be estimated with approximate integration or sampling techniques. In domains with discrete actions however, the expectation is replaced by a sum over all possible actions.

Goal-only rewards are often defined as deterministic, as they simply reflect goal and penalty states. Because our framework handles exploration in a deterministic way, we simply focus on deterministic policies. Although state-action values $Q$ are still non-stationary (because $\pi$ is), they are learned from a stationary objective $r$. This makes learning policies for exploitation easier.

Following the definition in Equation 6, actions are selected to maximize the expected balanced return $D^{\pi}$ at a given state $s$:

Notice the similarity between the policy given in Equation 13 and UCB acquisition functions from the Bayesian optimization and bandits literature. No additional exploration term is needed, as this policy explicitly balances exploration and exploitation with parameter $\kappa$. This parameter can be tuned at any time to generate trajectories for pure exploration or exploitation, which can be useful to assess agent learning status. Furthermore, strategies can be devised to control $\kappa$ manually or automatically during the learning process. We propose a few strategies in Section 4.

The policy from Equation 13 can further be decomposed into present and future terms:

where the term denoted future is effectively $D^{\pi}(s^{\prime})$. This decomposition highlights the link between this framework and other active learning methods; by setting $\gamma$ to $0$, only the myopic term remains, and we recover the traditional UCB acquisition function from bandits or Bayesian optimization. This decomposition can be seen as an extension of these techniques to a non-myopic setting. Indeed, future discounted exploration and exploitation are also considered within the action selection process. Drawing this connection opens up new avenues for leveraging exploration techniques from the bandits literature.

The method presented in this section for explicitly balancing exploration and exploitation at a policy level is concisely summed up in Algorithm 1. The method is general enough so that it allows learning both $Q$ and $U$ with any RL algorithm, and does not make assumptions on the choice of exploration reward used. Section 5 presents a practical method implementing this framework, while the next section presents advantages and strategies for controlling exploration balance during the agent learning process.

We first present simple pathological cases in which using exploration values provides advantages over additive rewards for exploration, on the Cliff Walking domain.

The first two experiments show that exploration values allow for direct control over exploration such stopping and continuing exploration. Stopping exploration after a budget is reached is simulated by setting exploration parameters (eg. $\kappa$) to $0$ after $30$ episodes and stopping model learning. While exploration values maintain high performance after exploration stops, returns achieved with additive rewards dramatically drop and yield a degenerative policy. When exploration is enabled once again, the two methods continue improving at a similar rate; see Figures 2a and 2b. Note that when exploration is disabled, there is a jump in returns with exploration values, as performance for pure exploitation is evaluated. However, it is never possible to be sure a policy is generated from pure exploitation when using additive rewards, as parts of bonus exploration rewards are encoded within learned Q-values.

The third experiment demonstrates that stochastic transitions with higher probability of random action ($p=0.1$) lead to increased return variance and poor performance with additive rewards, while exploration values only seem mildly affected. As shown in Figure 2c, even $\epsilon$-greedy appears to solve the task, suggesting stochastic transitions provide additional random exploration. It is unclear why the additive rewards method is affected negatively.

Lastly, the fourth experiments shows environment reward magnitude is paramount to achieving good performance with additive rewards; see Figure 2d. Even though exploration parameters balancing environment and exploration bonus rewards are scaled to maintain equal amplitude between the two terms, additive rewards suffer from degraded performance. This is due to two reward quantities being incorporated into a single model for Q, which also needs to be initialized optimistically with respect to both quantities. When the two types of rewards have different amplitude, this causes a problem. Exploration values do not suffer from this drawback as separate models are learned based on these two quantities, hence resulting in unchanged performance.

We now present strategies for automatically controlling the exploration-exploitation balance during the learning process. The following experiments also make use of the Taxi domain.

Exploration parameter $\kappa$ is decreased over time according to the following schedule $\kappa(t)=\frac{1}{1+ct}$, where $c$ governs decay rate. Higher values of $c$ result in reduced exploration after only a few episodes, whereas lower values translate to almost constant exploration. Results displayed in Figure 3 indicate that decreasing exploration leads to fast convergence to returns relatively close to maximum return, as shown when setting $c=10^{5}$. However choosing a reasonable value $c=0.1$ first results in lower performance, but enables finding a policy with higher returns later. Such behaviour is more visible with very small values such as $c=10^{-3}$ which corresponds to almost constant $\kappa$.

We now show how direct control over exploration parameter $\kappa$ can be taken advantage of to stop learning automatically once a predefined target is met. On the taxi domain, exploration is first stopped after an exploration budget is exhausted. Results comparing additive rewards to exploration values for different budgets of $100$, $300$ and $500$ episodes are given in Figure 4a. These clearly show that when stopping exploration after the budget is reached, exploration value agents can generate purely exploiting trajectories achieving near optimal return whereas additive reward agents fail to converge on an acceptable policy.

Lastly, we investigate stopping exploration automatically once a target return is reached. After each learning episode, $5$ test episodes with pure exploitation are run to score the current policy. If all $5$ test episodes score returns above $0.1$, the target return is reached and exploration stops. Results for this experiment are shown in Figure 4b and Table 1. Compared to additive rewards, exploration values display better performance after target is reach as well as faster target reaching.

Exploration values were experimentally shown exploration advantages over additive reward on simple RL domains. The next section presents an algorithm built on the proposed framework which extends to fully continuous state and action spaces and is applicable to more advanced problems.

Modelling $Q$-values with a probabilistic model gives access to variance information representing model uncertainty in expected discounted returns. Because the probabilistic model is learned from expected discounted returns, discounted return variance is not considered. Hence the model variance only reflects epistemic uncertainty, which can be used to drive exploration. This formulation was explored in EMU-Q Morere and Ramos (2018), extending Equation 12 as follows:

where $\mathbb{V}_{max}$ is the maximum possible variance of $Q$, guaranteeing always negative rewards. In practice, $\mathbb{V}_{max}$ depends on the model used to learn $Q$ and its hyper-parameters, and can often be computed analytically. In Equation 15, the variance operator $\mathbb{V}$ computes the epistemic uncertainty of $Q$ values, that is it assesses how confident the model is that it can predict $Q$ values correctly. Note that the MDP stochasticity emerging from transitions, rewards and policy is absorbed by the expectation operator in Equation 2, and so no assumptions are required on the MDP components in this reward definition.

We now seek to obtain a model-free RL algorithm able to explore with few environment interactions, and providing a full predictive distribution on state-action values to fit the exploration reward definition given by Equation 15. Kernel methods such as Gaussian Process TD (GPTD) Engel et al. (2005) and Least-Squares TD (LSTD) Lagoudakis and Parr (2003) are among the most data-efficient model-free techniques. While the former suffers from prohibitive computation requirements, the latter offers an appealing trade-off between data-efficiency and complexity. We now derive a Bayesian RL algorithm that combines the strengths of both kernel methods and LSTD.

The distribution of long-term discounted exploitation returns $G$ can be defined recursively as:

which is an equality in the distributions of the two sides of the equation. Note that so far, no assumption are made on the nature of the distribution of returns. Let us decompose the discounted return $G$ into its mean $Q$ and a random zero-mean residual $q$ so that $Q(s,a)=\mathbb{E}[G(s,a)]$. Substituting and rearranging Equation 16 yields

The only extrinsic uncertainty left in this equation are the reward distribution $R$ and residuals $q$. Assuming rewards are disturbed by zero-mean Gaussian noise implies the difference of residuals $\epsilon$ is Gaussian with zero-mean and precision $\beta$. By modelling $Q$ as a linear function of a feature map $\bm{\phi}_{s,a}$ so that $Q(s,a)=\bm{w}^{T}\bm{\phi}_{s,a}$, estimation of state-action values becomes a linear regression problem of target $\bm{t}$ and weights $\bm{w}$. The likelihood function takes the form

where independent transitions are denoted as $x_{i}=(s_{i},a_{i},r_{i},s^{\prime}_{i},a^{\prime}_{i})$. We now treat the weights as random variables with zero-mean Gaussian prior $p(\bm{w})=\mathcal{N}(\bm{w}|\bm{0},\alpha^{-1}\bm{I})$. The weight posterior distribution is

where $\bm{\Phi}_{\bm{s},\bm{a}}=\{\bm{\phi}_{s_{i},a_{i}}\}^{N}_{i=1}$, $\bm{Q}^{\prime}=\{Q(s^{\prime}_{i},a^{\prime}_{i})\}^{N}_{i=1}$, and $\bm{r}=\{r_{i}\}^{N}_{i=1}$. The predictive distribution is also Gaussian, yielding

The predictive variance $\mathbb{V}[p(t|\bm{x},\bm{t},\alpha,\beta)]$ encodes the intrinsic uncertainty in $Q(s,a)$, due to the subjective understanding of the MDP’s model; it is used to compute $r^{e}$ in Equation 15.

The derivation for $U$ is similar, replacing $r$ with $r^{e}$ and $t$ with $t^{e}=R^{e}(s,a,s^{\prime})+\gamma U(s^{\prime},a^{\prime})$. Note that because $\bm{S}$ does not depend on rewards, it can be shared by both models. Hence, with $\bm{U}^{\prime}=\{U(s^{\prime}_{i},a^{\prime}_{i})\}^{N}_{i=1}$,

This model gracefully adapts to iterative updates at each step, by substituting the current prior with the previous posterior. Furthermore, the Sherman-Morrison equality is used to compute rank-1 updates of matrix $\bm{S}$ with each new data point $\phi_{s,a}$:

This update only requires a matrix-to-vector multiplication and saves the cost of inverting a matrix at every step. Hence the complexity cost is reduced from $O(M^{3})$ to $O(M^{2})$ in the number of features $M$. An optimized implementation of EMU-Q is given in algorithm 2.

End of episode updates for $\bm{m}_{Q}$ and $\bm{m}_{U}$ (line 15 onward) are analogous to policy iteration, and although not mandatory, greatly improve convergence speed. Note that because $r^{e}$ is a non-stationary target, recomputing it after each episode with the updated posterior on $Q$ provides the model on $U$ with more accurate targets, thereby improving learning speed.

We presented a simple method to learn $Q$ and $U$ as linear functions of states-actions features. While powerful when using a good feature map, linear models typically require experimenters to define meaningful features on a problem specific basis. In this section, we introduce random Fourier features (RFF) Rahimi and Recht (2008), a kernel approximation technique which allows linear models to enjoy the expressivity of kernel methods. It should be noted that these features are different from Fourier basis Konidaris et al. (2011) (detailed in supplementary material), which do not approximate kernel functions. Although RFF were recently used to learn policy parametrizations Rajeswaran et al. (2017), to the best of our knowledge, this is the first time RFF are applied to the value function approximation problem in RL.

For any shift invariant kernel, which can be written as $k(\tau)$ with $\tau=\bm{x}-\bm{x}^{\prime}$, a representation based on the Fourier transform can be computed with Bochner’s theorem Gihman and Skorohod (1974).

As the number of features increases, kernel approximation error decreases Sutherland and Schneider (2015); approximating popular shift-invariant kernels to within $\epsilon$ can be achieved with only $M=O(d\epsilon^{-2}\log\frac{1}{\epsilon^{2}})$ features. Additionally, sampling frequencies according to a quasi-random sampling scheme (used in our experiments) reduces kernel approximation error compared to classic Monte-Carlo sampling with the same number of features Yang et al. (2014).

EMU-Q with RFF combines the ease-of-use and expressivity of kernel methods brought by RFF with the convergence properties and speed of linear models.

We investigate EMU-Q’s exploration capabilities on a classic domain known to be hard to explore. It is composed of a chain of $N$ states and two actions, displayed in Figure 6a. Action right (dashed) has probability $1-1/N$ to move right and probability $1/N$ to move left. Action left (solid) is deterministic.

EMU-Q is further evaluated on more challenging RL domains. These feature fully continuous state and action spaces, and are adapted to the goal-only reward setting. In these standard control problems Brockman et al. (2016), classic exploration methods are unable to reach goal states.

In this final experiment, we show the applicability of EMU-Q to realistic problems by demonstrating its efficacy on an advanced robotics simulator. In this robotics problem, the agent needs to learn to control a Jaco manipulator solely from observing joint configuration. Given a position in the 3D space, the agent’s goal is to bring the manipulator finger tips to this goal location by sending torque commands to each of the manipulator joints; see Figure 8a. Designing such target-reaching policies is also known as inverse kinematics for robotic arms, and has been studied extensively. Instead, we focus here on learning a mapping from joint configuration to joint torques on a damaged manipulator. When a manipulator is damaged, previously computed inverse kinematics are not valid anymore, thus being able to learn a new target-reaching policy is important.

We model damage by immobilizing four of the arm joints, making previous inverse kinematics invalid. The target position is chosen randomly to form locations across the reachable space. Episodes terminate with unit reward when the target is reached within $50$ steps, zero rewards are given otherwise. We compare EMU-Q and RFF-Q on this domain, both using $600$ random Fourier features approximating an RBF kernel. Parameters $\alpha,\beta$ and $\kappa$ were manually selected to acceptable values of $0.1,1.0$ and $0.1$ respectively. Figure 8b displays results averaged over $10$ runs. The difference in number of episodes solved shows EMU-Q learns and manages to complete the task more consistently than RFF-Q. This confirms that directed exploration is beneficial, even in more realistic robotics scenario.