Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation

The same as previous meta-RL settings (Rakelly et al., 2019), we consider a task distribution $p(\mathcal{T})$ where each task instance $\mathcal{T}$ induces a Markov Decision Process (MDP). We use a tuple ${\mathcal{M}}_{\mathcal{T}}=\langle\mathcal{S},\mathcal{A},T,R_{\mathcal{T}}\rangle$ to denote the MDP specified by task $\mathcal{T}$. In this paper, we assume task $\mathcal{T}$ does not vary the environment configuration, including state space $\mathcal{S}$, action space $\mathcal{A}$, and transition function $T(s^{\prime}\mid s,a)$. Only reward function $R_{\mathcal{T}}(\cdot\mid s,a)$ conditions on task $\mathcal{T}$, which defines the agent’s objectives for task-specific goals. The meta-RL algorithm is allowed to sample a suite of training tasks $\{{\mathcal{T}}_{i}\}_{i=1}^{N}$ from the task distribution $p(\mathcal{T})$ to support meta-training. In the meta-testing phase, a new set of tasks are drawn from $p(\mathcal{T})$ to evaluate the adaptation performance.

We study the few-shot adaptation with preference-based feedback, in which the agent interacts with a black-box preference oracle $\Omega_{\mathcal{T}}$ rather than directly receiving task-specific rewards from the environment $\mathcal{M}_{\mathcal{T}}$ for meta-testing adaptation. More specifically, the agent can query the preference order of a pair of trajectories $\langle\tau^{(1)},\tau^{(2)}\rangle$ through the black-box oracle $\Omega_{\mathcal{T}}$. Each trajectory is a sequence of observed states and agent actions, denoted by $\tau=\langle s_{0},a_{0},s_{1},a_{1},\cdots,s_{L}\rangle$ where $L$ is the trajectory length. For each query trajectory pair $\langle\tau^{(1)},\tau^{(2)}\rangle$, the preference oracle $\Omega_{\mathcal{T}}$ would return either $\tau^{(1)}\succ\tau^{(2)}$ or $\tau^{(1)}\prec\tau^{(2)}$ according to the task specification $\mathcal{T}$, where $\tau^{(1)}\succ\tau^{(2)}$ means the oracle prefers trajectory $\tau^{(1)}$ to trajectory $\tau^{(2)}$ under the context of task $\mathcal{T}$. When the preference orders of $\tau^{(1)}$ and $\tau^{(2)}$ are equal, both returns are valid. The preference-based adaptation is a weak-supervision setting in comparison with previous meta-RL works using per-step reward signals for few-shot adaptation (Finn et al., 2017; Rakelly et al., 2019). The purpose of this adaptation setting is to simulate the practical scenarios with human-in-the-loop supervision (Wirth et al., 2017; Christiano et al., 2017). We consider two aspects to evaluate the ability of an adaptation algorithm:

• •

  Feedback Efficiency. The central goal of meta-RL is conducting fast adaptation to unseen tasks with a few task-specific feedback. We consider the adaptation efficiency in terms of the number of preference queries to oracle $\Omega_{\mathcal{T}}$. This objective expresses the practical demand that we aim to reduce the burden of preference oracle since human feedback is expensive.

• •

  Error Tolerance. Another performance metric is on the robustness against the noisy oracle feedback. The preference oracle $\Omega_{\mathcal{T}}$ may carry errors in practice, e.g., the human oracle may misunderstand the query message and give wrong feedback. Tolerating such oracle errors is a practical challenge for preference-based reinforcement learning.

We follow the algorithmic framework of Probabilistic Embeddings for Actor-critic RL (PEARL; Rakelly et al., 2019). The task specification $\mathcal{T}$ is modeled by a latent task variable (or latent task embedding) $z\in\mathcal{Z}=\mathbb{R}^{d}$ where $d$ denotes the dimension of the latent space. With this formulation, the overall paradigm of the meta-training procedure resembles a multi-task RL algorithm. Both policy $\pi(a|s;z)$ and value function $Q(s,a;z)$ condition on the latent task variable $z$ so that the representation of $z$ can be end-to-end learned with the RL objective to distinguish different task specifications. During meta-testing, the adaptation is performed in the low-dimensional task embedding space rather than the high-dimensional parameter space.

To infer the task embedding $z$ from the latent task space, PEARL trains an inference network (or context encoder) $q(z|\mathbf{c})$ where $\mathbf{c}$ is the context information including agent actions, observations, and rewards. The output of $q(z|\mathbf{c})$ is probabilistic, i.e., the agent has a probabilistic belief over the latent task space $\mathcal{Z}$ based on its observations and received rewards. We use $q(z)$ to denote the prior distribution for $\mathbf{c}=\emptyset$. The adaptation protocol of PEARL follows the framework of posterior sampling (Strens, 2000). The agent continually updates its belief by interacting with the environment and refine its policy according to the belief state. This algorithmic framework is generalized from Bayesian inference (Thompson, 1933) and has solid background in reinforcement learning theory (Agrawal and Goyal, 2012; Osband et al., 2013; Russo and Van Roy, 2014). However, some recent works show that the empirical performance of neural inference networks highly rely on the access to a dense reward function (Zhang et al., 2021; Hua et al., 2021). When the task-specific reward signals are sparsely distributed along the agent trajectory, the task inference given by context encoder $q(z|\mathbf{c})$ would suffer from low sample efficiency and cannot accurately decode task specification. This issue may worsen in our adaptation setting since only trajectory-wise preference comparisons are available to the agent. It motivates us to explore new methodology for few-shot adaptation beyond classical approaches based on posterior sampling.

The key step of the problem transformation is establishing the correspondence between the preference query in preference-based task inference and the natural-language-based questions in Rényi-Ulam’s game. In classical solutions of Rényi-Ulam’s game, a general format of player B’s questions is to ask whether the element in player A’s mind belongs to a certain subset of the element universe (Pelc, 2002), whereas the counterpart of question format in preference-based task inference is restricted to querying whether the oracle prefers one trajectory to another. To bridge this gap, we use a model-based approach to connect the oracle preference feedback with the latent space of task embeddings. We train a preference predictor $f_{\psi}(\cdot;z)$ that predicts the oracle preference according to the task embedding $z$. This preference predictor can transform each oracle preference feedback to a separation on the latent task space, i.e., the preference prediction $f_{\psi}(\cdot;z)$ given by task embeddings in a subspace of $z$ can match the oracle feedback, and the task embeddings in the complementary subspace lead to wrong predictions. Through this transformation, the task inference algorithm can convert the binary preference feedback to the assessment of a subspace of latent task embeddings, which works in the same mechanism as previous solutions to Rényi-Ulam’s game.

To realize the problem transformation, we consider a simple and direct implementation of the preference predictor. We train $f_{\psi}(\cdot;z)$ on the meta-training tasks by optimizing the preference loss:

where $\psi$ denotes the parameters of the preference predictor, $D_{\text{KL}}(\cdot\|\cdot)$ denotes the Kullback–Leibler divergence, and $\mathbb{I}[\tau^{(1)}\succ\tau^{(2)}\mid\mathcal{T}]$ is the ground-truth preference order specified by the task specification $\mathcal{T}$ (e.g., specified by the reward function on training tasks). The trajectory pair $\langle\tau^{(1)},\tau^{(2)}\rangle$ is drawn from the experience buffer, and $z$ is the task embedding vector encoding $\mathcal{T}$. More implementation details are included in Appendix B.2.

To facilitate discussions, we introduce some notations to model the interaction with the preference oracle $\Omega_{\mathcal{T}}$. Suppose the adaptation budget supports $K$ online preference queries to oracle $\Omega_{\mathcal{T}}$, which divides this interactive procedure into $K$ rounds. We define the notation of query context set as Definition 2 to represent the context information extracted from the preference queries during the online oracle interaction.

The query context set $\mathcal{Q}_{k}$ concludes the context information obtained from the oracle $\Omega$ in the first $k$ rounds. After completing the query at round $k$, the task inference algorithm needs to decide the next-round query trajectory pair $\langle\tau_{k+1}^{(1)},\tau_{k+1}^{(2)}\rangle$ based on the context information stored in $\mathcal{Q}_{k}$. By leveraging the model-based problem transformation to Rényi-Ulam’s game, we can assess the quality of a task embedding $z$ by counting the number of mismatches with respect to oracle preferences, denoted by $\mathcal{E}(z;\mathcal{Q}_{k})$:

The overall algorithmic framework of our preference-based few-shot adaptation method, Adaptation with Noisy OracLE (ANOLE), is summarized in Algorithm 1.

The first step of our task inference algorithm is sampling a candidate pool $\widehat{Z}$ of latent task embeddings from the prior distribution $q(z)$. Then we perform $K$ rounds of candidate selection by querying the preference oracle $\Omega_{\mathcal{T}}$. The design of the query generation protocol $\textsc{GenerateQuery}(\cdot)$ is a critical component of our task inference algorithm and will be introduced in section 3.3. The final decision would be the task embedding with minimum mismatch with the oracle preference, i.e., $\arg\min_{z\in\widehat{Z}}\mathcal{E}(z;\mathcal{Q}_{K})$.

The basic idea is conducting a binary-search-like protocol to leverage the binary structure of preference feedback: After each round of oracle interaction, we shrink the candidate pool $\widehat{Z}$ by removing those task embeddings leading to wrong preference predictions $f_{\psi}(\cdot;z)$. An ideal implementation of such a binary-search protocol with noiseless feedback is expected to roughly eliminate half of candidates using each single oracle preference feedback, which achieves the information-theoretic lower bound of interaction costs. In practice, we pursue to handle noisy feedback, since both the preference oracle $\Omega_{\mathcal{T}}$ and the preference predictor $f_{\psi}(\cdot;z)$ may carry errors. An error-tolerant binary-search protocol requires to establish an information quantification (e.g., an uncertainty metric) to evaluate the information gain of each noisy oracle feedback. The goal of oracle interaction is to rapidly reduce the uncertainty of task inference rather than simply eliminate the erroneous candidates. In this paper, we extend an information-theoretic tool called Berlekamp’s volume (Berlekamp, 1968) to develop such an uncertainty quantification.

One classical tool to deal with erroneous information in search problems is Berlekamp’s volume, which is first proposed by Berlekamp (1968) and has been explored by subsequent works in numerous variants of Rényi-Ulam’s game (Rivest et al., 1980; Pelc, 1987; Lawler and Sarkissian, 1995; Aigner, 1996; Cicalese and Deppe, 2007). The primary purpose of Berlekamp’s volume is to mimic the notion of Shannon entropy from information theory (Shannon, 1948) and specialize in the applications to noisy-channel communication (Shannon, 1956) and error-tolerant search (Rényi, 1961). We refer readers to Pelc (2002) for a comprehensive literature review of the applications of Berlekamp’s Volume and the solutions to Rényi-Ulam’s game.

In Definition 3, we rearrange the definition of Berlekamp’s volume to suit the formulation of preference-based learning.

As stated in Definition 3, the configuration of Berlekamp’s volume has two hyper-parameters: the total number of queries $K$, and the maximum number of erroneous feedback $K_{E}$ within all queries. This error mode refers to Berlekamp (1968)’s noisy-channel model. Berlekamp’s volume is a tool for designing robust query strategy to guarantee the tolerance of bounded number of feedback errors.

Berlekamp’s volume measures the uncertainty of the unknown latent task variable $z$ together with the randomness carried by the noisy oracle feedback. To give a more concrete view, we first show how the value of Berlekamp’s volume ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}_{k})$ changes when receiving the feedback of a new preference query. Depending on the binary oracle feedback for the query trajectory pair $(\tau_{k}^{(1)},\tau_{k}^{(2)})$, the query context set may be updated to two possible statuses:

where $\mathcal{Q}^{(u)\succ(v)}_{k}$ denotes the updated status in the case of receiving oracle feedback $\tau_{k}^{(u)}\succ\tau_{k}^{(v)}$. The relation between ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}_{k-1})$ and ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}_{k})$ is characterized by Proposition 3.3, which is the foundation of Berlekamp’s volume for developing error-tolerant algorithms. {restatable}[Volume Conservation Law]propositionVolumeProposition For any query context set $\mathcal{Q}_{k-1}$ and arbitrary query trajectory pair $(\tau_{k}^{(1)},\tau_{k}^{(2)})$, the relation between ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}_{k-1})$ and ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}_{k})$ satisfies

The proofs of all statements presented in this section are deferred to Appendix A. As shown by Proposition 3.3, each preference query would partition the volume ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}_{k-1})$ into two subsequent branches, ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}^{(1)\succ(2)}_{k})$ and ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}^{(2)\succ(1)}_{k})$. The selection of preference queries does not alter the volume sum of subsequent branches. Since the values of ${\mathcal{BV}ol}_{\widehat{Z}}(\cdot)$ are non-negative integers, the volume is monotonically decreased with the online query procedure. When the volume is eliminated to the unit value, the selection of task embedding with minimum mismatches $\mathcal{E}(z;\mathcal{Q}_{k})$ would become deterministic (see Proposition 3.3).

[Unit of Volume]propositionUnitProposition Given a query context set $\mathcal{Q}_{k}$ with ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}_{k})=1$, there exists exactly one task embedding candidate $z\in\widehat{Z}$ satisfying $\mathcal{E}(z;\mathcal{Q}_{k})\leq K_{E}$.

We can represent the preference-based task inference protocol by a decision tree, where each $\mathcal{Q}_{k}$ with ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}_{k})=1$ corresponds to a leaf node and each preference query is a decision rule. The value of Berlekamp’s volume ${\mathcal{BV}ol}_{\widehat{Z}}(\mathcal{Q}_{k})$ corresponds to the number of leaf nodes remaining in the subtree rooted with context set $\mathcal{Q}_{k}$. The value of $z$-conditioned volume $vol_{z}(\mathcal{Q}_{k})$ counts the number of leaf nodes with $z$ as the final decision. Each path from an ancestor node to a leaf node can be mapped to a valid feedback sequence that does not violate $\mathcal{E}(z;\mathcal{Q}_{K})\leq K_{E}$. From this perspective, Berlekamp’s volume quantifies the uncertainty of task inference by counting the number of valid feedback sequences for the incoming queries.

Given Berlekamp’s volume as the uncertainty quantification, the query strategy can be constructed directly by maximizing the worst-case uncertainty reduction:

where $\textsc{GenerateQuery}(\cdot)$ refers to the query generation step at line 5 of Algorithm 1. This design follows the principle of binary search. If the Berlekamp’s volumes of two subsequent branches are well balanced, the uncertainty can be exponentially reduced no matter which feedback the oracle responds. In our implementation, the $\arg\min$ operator in Eq. (6) is approximated by sampling a mini-batch of trajectory pairs from the experience buffer. The query content is determined by finding the best trajectory pair within the sampled mini-batch. More implementation details are included in Appendix B.2.

We adopt six meta-RL benchmark tasks created by Rothfuss et al. (2019), which are widely used by meta-RL works to evaluate the performance of few-shot policy adaptation (Rakelly et al., 2019; Zintgraf et al., 2020; Fakoor et al., 2020). These environment settings consider four ways to vary the task specification $\mathcal{T}$: forward/backward (-Fwd-Back), random target velocity (-Rand-Vel), random target direction (-Rand-Dir), and random target location (-Rand-Goal). We simulate the preference oracle $\Omega_{\mathcal{T}}$ by comparing the ground-truth trajectory return given by the MuJoCo-based environment simulator. The adaptation protocol cannot observe these environmental rewards during meta-testing and can only query the preference oracle $\Omega_{\mathcal{T}}$ to extract the task information. In addition, we impose a random noisy perturbation on the oracle feedback. Each binary feedback would be flipped with probability $\epsilon$. We consider such independently distributed errors rather than the bounded-number error mode to evaluate the empirical performance, since it is more realistic to simulate human’s unintended errors. A detailed description of the experiment setting is included in Appendix B.1.

Note that ANOLE is an adaptation module and can be built upon any meta-RL or multi-task RL algorithms with latent policy encoding (Hausman et al., 2018; Eysenbach et al., 2019; Pong et al., 2020; Lynch et al., 2020; Gupta et al., 2020). To align with the baseline algorithms, we implement a meta-training procedure similar to PEARL (Rakelly et al., 2019). The same as PEARL, our policy optimization module extends soft actor-critic (SAC; Haarnoja et al., 2018) with the latent task embedding $z$, i.e., the policy and value functions are represented by $\pi_{\theta}(a\mid s;z)$ and $Q_{\phi}(s,a;z)$ where $\theta$ and $\phi$ denote the parameters. One difference from PEARL is the removal of the inference network, since it is no longer used in meta-testing. Instead, we set up a bunch of trainable latent task variables $\{z_{i}\}_{i=1}^{N}$ to learn the multi-task policy. More implementation details are included in Appendix B.2. The source code of our ANOLE implementation and experiment scripts are available at https://github.com/Stilwell-Git/Adaptation-with-Noisy-OracLE.

We compare the performance with four baseline adaptation strategies. Two of these baselines are built upon the same meta-training pre-processing as ANOLE but do not use Berlekamp’s volume to construct preference queries.

• •

  Greedy Binary Search. We conduct a simple and direct implementation of binary-search-like task inference. When generating preference queries, it simply ignores all candidates that have made at least one wrong preference predictions, and the query trajectory pair only aims to partition the remaining candidate pool into two balanced separations.

• •

  Random Query. We also include the simplest baseline that constructs the preference query by drawing a random trajectory pair from the experience buffer. This baseline serves an ablation study to investigate the benefits of removing the inference network.

In addition, we implement two variants of PEARL (Rakelly et al., 2019) that use a probabilistic context encoder to infer task specification from preference-based feedback.

• •

  PEARL. We modify PEARL’s context encoder to handle the preference-based feedback. More specifically, the context encoder is a LSTM-based network that takes an ordered trajectory pair as inputs to make up the task embedding $z$, i.e., the input trajectory pair has been sorted by oracle preference. In meta-training, the preference is labeled by the ground-truth rewards from the environment simulator. During adaptation, the PEARL-based agent draws random trajectory pairs from the experience buffer to query the preference oracle, and the oracle feedback is used to update the posterior of task variable $z$.

• •

  PEARL+Augmentation. We implement a data augmentation method for PEARL’s meta-training to pursue the error tolerance of preference-based inference network. We impose random errors to the preference comparison when training the preference-based context encoder so that the inference network is expected to have some extent of error tolerance.

We include more implementation details of these baselines in Appendix B.3.