Is RLHF More Difficult than Standard RL?
A Theoretical Perspective

Dueling bandits. Dueling bandits (Yue et al., 2012; Bengs et al., 2021) can be seen as a special case of preference based RL with $H=1$. Many assumptions later applied to preference based RL, such as an underlying utility model with a link function (Yue and Joachims, 2009), the Plackett-Luce model (Saha and Gopalan, 2019), and the Condorcet winner (Zoghi et al., 2014) can be traced back to literature on dueling bandits. A reduction from preference feedback to reward feedback for bandits is proposed by Ailon et al. (2014). The concept of the von Neumann winner, which we employ for general preferences, has been considered in Dudík et al. (2015) for contextual dueling bandits.

RL from human feedback. Using human preferences in RL has been studied for at least a decade (Jain et al., 2013; Busa-Fekete et al., 2014) and is later incorporated with Deep RL (Christiano et al., 2017). It has found empirical success in robotics (Jain et al., 2013; Abramson et al., 2022; Ding et al., 2023), game playing (Ibarz et al., 2018) and fine-tuning large language models (Ziegler et al., 2019; Ouyang et al., 2022; Bai et al., 2022). RL with other forms of human feedback, such as demonstrations and scalar ratings, has also been considered in previous research (Finn et al., 2016; Warnell et al., 2018; Arakawa et al., 2018) but falls beyond the scope of this paper.

Theory of Preference-based RL. For utility-based preferences, Novoseller et al. (2020), Pacchiano et al. (2021) and Zhan et al. (2023b) consider tabular or linear MDPs, and assume that the per-trajectory reward is linear in trajectory features.  Xu et al. (2020) also considers the tabular setting. However, instead of assuming an explicit link function, several structural properties of the preference is assumed which guarantee a Condorcet winner and ensure small regret of a black-box dueling bandit algorithm. Finally, recent works of  Zhu et al. (2023); Zhan et al. (2023a) consider utility-based preferences in the offline setting, assuming the algorithm is provided with a pre-collected human preference (and transition) dataset with good coverage. Compared to the above works, this paper considers the online setting and derives results for utility-based preferences with general function approximation, which are significantly more general than those for tabular or linear MDPs.

For general preferences, Chen et al. (2022) also develops sample-efficient algorithms for finding the von Neumann winner ²²2 While their paper makes Assumption 3.1 in Chen et al. (2022) that seemingly assumes a Condorcet winner, this assumption actually always holds as $\pi^{\star}$ can be a general randomized policy, so their solution concept coincides with the von Neumann winner.. Their algorithm is computationally inefficient in general even when restricted to the tabular setting. Compared to this result, our AMDP-based reduction algorithm (Section 4.2) is computationally efficient in the tabular or linear setting when the comparison only depends on the final states. For general trajectory-based comparison, our results apply to a richer class of RL problems including POMDPs.

Finally, we remark that all prior results develop specialized algorithms and analysis for preference-based RL in a white-box fashion. In contrast, we develop reduction-based algorithms which can directly utilize state-of-the-art results in reward-based RL for preference-based RL. This reduction approach enables the significant generality of the results in this paper compared to prior works.

For utility-based comparison, we assume there exists an underlying reward function $r^{\star}:(\mathcal{S}\times\mathcal{A})^{H}\to[0,1]$. Given a reward function, the value of a policy can be defined as $\mathbb{E}_{\pi}[\sum_{h=1}^{H}r^{\star}(s_{h},a_{h})]$, i.e. the expected cumulative reward obtained by executing the policy. An optimal policy is one that maximizes $\mathbb{E}_{\pi}[\sum_{h=1}^{H}r^{\star}(s_{h},a_{h})]$. We say that $\pi$ is $\epsilon$-optimal if $\mathbb{E}_{\pi}[\sum_{h=1}^{H}r^{\star}(s_{h},a_{h})]\geq\max_{\pi^{\prime}}\mathbb{E}_{\pi^{\prime}}[\sum_{h=1}^{H}r^{\star}(s_{h},a_{h})]-\epsilon$. We also consider a setting where the utility is only available for a whole trajectory. In this case, we assume that there is an underlying trajectory reward function $r^{\star}:(\mathcal{S}\times\mathcal{A})^{H}\to[0,H]$, which assigns a scalar utility to each trajectory. In this case the value of a policy can be defined similarly as $\mathbb{E}_{\tau\sim\pi}[r(\tau)]$.

In preference based RL, the reward is assumed to be unobservable, but is respected by the comparison oracle which models human evaluators.

Here Ber$(p)$ denotes a Bernoulli distribution with mean $p$. The comparison outcome $o=1$ indicates $\tau_{1}\succ\tau_{2}$, and vice versa. We additionally require that the inputs $\tau_{1}$, $\tau_{2}$ to the comparison oracle are feasible in the sense that they should be actual trajectories generated by the algorithm and cannot be synthesized artificially. This is motivated by the potential difficulty of asking human evaluators to compare out-of-distribution samples (e.g. random pixels).

In this work, we also consider $\boldsymbol{(s,a)}$ preferences, where $\Pr[(s_{1},a_{1})\succ(s_{2},a_{2})]=\sigma(r(s_{1},a_{1})-r(s_{2},a_{2}))$. For notational simplicity, we will use $\tau$ to denote a state-action pair $(s,a)$ (which can be thought as an incomplete trajectory) so that $(s,a)$ preferences can be seen as a special case of the comparison oracle (Definition 1). See more details in Remark 3.1.

For general preferences, we assume that for every trajectory pair $\tau,\tau^{\prime}$, the probability that a human evaluator prefers $s$ over $s^{\prime}$ is

A general preference may not be realizable by a utility model, so we cannot define the optimal policy in the usual sense. Instead, we follow Dudík et al. (2015) and consider an alternative solution concept, the von Neumann winner (see Definition 5).

We first introduce the concept of eluder dimension, which has been widely used in RL to measure the difficulty of function approximation.

Intuitively, eluder dimension measures the number of worst-case mistakes one has to make in order to identify an unknown function from the class $\mathcal{F}$. It is often used as a sufficient condition to prove sample efficiency guarantees for optimism-based algorithms.

As in many works on function approximation, we assume knowledge a class of reward functions $\mathcal{R}$ and realizability.

We further use $\overline{\mathcal{R}}:=\{r+c|c\in[-H,0],r\in\mathcal{R}\}$ to denote the reward function class augmented with a bias term. The inclusion of an additional bias is due to the assumption that preference feedback is based on reward differences, so we could only learn $r^{\star}$ up to a constant. We note that for the $d$-dimension linear reward class $\mathcal{R}_{\rm linear}$, ${\rm dim}_{\rm E}(\overline{\mathcal{R}}_{\rm linear})\leq\tilde{O}(d)$, and that for a general reward class $\mathcal{R}$, ${\rm dim_{E}}(\overline{\mathcal{R}},\epsilon)\leq\mathcal{O}(H{\rm dim_{E}}(\mathcal{R},\epsilon/2)^{1.5}/\epsilon)$. The details can be found in Appendix B.

The interaction protocol of an RL algorithm with the Preference-to-Reward (P2R) Interface is shown in Fig. 1.

P2R maintains a confidence set of rewards $B_{r}$. When the RL algorithm wishes to learn the reward label of a trajectory $\tau$, P2R checks whether $B_{r}$ approximately agrees on the reward of $\tau$. If so, it can return a reward label with no queries to the comparison oracle; if not, it will query the comparison oracle on $\tau$ and a fixed trajectory $\tau_{0}$, and update the confidence set of reward functions. The details of P2R are presented in Algorithm 1.

The performance of P2R depends on the RL algorithm running on top of it. In particular, we define sample complexity of a standard reward-based RL algorithm $\mathscr{A}$ as follows.

We would like to note that the requirement of being $g(\epsilon)$-robust is typically not restrictive. In fact, any tabular RL algorithm with sample complexity would be $O(\epsilon/H)$-robust with the same sample complexity, while many algorithms with linear function approximation are $O(\epsilon/{\rm poly}(d,H))$-robust (Jin et al., 2020b; Zanette et al., 2020) (again with the same sample complexity). We can further show that one can effortlessly convert any standard RL algorithms with sample complexity $\mathcal{C}$ into an $O\left(1/\mathcal{C}\right)$-robust one by using the procedure described in Lemma B.3.

When combined with existing sample complexity results, Theorem 3.4 directly implies concrete sample and query complexity bounds for preference-based RL in many settings, with no statistical and small computational overhead.

While the P2R interface provides a clean and effortless recipe for modifying standard RL algorithms to work with preference feedback, it incurs a large query cost to the comparison oracle, e.g., cubic dependence on $d_{\overline{\mathcal{R}}}$ in Example 3. We believe that this disadvantage is caused by the black-box nature of P2R, and that better query complexities can be achieved by modifying standard RL algorithms in a white-box manner and specialized analysis. In this section, we take OMLE (Liu et al., 2022a) as an example and introduce a standalone adaptation to trajectory preference feedback with improved query complexity. We expect other optimism-based algorithms (e.g., UCBVI-BF and GOLF) can be modified in a similar white-box manner to achieve better query complexity.

The details of the Preference-based OMLE algorithm (P-OMLE) is provided in Algorithm 2. Compared to OMLE with reward feedback (Algorithm 3), the only difference is that now the reward confidence set is computed using preference feedback directly using log-likelihood. Denote by $V^{\pi}_{r,p}$ the expected cumulative reward the leaner will receive if she follows policy $\pi$ in model $(p,r)$. In the $t$-th iteration, the algorithm follows the following steps:

• •

  Optimistic planning: Find the policy-model pair $(\pi,r,p)$ that maximizes the value function $V^{\pi}_{r,p}$;

• •

  Data collection: Construct an exploration policy set $\Pi_{\rm exp}(\pi^{t})$ ³³3 The exploration policy set is problem-dependent and can be simply $\{\pi^{t}\}$ for many settings. and collect trajectories by running all policies in $\Pi_{\rm exp}(\pi^{t})$;

• •

  Confidence set update: Update the confidence set using the updated log-likelihood:

We have the follwoing guarantee for Algorithm 2.

For problems that satisfy $\xi=\widetilde{\mathcal{O}}(\sqrt{d_{\mathcal{P}}\beta_{\mathcal{P}}|\Pi_{\rm exp}|T})$, Theorem 3.8 implies both the sample complexity and the query complexity for learning an $\epsilon$-optimal policy is upper bounded by:

Compared to the result derived through the P2R interface (Example 3), the sample complexity here is basically the same while the query complexity has improved dependence on $d_{\overline{\mathcal{R}}}$ (from cubic to linear). Nonetheless, the query complexity here additionally depends on the complexity of learning the transition model, which is not the case in Example 3.

In this subsection, we briefly discuss how our results can be extended to $K$-wise comparison assuming the following Plackett-Luce (PL) model (Luce, 1959; Plackett, 1975) of $K$ item preferences.

Note that when $K=2$, the above PL model reduces to a pair-wise trajectory-type comparison oracle (Definition 1) with $\sigma(x)=\exp(\eta x)/(\exp(\eta x)+1)$ which satisfies Assumption 2 with $\alpha=\Theta(\eta\exp(-\eta H))$. The PL model satisfies the following useful property, which is a corollary of its internal consistency (Hunter, 2004).

This property enables “batch querying” the preferences on $\lfloor K/2\rfloor$ pairs of $(\tau_{1},\tau_{2})$ in parallel, which returns $\lfloor K/2\rfloor$ independent pairwise comparisons outcomes. This would enable us to reduce the number of queries by a factor of $\Omega(K)$ for small $K$ in both Algorithm 1 and 3.

Theorem 3.10 is a direct consequence of Theorem 3.4: If $K\geq 2m$, we can obtain $m$ independent comparisons between two trajectories by a single query to the $K$-wise comparison oracle and therefore reduce the overall query complexity in Theorem 3.4 by a factor of $m$; otherwise, we can get $m$ independent comparisons by making $\mathcal{O}(m/K)$ queries to the $K$-wise comparison oracle, which reduces the overall query complexity by a factor of $K$.

Similarly, we can combine the idea of batch querying with P-OMLE by adding $K/2$ independent comparisons between $\tau$ and $\tau^{0}$ into $\mathcal{D}_{\tt rwd}$ each time. And by slightly modifying the proof of Theorem 3.8, we obtain the following improved sample complexity and query complexity for learning an $\epsilon$-optimal policy with $K$-wise comparison oracle:

However, the above parallelization benefits of using $K$-wise comparison might be an artifact of the PL model: it seems improbable that the same human evaluator would independently rank $\lfloor K/2\rfloor$ copies of item $A$ and item $B$. It remains an interesting problem to develop $K$-wise comparison models more suitable to RLHF.

In this section, we consider a special case where the preference depends solely on the final states of the two input trajectories, i.e., $M(\tau,\tau^{\prime})=M(s_{H},s_{H}^{\prime})$. ⁴⁴4 This could be true, for instance, if the agent’s task is to clean a kitchen, and is evaluated based on the kitchen’s configuration in the end. Given the previous equivalence relation between von Neumann winner and restricted Nash in FI-MG, one natural idea is to apply no-regret learning algorithms, as it is well-known that running two copies of no-regret online learning algorithms against each other can be used to compute Nash equilibria in zero-sum normal-form games. Since this paper focuses on sequential decision making, we need no-regret learning algorithms for adversarial MDPs, which we define below.

In this section, we consider the more general case where the preference $M[\tau,\tau^{\prime}]$ is allowed to depend arbitrarily on the two input trajectories. Similar to the utility-based setting, we assume that the learner is provided with a preference class $\mathcal{M}\subseteq((\mathcal{S}\times\mathcal{A})^{H}\times(\mathcal{S}\times\mathcal{A})^{H}\rightarrow[0,1])$ and transition function class $\mathcal{P}$ a priori, which contains the groundtruth preference and transition we are interacting with. Previously, we have established the reduction from learning the von Neumann winner to learning restricted Nash in FI-MG. In addition, learning restricted Nash in FI-MG is in fact a special case of learning Nash equilibrium in partially observable Markov games (POMGs). As a result, we can directly adapt the existing OMLE algorithm for learning Nash in POMGs (Liu et al., 2022b) to our setting, with only minor modifications required to learn the von Neumann winner. We defer the algorithmic details for this approach (Algorithm 5) to Appendix F, and present only the theoretical guarantee here.

It has been proven that a wide range of RL problems admit a regret formulation of $\xi=\widetilde{\mathcal{O}}(\sqrt{d_{\mathcal{P}}\beta|\Pi_{\rm exp}|T})$ with mild $d_{\mathcal{P}}$ and $|\Pi_{\rm exp}|$ (Liu et al., 2022a). These problems include, but are not limited to, tabular MDPs, factored MDPs, linear kernel MDPs, observable POMDPs, and decodable POMDPs. For more details, please refer to Appendix C.1 or Liu et al. (2022a). For problems that satisfy $\xi=\widetilde{\mathcal{O}}(\sqrt{d_{\mathcal{P}}\beta_{\mathcal{P}}|\Pi_{\rm exp}|T})$, Theorem C.1 implies a sample complexity of

The sample complexity for specific tractable problems can be derived by plugging their precise formulation of $\xi$ (provided in Appendix C.1) into the above bound.