Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient

Learning by interacting with an environment, in the standard online reinforcement learning (RL) protocol, has led to impressive results across a number of domains. State-of-the-art RL algorithms are quite general, employing function approximation to scale to complex environments with minimal domain expertise and inductive bias. However, online RL agents are also notoriously sample inefficient, often requiring billions of environment interactions to achieve suitable performance. This issue is particularly salient when the environment requires sophisticated exploration and a high quality reset distribution is unavailable to help overcome the exploration challenge. As a consequence, the practical success of online RL and related policy gradient/improvement methods has been largely restricted to settings where a high quality simulator is available.

To overcome the issue of sample inefficiency, attention has turned to the offline RL setting [Levine et al., 2020], where, rather than interacting with the environment, the agent trains on a large dataset of experience collected in some other manner (e.g., by a system running in production or an expert). While these methods still require a large dataset, they mitigate the sample complexity concerns of online RL, since the dataset can be collected without compromising system performance. However, offline RL methods can suffer from distribution shift, where the state distribution induced by the learned policy differs significantly from the offline distribution [Wang et al., 2021]. Existing provable approaches for addressing distribution shift are computationally intractable, while empirical approaches rely on heuristics that can be sensitive to the domain and offline dataset (as we will see).

In this paper, we focus on a hybrid reinforcement learning setting, which we call Hybrid RL, that draws on the favorable properties of both offline and online settings. In Hybrid RL, the agent has both an offline dataset and the ability to interact with the environment, as in the traditional online RL setting. The offline dataset helps address the exploration challenge, allowing us to greatly reduce the number of interactions required. Simultaneously, we can identify and correct distribution shift issues via online interaction. Variants of the setting have been studied in a number of empirical works [Rajeswaran et al., 2017, Hester et al., 2018, Nair et al., 2018, 2020, Vecerik et al., 2017] which mainly focus on using expert demonstrations as offline data.

Hybrid RL is closely related to the reset setting, where the agent can interact with the environment starting from a “nice” distribution. A number of simple and effective algorithms, including CPI [Kakade and Langford, 2002], PSDP [Bagnell et al., 2003], and policy gradient methods [Kakade, 2001, Agarwal et al., 2020b] – methods which have inspired recent powerful heuristic RL methods such as TRPO [Schulman et al., 2015] and PPO [Schulman et al., 2017]— are provably efficient in the reset setting. Yet, leveraging a reset distribution is a strong requirement (often tantamount to having access to a detailed simulation) and unlikely to be available in real world applications. Hybrid RL differs from the reset setting in that (a) we have an offline dataset, but (b) our online interactions must come from traces that start at the initial state distribution of the environment, and the initial state distribution is not assumed to have any nice properties. Both features (offline data and a nice reset distribution) facilitate algorithm design by de-emphasizing the exploration challenge. However, Hybrid RL is more practical since an offline dataset is much easier to access, while a nice reset distribution or even generative model is generally not guaranteed in practice.

We showcase the Hybrid RL setting with a new algorithm, Hybrid Q learning or Hy-Q (pronounced: Haiku). The algorithm is a simple adaptation of the classical fitted Q-iteration algorithm (FQI) and it accommodates value-based function approximation in a relatively general setup. ⁶⁶6We use Q-learning and Q-iteration interchangeably, although they are not strictly speaking the same algorithm. Our theoretical results analyze Q-iteration, but we use an algorithm with an online/mini-batch flavor that is closer to Q-learning for our experiments. For our theoretical results, we prove that Hy-Q is both statistically and computationally efficient assuming that: (1) the offline distribution covers some high quality policy, (2) the MDP has low bilinear rank, (3) the function approximator is Bellman complete, and (4) we have a least squares regression oracle. The first three assumptions are standard statistical assumptions in the RL literature while the fourth is a widely used computational abstraction for supervised learning. No computationally efficient algorithms are known under these assumptions in pure offline or pure online settings, which highlights the advantages of the hybrid setting.

We also implement Hy-Q and evaluate it on two challenging RL benchmarks: a rich observation combination lock [Misra et al., 2020] and Montezuma’s Revenge from the Arcade Learning Environment [Bellemare et al., 2013]. Starting with an offline dataset that contains some transitions from a high quality policy, our approach outperforms: an online RL baseline with theoretical guarantees, an online deep RL baseline tuned for Montezuma’s Revenge, a pure offline RL baseline, an imitation learning baseline, and an existing hybrid method. Compared to the online methods, Hy-Q requires only a small fraction of the online experience, demonstrating its sample efficiency (e.g., Figure 1). Compared to the offline and hybrid methods, Hy-Q performs most favorably when the offline dataset also contains many interactions from low quality policies, demonstrating its robustness. These results reveal the significant benefits that can be realized by combining offline and online data.

We discuss related works from four categories: pure online RL, online RL with access to a reset distribution, offline RL, and prior work in hybrid settings. We note that pure online RL refers to the setting where one can only reset the system to initial state distribution of the environment, which is not assumed to provide any form of coverage.

We consider finite horizon Markov Decision Process $M({\mathcal{S}},\mathcal{A},H,R,P,d_{0})$, where ${\mathcal{S}}$ is the state space, $\mathcal{A}$ is the action space, $H$ denotes the horizon, stochastic rewards $R(s,a)\in\Delta([0,1])$ and $P(s,a)\in\Delta({\mathcal{S}})$ are the reward and transition distributions at $(s,a)$, and $d_{0}\in\Delta({\mathcal{S}})$ is the initial distribution. We assume the agent can only reset from $d_{0}$ (at the beginning of each episode). Since the optimal policy is non-stationary in this setting, we define a policy $\pi:=\{\pi_{0},\dots,\pi_{H-1}\}$ where $\pi_{h}:{\mathcal{S}}\mapsto\Delta(\mathcal{A})$. Given $\pi$, $d^{\pi}_{h}\in\Delta({\mathcal{S}}\times\mathcal{A})$ denotes the state-action occupancy induced by $\pi$ at step $h$. Given $\pi$, we define the state and state-action value functions in the usual manner: $V^{\pi}_{h}(s)=\mathbb{E}[\sum_{\tau=h}^{H-1}r_{\tau}|\pi,s_{h}=s]$ and $Q^{\pi}_{h}(s,a)=\mathbb{E}[\sum_{\tau=h}^{H-1}r_{\tau}|\pi,s_{h}=s,a_{h}=a]$. $Q^{\star}$ and $V^{\star}$ denote the optimal value functions. We denote $V^{\pi}=\mathbb{E}_{s_{0}\sim d_{0}}V^{\pi}_{0}(s_{0})$ as the expected total reward of $\pi$. We define the Bellman operator $\mathcal{T}$ such that for any $f:{\mathcal{S}}\times\mathcal{A}\mapsto\mathbb{R}$,

We assume that for each $h$ we have an offline dataset of $m_{\mathrm{off}}$ samples $(s,a,r,s^{\prime})$ drawn iid via $(s,a)\sim\nu_{h},r\in R(s,a),s^{\prime}\sim P(s,a)$. Here $\nu=\{\nu_{0},\dots,\nu_{H-1}\}$ denote the corresponding offline data distributions. For a dataset $\mathcal{D}$, we use $\hat{\mathbb{E}}_{\mathcal{D}}[\cdot]$ to denote a sample average over this dataset. For our theoretical results, we will assume that $\nu$ covers some high-quality policy. Note that covering a high quality policy does not mean that $\nu$ itself is a distribution of some high quality policy. For instance, $\nu$ could be a mixture distribution of some high quality policy and a few low-quality policies, in which case, treating $\nu$ as expert demonstration will fail completely (as we will show in our experiments). We consider the value-based function approximation setting, where we are given a function class $\mathcal{F}=\mathcal{F}_{0}\times\dots\times\mathcal{F}_{H-1}$ with $\mathcal{F}_{h}\subset{\mathcal{S}}\times\mathcal{A}\mapsto[0,V_{\mathrm{max}}]$ that we use to approximate the value functions for the underlying MDP. Here $V_{\mathrm{max}}$ denotes the maximum total reward of a trajectory. For ease of notation, we define $f=\{f_{0},\ldots,f_{H-1}\}$ and define $\pi^{f}$ to be the greedy policy w.r.t., $f$, which chooses actions as $\pi_{h}^{f}(s)=\mathop{\mathrm{argmax}}_{a}f_{h}(s,a)$.

In this section, we present our algorithm Hybrid Q Learning – Hy-Q in Algorithm 1. Hy-Q takes an offline dataset $\{\mathcal{D}^{\nu}\}_{h=1}^{H}$ that contains $(s,a,r,s^{\prime})$ tuples and a Q function class $\mathcal{F}_{h}\subset{\mathcal{S}}\times\mathcal{A}\mapsto[0,H],h=1,\dots,H,$ as inputs, and outputs a policy that optimizes the given reward function. The algorithm is conceptually simple: it iteratively executes the FQI procedure (line 6) using the offline dataset and on-policy samples generated by the learned policies.

Specifically, at iteration $t$ and timestep $h$, we have an estimate $f_{h}^{t}$ of the $Q_{h}^{\star}$ function and we set $\pi_{h}^{t}$ to be the greedy policy for $f_{h}^{t}$. We execute $\pi_{h}^{t}$ to collect a dataset $\mathcal{D}_{h}^{t}$ of online samples in line 4. More formally, we sample $s_{h}\sim d^{\pi^{t}}_{h},a_{h}\sim\pi_{h}^{t}(\cdot|s_{h}),s_{h+1}\sim P(\cdot|s_{h},a_{h})$ and add the tuple $(s_{h},a_{h},r_{h},s_{h+1})$ to $\mathcal{D}_{h}^{t}$. Then we run FQI, the dynamic programming style algorithm on both the offline dataset $\mathcal{D}^{\nu}_{h}$ and all previously collected online samples $\{\mathcal{D}_{h}^{\tau}\}_{\tau=1}^{t}$. The FQI update works backward from time step $H$ to $0$ and computes $f_{h}^{t+1}$ via least squares regression with input $(s,a)$ and regression target $r+\max_{a^{\prime}}f_{h+1}^{t+1}(s^{\prime},a^{\prime})$.

Let us make several remarks, here we drop timestep $h$ for generality. Intuitively, the FQI updates in Hy-Q try to ensure that the estimate $f^{t}$ has small Bellman error under both the offline distribution $\nu$ and the online distributions $d^{\pi^{t}}$. The standard offline version of FQI ensures the former, but this alone is insufficient when the offline dataset has poor coverage. Indeed FQI may have poor performance in such cases [see examples in Zhan et al., 2022, Chen and Jiang, 2022]. The key insight in Hy-Q is to use online interaction to ensure that we also have small Bellman error on $d^{\pi^{t}}$. As we will see, the moment we find an $f^{t}$ that has small Bellman error on the offline distribution $\nu$ and its own greedy policy’s distribution $d^{\pi^{t}}$, FQI guarantees that $\pi^{t}$ will be at least as good as any policy covered by $\nu$. This observation results in an explore-or-terminate phenomenon: either $f^{t}$ has small Bellman error on its distribution and we are done, or $d^{\pi^{t}}$ must be significantly different from distributions we have seen previously and we make progress. Crucially, no explicit exploration is required for this argument, which is precisely how we avoid the computational difficulties with implementing optimism.

Another important point pertains to catastrophic forgetting. We will see that the size of the offline dataset $m_{\mathrm{off}}$ should be comparable to the total amount of online data $\{\mathcal{D}_{h}^{\tau}\}_{\tau=1}^{T}$, so that the two terms in Eq. 1 have similar weight and we ensure low Bellman error on $\nu$ throughout the learning process. In practice, we implement this by having all model updates use a fixed (significant) number of offline samples even as we collect more online data, so that we do not “forget” the distribution $\nu$. This is quite different from warm-starting with $\mathcal{D}^{\nu}$ and then switching to online RL, which may result in catastrophic forgetting due to a vanishing proportion of offline samples being used for model training as we collect more online samples. We note that this balancing scheme is analogous to and inspired by the one used by Ross and Bagnell [2012] in the context of model-based RL with a reset distribution. Previously, similar techniques have also been explored for various applications (for example, see Appendix F.3 of Kalashnikov et al. [2018]). As in  Ross and Bagnell [2012], a key practical insight from our analysis is that the offline data should be used throughout training to avoid catastrophic forgetting.

In this section we present the main theoretical guarantees for Hy-Q. We start by stating the main assumptions and definitions for the function approximator, the offline data distribution, and the MDP. We state the key definitions and then provide some discussion.

The transfer coefficient definition above is somewhat non-standard, but is actually weaker than related notions used in prior offline RL results. First, the average Bellman error appearing in the numerator is weaker than the squared Bellman error notion of [Xie et al., 2021a]; a simple calculation shows that $C_{\pi}^{2}$ is upper bounded by their coefficient. Second, by using Bellman errors, both of these are bounded by notions involving density ratios [Kakade and Langford, 2002, Munos and Szepesvári, 2008, Chen and Jiang, 2019]. Third, when function $f\in\mathcal{F}$ is linear in some known feature which is the case for models such as linear MDPs, the above transfer coefficient can be refined to relative condition number defined using the features. Finally, many works, particularly those that do not employ pessimism [Munos and Szepesvári, 2008, Chen and Jiang, 2019], require “all-policy” analogs, which places a much stronger requirement on the offline data distribution $\nu$. In contrast, we will only ask that $C_{\pi}$ is small for some high-quality policy that we hope to compete with. In Appendix 12, we showcase that our transfer coefficient is weaker than any related notions used in prior works under various settings such as tabular MDPs, linear MDPs, low-rank MDPs, and MDPs with general value function approximation.

All concepts defined above are frequently used in the statistical analysis of RL methods with function approximation. Realizability is the most basic function approximation assumption, but is known to be insufficient for offline RL [Foster et al., 2021] unless other strong assumptions hold [Xie and Jiang, 2021, Zhan et al., 2022, Chen and Jiang, 2022]. Completeness is the most standard strengthening of realizability that is used routinely in both online [Jin et al., 2021a] and offline RL [Munos and Szepesvári, 2008, Chen and Jiang, 2019] and is known to hold in several settings including the linear MDP and the linear quadratic regulator. These assumptions ensure that the dynamic programming updates of FQI are stable in the presence of function approximation.

Lastly, the bilinear model was developed in a series of works [Jiang et al., 2017, Jin et al., 2021a, Du et al., 2021] on sample efficient online RL.⁷⁷7Jin et al. [2021a] consider the Bellman Eluder dimension, which is related but distinct from the Bilinear model. However, our proofs can be easily translated to this setting; see Appendix C for more details. The setting is known to capture a wide class of models including linear MDPs, linear Bellman complete models, low-rank MDPs, Linear Quadratic Regulators, reactive POMDPs, and more. As a technical note, the main paper focuses on the “Q-type” version of the bilinear model, but the algorithm and proofs easily extend to the “V-type” version. See Appendix A.3 for details.

The parameters setup in the above theorem indicates that ratio between the total offline samples and the total online sample is each FQI iteration is at least $1$. This ensures that during learning, we never forget the offline distribution. A standard online-to-batch conversion [Shalev-Shwartz and Ben-David, 2014] immediately gives the following sample complexity guarantee for Algorithm 1 for finding an $\epsilon$-suboptimal policy w.r.t. the optimal policy $\pi^{*}$ for the underlying MDP.

The results formalize the statistical properties of Hy-Q. In terms of sample complexity, a somewhat unique feature of the hybrid setting is that both transfer coefficient and bilinear rank parameters are relevant, whereas these (or related) parameters typically appear in isolation in offline and online RL respectively. In terms of coverage, Theorem 1 highlights an “oracle property” of Hy-Q: it competes with any policy that is sufficiently covered by the offline dataset.

We also highlight the computational efficiency of Hy-Q: it only requires solving least squares problems over the function class $\mathcal{F}$. To our knowledge, no purely online or purely offline methods are known to be efficient in this sense, except under much stronger “uniform” coverage conditions.

In this section we discuss empirical results comparing Hy-Q to several representative RL methods on two challenging benchmarks. Our experiments focus on answering the following questions:

1. 1.

   Can Hy-Q efficiently solve problems that SOTA offline RL methods simply cannot?

2. 2.

   Can Hy-Q, via the use of offline data, significantly improve the sample efficiency of online RL?

3. 3.

   Does Hy-Q scale to challenging deep-RL benchmarks?

Our empirical results provide positive answers to all of these questions. To study the first two, we consider the diabolical combination lock environment [Misra et al., 2020, Zhang et al., 2022c], a synthetic environment designed to be particularly challenging for online exploration. The synthetic nature allows us to carefully control the offline data distribution to modulate the difficulty of the setup and also to compare with a provably efficient baseline [Zhang et al., 2022c]. To study the third question, we consider the Montezuma’s Revenge benchmark from the Arcade Learning environment, which is one of the most challenging empirical benchmarks with high-dimensional image inputs, largely due to the difficulties of exploration. Additional details are deferred to Appendix E.

We demonstrate the potential of hybrid RL with Hy-Q, a simple, theoretically principled, and empirically effective algorithm. Our theoretical results showcase how Hy-Q circumvents the computational issues of pure offline or online RL, while our empirical results highlight its robustness and sample efficiency. Yet, Hy-Q is perhaps the most natural hybrid algorithm, and we are optimistic that there is much more potential to unlock from the hybrid setting. We look forward to studying this in the future.

AS thanks Karthik Sridharan for useful discussions. WS acknowledges funding support from NSF IIS-2154711. We thank Simon Zhai for their careful reading of the manuscript and improvement on the technical correctness of our paper. We also thank Uri Sherman for their discussion on the computational efficiency of the original draft.