Avoiding Catastrophe in Online Learning by Asking for Help

There has been mounting concern over catastrophic risk from AI, including but not limited to autonomous weapon accidents (Abaimov & Martellini, 2020), bioterrorism (Mouton et al., 2024), cyberattacks on critical infrastructure (Guembe et al., 2022), and loss of control (Bengio et al., 2024). See Critch & Russell (2023) and Hendrycks et al. (2023) for taxonomies of societal-scale AI risks. In this paper, we use “catastrophe” to refer to any kind of irreparable harm. In addition to the large-scale risks above, our definition also covers smaller-scale (yet still unacceptable) incidents such as serious medical errors (Rajpurkar et al., 2022), crashing a robotic vehicle (Kohli & Chadha, 2020), and discriminatory sentencing (Villasenor & Foggo, 2020).

The gravity of these risks contrasts starkly with the dearth of theoretical understanding of how to avoid them. Nearly all of learning theory explicitly or implicitly assumes that no single mistake is too costly. We focus on online learning, where an agent repeatedly interacts with an unknown environment and uses its observations to gradually improve its performance. Most online learning algorithms essentially try all possible behaviors and see what works well. We do not want autonomous weapons or surgical robots to try all possible behaviors.

More precisely, trial-and-error-style algorithms only work when catastrophe is assumed to be impossible. This assumption can take multiple forms, such as that the agent’s actions do not affect future inputs (e.g., Slivkins, 2011), that no action has irreversible effects (e.g., Jaksch et al., 2010) or that the environment is reset at the start of each “episode” (e.g., Azar et al., 2017). One could train an agent entirely in a controlled lab setting where one of those assumptions does hold, but we argue that sufficiently general agents will inevitably encounter novel scenarios when deployed in the real world. Machine learning models often behave unpredictably in unfamiliar environments (see, e.g., Quiñonero-Candela et al., 2022), and we do not want AI biologists or robotic vehicles to behave unpredictably.

The goal of this paper is to understand the conditions under which it is possible to formally guarantee avoidance of catastrophe in online learning. Certainly some conditions are necessary, because the problem is hopeless if the agent must rely purely on trial-and-error: any untried action could lead to paradise or disaster and the agent has no way to predict which. In the real world, however, one need not learn through pure trial-and-error: one can also ask for help. We think it is critical for high-stakes AI applications to employ a designated supervisor who can be asked for help. Examples include a human doctor supervising AI doctors, a robotic vehicle with a human driver who can take over in emergencies, autonomous weapons with a human operator, and many more. We hope that our work constitutes a step in the direction of practical safety guarantees for such applications.

Learning with irreversible costs. Despite the ubiquity of irreversible costs in the real world, theoretical work on this topic remains limited. This may be due to the fundamental modeling question of how the agent should learn about novel inputs or actions without trying them directly.

The most common approach is to allow the agent to ask for help. This alone is insufficient, however: the agent must have some way to decide when to ask for help. A popular solution is to perform Bayesian inference on the world model, but this has two tricky requirements: (1) a prior distribution which contains the true world model (or an approximation), and (2) an environment where computing (or approximating) the posterior is tractable. A finite set of possible environments satisfies both conditions but is unrealistic in many real-world scenarios. In contrast, our algorithm can handle an uncountable policy class and a continuous unbounded input space, which is crucial for many real-world scenarios in which one never sees the exact same input twice.

Bayesian inference combined with asking for help is studied by Cohen et al. (2021); Cohen & Hutter (2020); Kosoy (2019); Mindermann et al. (2018). We also mention Hadfield-Menell et al. (2017); Moldovan & Abbeel (2012); Turchetta et al. (2016), who utilize Bayesian inference in the context of safe (online) reinforcement learning without asking for help (and without regret bounds).

We are only aware of two papers that theoretically address irreversibility without Bayesian inference: Grinsztajn et al. (2021) and Maillard et al. (2019). The former proposes to sample trajectories and learn reversibility based on temporal consistency between states: intuitively, if $s_{1}$ always precedes $s_{2}$, we can infer that $s_{1}$ is unreachable from $s_{2}$. Although the paper theoretically grounds this intuition, there is no formal regret guarantee. The latter presents an algorithm which asks for help in the form of rollouts from the current state. However, the regret bound and number of rollouts are both linear in the worst case, due to the dependence on the $\gamma^{*}$ parameter which roughly captures how bad an irreversible action can be. In contrast, our algorithm achieves good regret even when actions are maximally bad.

To our knowledge, we are the first to provide an algorithm which formally guarantees avoidance of catastrophe (with high probability) without Bayesian inference. We are also not aware of prior results comparable to our negative result, including in the Bayesian regime.

Safe reinforcement learning (RL). The safe RL problem is typically formulated as a constrained Markov Decision Process (CMDP) (Altman, 2021). In CMDPs, the agent must maximize reward while also satisfying safety constraints. See Gu et al. (2024); Zhao et al. (2023); Wachi et al. (2024) for surveys. The two most relevant safe RL papers are Liu et al. (2021) and Stradi et al. (2024), both of which provide algorithms guaranteed to satisfy initially unknown safety constraints. Since neither paper allows external help, they require strong assumptions to make the problem tractable: the aforementioned results assume that the agent (1) knows a strictly safe policy upfront (i.e., a policy which satisfies the safety constraints with slack), (2) is periodically reset, and (3) observes the safety costs. In contrast, our agent has no prior knowledge, is never reset, and never observes payoffs.

Online learning. See Cesa-Bianchi & Lugosi (2006) and Chapter 21 of Shalev-Shwartz & Ben-David (2014) for introductions to online learning. A classical result states that sublinear regret is possible if and only if the hypothesis class has finite Littlestone dimension (Littlestone, 1988). However, even some simple hypothesis classes have infinite Littlestone dimension, such as the class of thresholds on $[0,1]$ (Example 21.4 in Shalev-Shwartz & Ben-David, 2014). Recently, Haghtalab et al. (2024) showed that if the adversary only chooses a distribution over inputs rather than the precise input, only finite VC dimension (Vapnik & Chervonenkis, 1971) is needed for sublinear regret. Specifically, they assume that each input is sampled from a distribution whose concentration is upper bounded by $\frac{1}{\sigma}$ times the uniform distribution. This framework is known as smoothed analysis, originally due to Spielman & Teng (2004).

Multiplicative objectives. Although online learning traditionally studies the sum of payoffs, there is some work which aims to maximize the product of payoffs (or equivalently, the sum of logarithms). See, e.g., Chapter 9 of Cesa-Bianchi & Lugosi (2006). However, these regret bounds are still sublinear in $T$, in comparison to our subconstant regret bounds. Also, like most online learning work, those results assume that payoffs are observed on every time step. In contrast, our agent only receives feedback in response to queries (Table 1) and never observes payoffs. Barman et al. studied a multiplicative objective in a multi-armed bandit context, but their objective is the geometric mean of payoffs instead of the product. Interpreted in our context, their regret bounds imply that the average chance of catastrophe goes to zero, while we guarantee that the total chance of catastrophe goes to zero. This distinction is closely related to the difference between subconstant and sublinear regret.

Active learning and imitation learning. Our assumption that the agent only receives feedback in response to queries falls under the umbrella of active learning (Hanneke, 2014). This contrasts with passive learning, where the agent receives feedback automatically. The way our agent learns from the mentor is also reminiscent of imitation learning (Osa et al., 2018). Although ideas from these areas could be useful in our setting, we are not aware of any results from that literature which account for irreversible costs.

Inputs. Let $\mathbb{N}$ denote the strictly positive integers and let $T\in\mathbb{N}$ be the time horizon. Let $\boldsymbol{x}=(x_{1},x_{2},\dots,x_{T})\in\mathcal{X}^{T}$ be the sequence of inputs. In the fully adversarial setting, each $x_{t}$ can have arbitrary (possibly randomized) dependence on the events of prior time steps. In the smoothed setting, the adversary only chooses the distribution $\mathcal{D}_{t}$ from which $x_{t}$ is sampled. Formally, a distribution $\mathcal{D}$ over $\mathcal{X}$ is $\sigma$-smooth if for any $S\subseteq\mathcal{X}$, $\mathcal{D}(S)\leq\frac{1}{\sigma}U(S)$. (In the smoothed setting, we assume that $\mathcal{X}$ supports a uniform distribution $U$.⁵⁵5For example, it suffices for $\mathcal{X}$ to have finite Lebesgue measure. Note that this does not imply boundedness. Alternatively, $\sigma$-smoothness can be defined with respect to a different distribution; see Definition 1 of Block et al. (2022).) If each $x_{t}$ is sampled from a $\sigma$-smooth $\mathcal{D}_{t}$, we say that $\boldsymbol{x}$ is $\sigma$-smooth. The sequence $\boldsymbol{\mathcal{D}}=\mathcal{D}_{1},\dots,\mathcal{D}_{T}$ can still be adaptive, i.e., the choice of $\mathcal{D}_{t}$ can depend on the events of prior time steps.

Actions and payoffs. Let $\mathcal{Y}$ be a finite set of actions. There also exists a special action $\tilde{y}$ which corresponds to querying the mentor. For $k\in\mathbb{N}$, let $[k]=\{1,2,\dots,k\}$. On each time step $t\in[T]$, the agent must select action $y_{t}\in\ \mathcal{Y}\cup\{\tilde{y}\}$, which generates a payoff. Let $\boldsymbol{y}=(y_{1},\dots,y_{T})$. We allow the payoff function to vary between time steps: let $\boldsymbol{\mu}=(\mu_{1},\dots,\mu_{T})\in(\mathcal{X}\times\mathcal{Y}\to[0,1])^{T}$ be the sequence of payoff functions. Then $\mu_{t}(x_{t},y_{t})\in[0,1]$ is the agent’s payoff at time $t$. Like $\boldsymbol{\mathcal{D}}$, we allow $\boldsymbol{\mu}$ to be adaptive. Unless otherwise noted, all expectations are over any randomization in the agent’s decisions, any randomization in $\boldsymbol{x}$, and any randomization in the adaptive choice of $\boldsymbol{\mu}$.

Asking for help. The mentor is endowed with a (possibly suboptimal) policy $\pi^{m}:\mathcal{X}\to\mathcal{Y}$. When action $\tilde{y}$ is chosen, the mentor informs the agent of the action $\pi^{m}(x_{t})$ and the agent obtains payoff $\mu_{t}(x_{t},\pi^{m}(x_{t}))$. For brevity, let $\mu_{t}^{m}(x)=\mu_{t}(x,\pi^{m}(x))$. The agent never observes payoffs: the only way to learn about $\boldsymbol{\mu}$ is by querying the mentor.

We would like an algorithm which becomes “self-sufficient” over time: the rate of querying the mentor should go to 0 as $T\to\infty$, or equivalently, the cumulative number of queries should be sublinear in $T$. Formally, let $Q_{T}(\boldsymbol{\mu},\pi^{m})=\{t\in[T]:y_{t}=\tilde{y}\}$ be the random variable denoting the set of time steps with queries. Then we say that the (expected) number of queries is sublinear in $T$ if $\sup_{\boldsymbol{\mu},\pi^{m}}\operatorname*{\mathbb{E}}[|Q_{T}(\boldsymbol{\mu},\pi^{m})|]\in o(T)$. In other words, there must exist $g:\mathbb{N}\to\mathbb{N}$ which does not depend on $\boldsymbol{\mu}$ or $\pi^{m}$ such that $g(T)\in o(T)$ and $\sup_{\boldsymbol{\mu},\pi^{m}}\operatorname*{\mathbb{E}}[|Q_{T}(\boldsymbol{\mu},\pi^{m})|]\leq g(T)$. For brevity, we will usually write $Q_{T}=Q_{T}(\boldsymbol{\mu},\pi^{m})$.

Local generalization. We assume that $\boldsymbol{\mu}$ and $\pi^{m}$ satisfy local generalization. Informally, if the agent is given an input $x$, taking the mentor action for a similar input $x^{\prime}$ is almost as good as taking the mentor action for $x$. Formally, we assume $\mathcal{X}\subseteq\mathbb{R}^{n}$ and there exists $L>0$ such that for all $x,x^{\prime}\in\mathcal{X}$ and $t\in[T]$, $|\mu_{t}^{m}(x)-\mu_{t}(x,\pi^{m}(x^{\prime}))|\leq L||x-x^{\prime}||$, where $||\cdot||$ denotes Euclidean distance. This represents the ability to transfer knowledge between similar inputs:

This ability seems fundamental to intelligence and is well-understood in psychology (e.g., Esser et al., 2023) and education (e.g., Hajian, 2019). Note that the input space $\mathcal{X}\subseteq\mathbb{R}^{n}$ can be any encoding of the agent’s situation, not just its physical positioning. See Section 5.2 for further discussion.

All suprema over $\boldsymbol{\mu},\pi^{m}$ pairs are assumed to be restricted to $\boldsymbol{\mu},\pi^{m}$ pairs which satisfy local generalization.

Regret. If $\mu_{t}(x_{t},y_{t})\in[0,1]$ is the chance of avoiding catastrophe at time $t$ (conditioned on no prior catastrophe), then by the chain rule of probability, $\prod_{t=1}^{T}\mu_{t}(x_{t},y_{t})$ is the agent’s overall chance of avoiding catastrophe. For given $\boldsymbol{x},\boldsymbol{y},\boldsymbol{\mu},\pi^{m}$, the agent’s multiplicative regret⁶⁶6One could also define the multiplicative regret as $R_{T}^{\prime}=\prod_{t=1}^{T}\mu_{t}^{m}(x_{t})-\prod_{t=1}^{T}\mu_{t}(x_{t},y_{t})$, but our definition is actually stricter: $\lim_{T\to\infty}R_{T}^{\times}\to 0$ implies $\lim_{T\to\infty}R_{T}^{\prime}\to 0$, while the reverse is not true. In particular, $\lim_{T\to\infty}R_{T}^{\prime}\to 0$ is trivial if $\lim_{T\to\infty}\prod_{t=1}^{T}\mu_{t}^{m}(x_{t})\to 0$. is

when all payoffs are strictly positive. To handle the case where some payoffs are zero, we assume the existence of $\mu_{0}^{m}>0$ such that $\mu^{m}(x_{t})\geq\mu_{0}^{m}$ always. Thus only the agent’s payoffs can be zero, so we can safely define $R_{T}^{\times}(\boldsymbol{x},\boldsymbol{y},\boldsymbol{\mu},\pi^{m})=\infty$ whenever $\mu_{t}(x_{t},y_{t})=0$ for some $t\in[T]$. We write $R_{T}^{\times}=R_{T}^{\times}(\boldsymbol{x},\boldsymbol{y},\boldsymbol{\mu},\pi^{m})$ for brevity.

The assumption of $\mu^{m}(x_{t})\geq\mu_{0}^{m}>0$ means that the mentor cannot be abysmal. In fact, we argue that high-stakes AI applications should employ a mentor who is almost always safe, i.e., $\mu_{0}^{m}\approx 1$. If no such mentor exists for some application, perhaps that application should be avoided altogether.

We also define the agent’s additive regret as

and similarly write $R_{T}^{+}=R_{T}^{+}(\boldsymbol{x},\boldsymbol{y},\boldsymbol{\mu},\pi^{m})$ for brevity. For both objectives, we desire subconstant worst-case regret: the total (not average) expected regret should go to 0 for any $\boldsymbol{\mu}$ and $\pi^{m}$. Formally, we want $\lim_{T\to\infty}\sup_{\boldsymbol{\mu},\pi^{m}}\operatorname*{\mathbb{E}}[R_{T}^{\times}]=0$ and $\lim_{T\to\infty}\sup_{\boldsymbol{\mu},\pi^{m}}\operatorname*{\mathbb{E}}[R_{T}^{+}]=0$.

VC and Littlestone dimensions. VC dimension (Vapnik & Chervonenkis, 1971) and Littlestone dimension (Littlestone, 1988) are standard measures of learning difficulty which capture the ability of a hypothesis class (in our case, a policy class) to realize arbitrary combinations of labels (in our case, actions). We omit the precise definitions since we only utilize these concepts via existing results. See Shalev-Shwartz & Ben-David (2014) for a comprehensive overview.

Misc. The diameter of a set $S\subseteq\mathcal{X}$ is defined by $\operatorname*{diam}(S)=\max_{x,x^{\prime}\in S}||x-x^{\prime}||$. All logarithms and exponents are base $e$ unless otherwise noted.

We first show that in general, any algorithm with sublinear mentor queries has unbounded regret in the worst-case, even when inputs are i.i.d. on $[0,1]$ and $\boldsymbol{\mu}$ does not vary over time. The formal proofs are deferred to Appendix A, but we provide intuition and define the construction here.

Intuitively, the regret decreases as the number of queries increases. However, as long as the number of queries remains sublinear in $T$, the regret is unbounded as $T\to\infty$.

We also have the following corollary of Theorem 4.1:

In other words, even if the mentor never causes catastrophe, any algorithm with sublinear queries causes catastrophe with probability 1 as $T\to\infty$ in the worst case.

Theorem 4.1 shows that avoiding catastrophe is impossible in general. What if we restrict ourselves to settings where standard online learning is possible? Specifically, we assume that $\pi^{m}$ belongs to a policy class $\Pi$ where either (1) $\Pi$ has finite VC dimension $d$ and $\boldsymbol{x}$ is $\sigma$-smooth or (2) $\Pi$ has finite Littlestone dimension $d$.⁷⁷7Recall from Section 1.3 that standard online learning becomes tractable under either of these assumptions. This section presents a simple algorithm which guarantees subconstant regret (both multiplicative and additive) and sublinear queries under either of those assumptions. Formal proofs are deferred to Appendix B but we provide intuition and a proof sketch here.

In this paper, we proposed a model of avoiding catastrophe in online learning. We showed that achieving subconstant regret in our problem (with the help of a mentor and local generalization) is no harder than achieving sublinear regret in standard online learning.

Remaining technical questions. First, we have not resolved whether our problem is tractable for finite VC dimension and fully adversarial inputs (although Appendix D shows that the problem is tractable for at least some classes with finite VC but infinite Littlestone dimension). Second, the time complexity of Algorithm 1 currently stands at a hefty $\Omega(|\tilde{\Pi}|)$ per time step plus the time to compute $\tilde{\Pi}$. In the standard online learning setting, Block et al. (2022) and Haghtalab et al. (2022) show how to replace discretization approaches like ours with oracle-efficient approaches, where a small number of calls to an optimization oracle are made per round. We are optimistic about leveraging such techniques to obtain efficient algorithms in our setting.

Local generalization. Our algorithm crucially relies on the ability to detect when an input is unfamiliar, i.e., differs significantly from prior observations in a metric space which satisfies local generalization. Without this ability, the practicality of our algorithm would be fundamentally limited. One option is to use out-of-distribution (OOD) detection, which is conceptually similar and well-studied (see Yang et al., 2024 for a survey). However, it is an open question whether standard OOD detection methods are measuring distance in a metric space which satisfies local generalization.

We are also interested in alternatives to local generalization. Theorem E.2 shows that our positive result breaks down if local generalization is removed, so some sort of assumption is necessary. One possible alternative is Bayesian inference. We intentionally avoided Bayesian approaches in this paper due to tractability concerns, but it seems premature to abandon those ideas entirely.

MDPs. Finally, we are excited to apply the ideas in this paper to Markov Decision Processes (MDPs): specifically, MDPs where some actions are irreversible (“non-communicating”) and the agent only gets one attempt (“single-episode”). In such MDPs, the agent must not only avoid catastrophe but also obtain high reward. As discussed in Section 2, very little theory exists for RL in non-communicating single-episode MDPs. Can an agent learn near-optimal behavior in high-stakes environments while becoming self-sufficient over time? Formally, we pose the following open problem:

Is there an algorithm for non-communicating single-episode undiscounted MDPs which ensures that both the regret and the number of mentor queries are sublinear in $T$?

As AI systems become increasingly powerful, we believe that the safety guarantees of such systems should become commensurately robust. Irreversible costs are especially worrisome, and we hope that our work plays a small part in mitigating such risks. We do not believe that our work has any concrete potential risks that should be highlighted here.

B. Plaut conceived the project. B. Plaut designed the mathematical model with feedback from H. Zhu and S. Russell. B. Plaut proved all of the results. H. Zhu participated in proof brainstorming and designed counterexamples for several early conjectures. B. Plaut wrote the paper, with feedback from H. Zhu and S. Russell. S. Russell supervised the project and secured funding.

This work was supported in part by a gift from Open Philanthropy to the Center for Human-Compatible AI (CHAI) at UC Berkeley. This paper also benefited from discussions with many other researchers. We would like to especially thank (in alphabetical order) Aly Lidayan, Bhaskar Mishra, Cameron Allen, Juan Liévano-Karim, Matteo Russo, Michael Cohen, Nika Haghtalab, and Scott Emmons. We would also like to thank our anonymous reviewers for helpful feedback.