Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detection

Reinforcement learning (RL) techniques have been successfully applied to solve high-dimensional sequential decision problems. However, if the state transition and/or reward functions change unexpectedly, according to latent factors unobservable to the agent, the system is effectively tasked with optimizing behavior policies that maximize performance over a (possibly infinite) random sequence of Markov Decision Processes (MDPs). Each MDP is drawn from an unknown distribution and is henceforth referred to as a context. This is known as a non-stationary setting. Designing efficient algorithms to tackle this problem is a known challenge in RL Padakandla (2020). The key difficulties here result from (i) the need to quickly and reliably detect when the underlying system dynamics has changed; and (ii) the need to effectively learn and deploy adaptable prediction models and policies, specialized in particular contexts, while allowing the agent to (when appropriate) reuse previously-acquired knowledge. Non-stationary environments often result from systems whose dynamics are inherently time-dependent; from agents that are tasked with learning policies under noisy or missing sensors; or from problems where the agent faces sequences of unlabeled/unidentified tasks, each one with its own transition dynamics and reward functions.

Non-stationary settings arise naturally in many situations. Humans, for instance, are capable of learning to solve sequences of tasks from few experiences while preserving knowledge from older experiences Lake et al. (2015). Consider, for example, a person realizing the need to adapt their gait after an accident, learning novel gait patterns to use when walking with crutches, and then, after a period of recovery, successfully re-deploying normal walking gaits. This corresponds to a non-stationary scenario where the agent needs to learn specialized dynamics models and policies for tackling different contexts/learning scenarios.

We introduce an algorithm that efficiently learns decision-making strategies in this setting. It assumes that an agent experiences random sequences of contexts (MDPs) drawn from some unknown distribution, and it is capable of optimizing behaviors even when a pre-training phase (during which the agent interacts with sample contexts) is not available. The agent’s goal is to rapidly deploy policies that are appropriate for each randomly-arriving context—even when the number of latent contexts is unknown and when the context distribution cannot be modeled nor controlled by the agent. We are particularly interested in the case where quick readaptation and sample efficiency are paramount to achieving good performance; for instance, in novel contexts where collecting experiences and acquiring policies from scratch is unfeasible.

Many existing related works tackle non-stationary problems either by detecting when the underlying MDP changes, or via meta-learning approaches that construct a prior model (or policy) capable of rapidly generalizing to novel contexts. Hadoux et al., for example, introduced a technique based on change-point detection algorithms to deal with non-stationary problems with discrete state spaces Hadoux et al. (2014); Banerjee et al. (2017). We, by contrast, address the more general setting of high-dimensional continuous RL problems. Supervised meta-learning algorithms Finn et al. (2017) have also been recently combined with RL to enable fast adaptation under changing domains Nagabandi et al. (2019a); Nagabandi et al. (2019b). Nagabandi et al. introduced a model-based algorithm where a meta-learning technique is used to construct dynamics models capable of rapidly adapting to recent experiences—either by updating the hidden state of a recurrent neural network Duan et al. (2016), or by updating the model parameters via a small number of gradient steps Finn et al. (2017). Meta-learning methods typically assume disjoint training and testing phases, so that an agent can be pre-trained over randomly sampled contexts prior to its deployment. We, however, do not require a pre-training phase. Meta-learning methods also typically assume that the distribution over contexts experienced during training is the same as the one experienced during testing, so that agents can adapt to novel environments with structural similarities to those previously experienced. We, by contrast, do not require that contexts are sampled from a previously-seen distribution, nor that contexts share structural similarities with previously-experienced ones.

To address these limitations, we introduce an algorithm that analyzes a possibly infinite stream of data and computes, in real-time, high-confidence change-point detection statistics that reflect whether novel, specialized policies need to be deployed to tackle new contexts, or whether a previously-optimized policy may be reused. We call our algorithm Model-Based RL Context Detection, or MBCD. We formally show that it minimizes the delay until unforeseen changes to a context are detected, thereby allowing for rapid responses, and that it allows for formal bounds on the rate of false alarm—which is of interest when minimizing the agent’s regret over random sequences of contexts. Our method constructs a mixture model composed of a (possibly infinite) ensemble of probabilistic dynamics predictors that model the different modes of the distribution over underlying latent MDPs. Our method is capable of optimizing policies based on streams of arbitrarily different contexts, with unknown boundaries, and which may be drawn from an arbitrary, unknown distribution.

We evaluate our algorithm on high-dimensional continuous reinforcement learning tasks and empirically show (i) that state-of-the-art reinforcement learning algorithms struggle to deal with non-stationarity; and (ii) that our method outperforms state-of-the-art meta-learning methods specifically tailored to deal with non-stationary environments—in particular, when the agent is faced with MDPs that are off-distribution with respect to the set of training contexts provided to the agent, or when novel contexts are structurally different from previously-observed ones.

We define a non-stationary environment as a family of MDPs $\{\mathcal{M}_{z}\}_{z\in\mathbb{N}^{+}}$. Each MDP $\mathcal{M}_{z}$ is a tuple $(\mathcal{S},\mathcal{A},\mathcal{T}_{z},\mathcal{R}_{z},\gamma,d^{0})$, where $\mathcal{S}$ is a (possibly continuous) state space, $\mathcal{A}$ is a (possibly continuous) action space, $\mathcal{T}_{z}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ is a transition function specifying the distribution over next states, given the current state and action, $\mathcal{R}_{z}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is a reward function, $\gamma\in[0,1]$ is the discount factor, and $d^{0}$ is an initial state distribution. In what follows, $S_{t}$, $A_{t}$, and $R_{t}$ are the random variables corresponding to the state, action, and reward at time step $t$. We assume that the agent observes a random sequence $(\mathcal{M}_{0},\mathcal{M}_{1},\ldots)$ of MDPs—called contexts—drawn independently from some unknown distribution. We assume that the number of contexts, $|\{\mathcal{M}_{z}\}|$, is unknown. These definitions are similar to those discussed in Banerjee et al. (2017) and Padakandla (2020). Let $z$ be a latent index variable indicating a particular MDP, $\mathcal{M}_{z}$. We assume that each MDP’s transition and reward function are parameterized by a latent vector $\theta_{z}$. Let $p_{\theta_{z}}(S_{t+1},R_{t}|S_{t},A_{t})$ denote the joint conditional probability distribution over next-state and reward associated with MDP $\mathcal{M}_{z}$. We do not impose any smoothness assumptions on how variations to $\theta_{z}$ affect $\mathcal{T}_{z}$ and $\mathcal{R}_{z}$: contexts may be arbitrarily different and share no structural similarities.

Let the time steps in which context changes occur be an increasing sequence of integer random variables, $\{C_{i}\}_{i\geq 1}$, for which a prior $\phi(C_{i})$ is unknown or cannot be defined. We call each $C_{i}$ a change-point. At every change-point $C_{i}$, the current context $\mathcal{M}_{z}$ is replaced by a new randomly drawn MDP. To perform well, an agent must rapidly detect context changes and deploy an appropriate policy. If a new random context differs significantly from previously-experienced ones, the agent may have to learn a policy from scratch; otherwise, it may choose to reuse previously-acquired knowledge to accelerate learning and avoid catastrophic forgetting.

At each time $t$, when interacting with its environment, the agent selects an action $A_{t}$ based on its state $S_{t}$ according to a stochastic policy $\pi(\cdot|S_{t})$. Let $v^{H}_{\pi,\mathcal{M}_{z}}$ be the value function associated with policy $\pi$, MDP $\mathcal{M}_{z}$, and defined over a horizon $H$:

By contrast, consider an alternative policy $\Pi$ that follows policy $\pi_{1}$ only after a random time $\Gamma$, for $\Gamma>C$; that is, a policy that deploys the correct decision-making strategy for $\mathcal{M}_{1}$ with a delay of $\Delta=\Gamma-C$ steps. Its value function is given by:

Notice that we can rewrite Eq. 2 in the same form as Eq. 3:

Change-point detection (CPD) algorithms Aminikhanghahi and Cook (2016); Veeravalli and Banerjee (2012) are designed to detect whether (and when) a change occurs in the distribution generating random observations from an arbitrary stochastic process. These methods have been widely used in a variety of fields—from financial markets Lam and Yam (1997) to biomedical signal processing Sibanda and Sibanda (2007). Although CPDs have been applied to reinforcement learning problems Banerjee et al. (2017); Hadoux et al. (2014), the application and formal analysis of such methods have been restricted to discrete state spaces settings.

In the online CPD setting, a sequential detection procedure is defined to rapidly and reliably estimate when the parameter $\theta$ of some underlying distribution or stochastic process has changed. Online CPD algorithms should produce high-confidence estimates, $\Gamma$, of the true change-point time, $C$. Notice that $\Gamma$ is a random variable whose stochasticity results from the unknown stochastic prior over context changes, $\phi$, and from the fact that each MDP in $\{\mathcal{M}_{z}\}$ produces random trajectories of states, actions, and rewards.

Suppose that at each time $t$, while the agent interacts with $\mathcal{M}_{0}$, sample next-state and rewards are drawn from $p_{\theta_{0}}(S_{t+1},R_{t}|S_{t},A_{t})$, where $\theta_{0}$ is the latent vector parameterizing $\mathcal{M}_{0}$’s transition and reward functions. At some unknown random change-point $C$, the context changes to $\mathcal{M}_{1}$, and experiences that follow are drawn from $p_{\theta_{1}}(S_{t+1},R_{t}|S_{t},A_{t})$. We propose to identify such a change by computing high-confidence statistics that reflect whether $\theta_{0}$ has changed. This can be achieved by introducing a minimax formulation of the CPD problem, as discussed by Pollak Pollak (1985). In this formulation, the goal of a CPD algorithm is to compute a random estimate, $\Gamma$, of the latent change-point time $C$, such that (i) it minimizes the worst-case expected detection delay, $\Delta_{worst}(\Gamma)$, associated with the random estimates $\Gamma$ produced by a particular CPD algorithm, when considering all possible change-points $C$; and (ii) bounds on the maximum false alarm rate (FAR) may be imposed. The worst-case expected detection delay, $\Delta_{worst}(\Gamma)$, and the FAR, are defined as:

When $\theta_{0}$ and $\theta_{1}$ are known, the Log-Likelihood Ratio (LLR) statistic can be used to recursively compute the CUSUM statistic Page (1954). As we will discuss next, such a statistic can be used to construct a high-confidence change-point detection method. The LLR statistic, $L_{t}$, and the CUSUM statistic, $W_{t}$, are updated at each time $t$ as follows:

Importantly, notice that before the change-point $C$ is reached, $\mathbb{E}[L_{t}]<0$, which implies that the expected value of $W_{t}$ is zero. After the change-point $C$ is reached, $\mathbb{E}[L_{t}]>0$, and therefore $W_{t}$ will tend to increase. Higher values of $W_{t}$, thus, serve as principled statistics reflecting evidence that a change-point has occurred between $\theta_{0}$ to $\theta_{1}$. The random time $\Gamma$ when a change-point is estimated to have happened is defined as the first time when the CUSUM metric $W_{t}$ becomes greater than a detection threshold $h$:

In Lorden (1971), Lorden shows that choosing $h=|\log\alpha|$ ensures that $\operatorname{FAR}(\Gamma)\leq\alpha$. Furthermore, Lai demonstrated that the CUSUM detection time $\Gamma$ is asymptotically optimum Lai (1998) with respect to the problem specified in Eq. 6. In particular, they showed that the worst expected detection delay (under $h=|\log\alpha|$) respects the following approximation:

In Eq. 12, the denominator $\mathcal{D}_{KL}(p_{\theta_{1}}||p_{\theta_{0}})=\mathbb{E}_{\theta_{1}}\left[\log\frac{p_{\theta_{1}}(S_{t+1},R_{t}|S_{t},A_{t})}{p_{\theta_{0}}(S_{t+1},R_{t}|S_{t},A_{t})}\right]$ is the Kullback–Leibler divergence under $\theta_{1}$. Eq. 12 implies that the larger the difference between the distributions $p_{\theta_{1}}$ and $p_{\theta_{0}}$, the smaller the expected delay ($\Delta=\Gamma-C$) for detecting a change-point. The above results allow us to construct high-confidence statistics reflecting whether (and when) a context has changed; importantly, they are both accurate and have bounded false alarm rate.

In this section, we introduce an algorithm that iteratively applies a CUSUM-related procedure to detect context changes under the assumptions discussed in Section 2. The algorithm incrementally builds a library of models and policies for tackling arbitrarily different types of contexts; i.e., contexts that may result from quantitatively and qualitatively different underlying causes for non-stationarity—ranging from unpredictable environmental changes (such as random wind) to robot malfunctions. Our method can rapidly deploy previously-constructed policies whenever contexts approximately re-occur, or learn new decision-making strategies whenever novel contexts, with no structural similarities with respect to previously-observed ones, are first encountered. Unlike existing approaches (see Section 6), our method is capable of (i) optimizing policies online; (ii) without requiring a pre-training phase; (iii) based on streams of arbitrarily different contexts, with unknown change-point boundaries; and (iv) such that contexts may be drawn from an arbitrary, unknown distribution.

We now introduce a high-level description of our method (Model-Based RL Context Detection, or MBCD). In subsequent subsections, we provide details for each of the method’s components. As the agent interacts with a non-stationary environment, context changes are identified via a multivariate variant of CUSUM Healy (1987), called MCUSUM. MCUSUM-based statistics inherit the same formal properties as those presented in Section 3. In particular, they formally guarantee that MBCD can detect context changes with minimum expected delay, while simultaneously bounding the false alarm rate. As a consequence, MBCD can effectively identify novel environmental dynamics while ensuring, with high probability, that new context-specific policies will only be constructed when necessary.

As new contexts are identified by this procedure, MBCD updates a mixture model, $M$, composed of a (possibly infinite) ensemble of probabilistic context dynamics predictors, whose purpose is to model the different modes of the distribution over underlying latent MDPs/contexts. New models are added to the ensemble as qualitatively different contexts are first encountered. The mixture model $M$ associates, with each identified context $\mathcal{M}_{z}$, a learned joint distribution $p_{\theta_{z}}$ over next-state dynamics and rewards associated. Let $K$ be the number of context models currently in the mixture. After each agent experience, MBCD identifies the most likely context, $z_{t}$, by analyzing a set of incrementally-estimated MCUSUM statistics (see Section 4.2). Whenever a novel context—one with dynamics that are qualitatively different from those previously-experienced—is observed, a new model is added to the mixture. Context-specific policies, $\pi_{\psi_{z}}$, are trained via a Dyna-style approach Sutton (1990) based on the corresponding learned joint prediction model of $\mathcal{M}_{z}$, $p_{\theta_{z}}$ (see Section 4.3). We provide details in the next subsections. Pseudocode for MBCD is shown in Algorithm 1.

We evaluate MBCD in challenging continuous-state, continuous-action non-stationary environments, where the non-stationarity may result from qualitatively different reasons—ranging from abrupt changes to the system’s dynamics (such as changes to the configuration of the agent’s workspace); external latent environmental factors that impact the distribution over next states (such as random wind); robot malfunctions; and changes to the agent’s goals (its reward function). We compare MBCD both against state-of-the-art RL algorithms and against state-of-the-art meta-learning methods specifically tailored to deal with non-stationary environments.

We investigate the performance of MBCD in two settings: (i) a setting that satisfies all standard requirements made by meta-learning algorithms; in particular, that all contexts in ${M_{z}}$ are structurally similar, and that the agent is tested on a distribution of contexts that matches the training distribution; and (ii) a more general setting where such requirements are not be satisfied: contexts may differ arbitrarily, and future contexts experienced by the agent may be off-distribution with respect to those sampled during the training phase. We show that our method outperforms both standard state-of-the-art RL algorithms and also specialized meta-learning algorithms in both settings.

In what follows, we evaluate MBCD in two domains with qualitatively different non-stationary characteristics (see Fig. 1):

Dealing with non-stationary environments in RL via context change-point detection has been studied in discrete state and action spaces settings. da Silva et al. Silva et al. (2006) introduced Reinforcement Learning Context Detection (RLCD). RLCD is a model-based algorithm that estimates the prediction quality of different models, and instantiates new ones when none of the existing models performs well. Even though it does not require a pre-training phase (like meta-learning algorithms), it is only applicable to purely discrete settings. Hadoux et al. introduced an extension of RLCD that uses a CUSUM-based method to perform change-point detection Hadoux et al. (2014). Unlike our method, however, it is only applicable to discrete state settings. Banerjee et al. proposed a two-threshold switching policy based on KL divergence between transition models in order to rapidly detect context changes Banerjee et al. (2017). This is a principled method but—unlike MBCD—requires prior knowledge of the dynamics model of all contexts.

In Nagabandi et al. (2019a) and Nagabandi et al. (2019b), meta-learning algorithms are used to train a prior over dynamics models that can, when combined with recent data, be rapidly adapted to novel contexts. These methods, unlike MBCD, were designed to tackle settings where the non-stationarity solely results from changes to the dynamics, but not to the agent’s goals/reward function. Reward functions are assumed to be known a priori. These methods also require an explicit pre-training train phase, prior to deployment, and assume that the distribution of training and testing contexts is the same. Our method, by contrast, is better suited to continual online settings where a pre-training phase is not possible, and where the agent is tasked with dealing with streams of arbitrarily different contexts with unknown boundaries.

We introduced a model-based reinforcement learning algorithm (MBCD) that learns efficiently in non-stationary settings with continuous states and actions. It makes use of high-confidence change-point detection statistics to detect context changes with minimum delay, while bounding the rate of false alarm. It is capable of optimizing policies online, without requiring a pre-training phase, even when faced with streams of arbitrarily different contexts drawn from unknown distributions. We empirically show that it outperforms state-of-the-art (model-free and model-based) RL algorithms, and that it outperforms state-of-the-art meta-learning methods specially designed to deal with non-stationarity—in particular, if the agent is faced with MDPs that are off-distribution with respect to the set of training contexts. As future work, we would like to extend our method so that it can actively transfer knowledge of learned policies between contexts. This research direction suggests that our method may be combined with meta-learning techniques that operate over policies, instead of over dynamics models.

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001. Ana Bazzan was partially supported by CNPq under grant no. 307215/2017-2.