Unifying Clustered and Non-stationary Bandits

To offer a unified approach that addresses the two target problems, we formulate a general bandit learning setting that encompasses both non-stationarity in individual users and existence of clustering structure among users.

Consider a learner that interacts with a set of $n$ users, $\mathcal{U}=\left\{1,...,n\right\}$. At each time $t=1,2,...,T$, the learner receives an arbitrary user indexed by $i_{t}\in\mathcal{U}$, together with a set of available arms $C_{t}=\{\mathbf{x}_{t,1},\mathbf{x}_{t,2},\dots,\mathbf{x}_{t,K}\}$ to choose from, where $\mathbf{x}_{t,j}\in\mathbb{R}^{d}$ denotes the context vector associated with the arm indexed by $j$ at time $t$ (assume $\lVert\mathbf{x}_{t,j}\rVert\leq 1$ without loss of generality), and $K$ denotes the size of arm pool $C_{t}$. After the learner chooses an arm $\mathbf{x}_{t}$, its reward $y_{t}\in\mathbb{R}$ is fed back from the user $i_{t}$. We follow the linear reward assumption Abbasi-Yadkori et al. (2011); Chu et al. (2011); Li et al. (2010) and use $\theta_{i_{t},t}$ to denote the parameter of the reward mapping function in user $i_{t}$ at time $t$ (assume $\lVert\theta_{i_{t},t}\rVert\leq 1$). Under this assumption, reward at time $t$ is $y_{t}=\mathbf{x}_{t}^{\top}\theta_{i_{t},t}+\eta_{t}$, where $\eta_{t}$ is Gaussian noise drawn from $N(0,\sigma^{2})$. Interaction between the learner and users repeats, and the learner’s goal is to maximize the accumulated reward it receives from all users in $\mathcal{U}$ up to time $T$.

Denote the set of time steps when user $i\in\mathcal{U}$ is served up to time $T$ as $\mathcal{N}_{i}(T)=\left\{1\leq t\leq T:i_{t}=i\right\}$. Among time steps $t\in\mathcal{N}_{i}(T)$, user $i$’s parameter $\theta_{i,t}$ changes abruptly at arbitrary time steps $\{c_{i,1},...,c_{i,\Gamma_{i}(T)-1}\}$, but remain constant between any two consecutive change points. $\Gamma_{i}(T)$ denotes the total number of stationary periods in $\mathcal{N}_{i}(T)$. The set of unique parameters that $\theta_{i,t}$ takes for any user at any time is denoted as $\left\{\phi_{k}\right\}_{k=1}^{m}$ and their frequency of occurrences in $T$ is $\left\{p_{k}\right\}_{k=1}^{m}$. Note that the ground-truth linear parameters, the set of change points, the number and frequencies of unique parameters are unknown to the learner. Moreover, the number of users, i.e., $n$, and the number of unique bandit parameters across users, i.e., $m$, are finite but arbitrary.

Our problem setting defined above is general. The non-stationary and clustering structure of an environment can be specified by different associations between $\{\theta_{i,t}\}^{n}_{i=1}$ and $\left\{\phi_{k}\right\}_{k=1}^{m}$ across users over time $t=1,2,...,T$. For instance, by setting $n>m$ and $\Gamma_{i}(T)=1,\forall i\in\mathcal{U}$, the problem naturally reduces to the online clustering of bandits problem, which assumes sharing of bandit models among users with stationary reward distributions. By setting $n=1$, $m>1$ and $\Gamma_{i}(T)>1,\forall i\in\mathcal{U}$, it reduces to the piecewise stationary bandits problem, which only concerns a single user with non-stationary reward distributions in isolation.

To make our solution compatible with existing work in non-stationary bandits and clustered bandits, we also introduce the following three commonly made assumptions about the environment.

The first assumption establishes the detectability of change points in each individual user’s bandit models over time. The second assumption ensures separation within the global unique parameter set shared by all users, and the third assumption specifies the property of context vectors. Based on these assumptions, we establish the problem setup in this work and illustrate it on the left side of Figure 1.

As discussed in Section 2, the key problem in non-stationary bandits is to detect changes in the underlying reward distribution, and the key problem in online clustering of bandits is to measure the relatedness between different models. We view both problems as testing homogeneity between two sets of observations to unify these two seemingly distinct problems. For change detection, we test homogeneity between recent and past observations to evaluate whether there has been a change in the underlying bandit parameters for these two consecutive sets of observations. For cluster identification, we test homogeneity between observations of two different users to verify whether they share the same bandit parameters. On top of the test results, we can operate the bandit models accordingly for model selection, model aggregation, arm selection, and model update.

We use $\mathcal{H}_{1}=\left\{(\mathbf{x}_{i},y_{i})\right\}_{i=1}^{t_{1}}$ and $\mathcal{H}_{2}=\left\{(\mathbf{x}_{j},y_{j})\right\}_{j=1}^{t_{2}}$ to denote two sets of observations, where $t_{1},t_{2}\geq 1$ are their cardinalities. $(\mathbf{X}_{1},\mathbf{y}_{1})$ and $(\mathbf{X}_{2},\mathbf{y}_{2})$ denote design matrices and feedback vectors of $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ respectively, where each row of $\mathbf{X}$ is the context vector of a selected arm and the corresponding element in $\mathbf{y}$ is the observed reward for this arm. Under our linear reward assumption, $\forall(\mathbf{x}_{i},y_{i})\in\mathcal{H}_{1}$, $y_{i}\sim N(\mathbf{x}_{i}^{\top}\theta_{1},\sigma^{2})$, and $\forall(\mathbf{x}_{j},y_{j})\in\mathcal{H}_{2}$, $y_{j}\sim N(\mathbf{x}_{j}^{\top}\theta_{2},\sigma^{2})$. The test of homogeneity between $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ can thus be formally defined as testing whether $\theta_{1}=\theta_{2}$, i.e., whether observations in $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ come from a homogeneous population.

Because $\theta_{1}$ and $\theta_{2}$ are not observable, the test has to be performed on their estimates, for which maximum likelihood estimator (MLE) is a typical choice. Denote MLE for $\theta$ on a dataset $\mathcal{H}$ as ${\vartheta}=(\mathbf{X}^{\top}\mathbf{X})^{-}\mathbf{X}^{\top}\mathbf{y}$, where $\left(\cdot\right)^{-}$ stands for generalized matrix inverse. A straightforward approach to test homogeneity between $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ is to compare $\lVert{\vartheta}_{1}-{\vartheta}_{2}\rVert$ against the estimation confidence on ${\vartheta}_{1}$ and ${\vartheta}_{2}$. The clustering methods used in Gentile et al. (2014, 2017) essentially follow this idea. However, theoretical guarantee on the false negative probability of this method only exists when the minimum eigenvalues of $\mathbf{X}_{1}^{\top}\mathbf{X}_{1}$ and $\mathbf{X}_{2}^{\top}\mathbf{X}_{2}$ are larger than a predefined threshold. In other words, only when both $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ have sufficient observations, this test is effective.

We appeal to an alternative test statistic that has been proved to be uniformly most powerful for this type of problems Chow (1960); Cantrell et al. (1991); Wilson (1978):

where ${\vartheta}_{1,2}$ denotes the estimator using data from both $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$. The knowledge about $\sigma^{2}$ can be relaxed by replacing it with empirical estimate, which leads to Chow test that has F-distribution Chow (1960).

When $s(\mathcal{H}_{1},\mathcal{H}_{2})$ is above a chosen threshold $\upsilon$, it suggests the pooled estimator deviates considerably from the individual estimators on two datasets. Thus, we conclude $\theta_{1}\neq\theta_{2}$; otherwise, we conclude $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ are homogeneous. The choice of $\upsilon$ is critical, as it determines the type-I and type-II error probabilities of the test. Upper bounds of these two error probabilities and their proofs are given below.

Note that $s(\mathcal{H}_{1},\mathcal{H}_{2})$ falls under the category of $\chi^{2}$ test of homogeneity. Specifically, it is used to test whether the parameters of linear regression models associated with two datasets are the same, assuming equal variance. It is known that this test statistic follows the noncentral $\chi^{2}$-distribution as shown in Theorem 1 (Chow, 1960; Cantrell et al., 1991).

Then based on Theorem 1, the upper bounds for type I and type II error probabilities can be derived.

Proof of Lemma 2.

When datasets $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ are homogeneous, which means $\theta_{1}=\theta_{2}$, the non-centrality parameter becomes:

Therefore, when $\theta_{1}=\theta_{2}$, the test statistic $s(\mathcal{H}_{1},\mathcal{H}_{2})\sim\chi^{2}(df,0)$. The type-I error probability can be upper bounded by $P(s(\mathcal{H}_{1},\mathcal{H}_{2})>\upsilon|\theta_{1}=\theta_{2})\leq 1-F(\upsilon;df,0)$, which concludes the proof of Lemma 2.

Proof of Lemma 3.

Similarly, using the cumulative density function of $\chi^{2}(df,\psi)$, we can show that the type-II error probability $P\big{(}s(\mathcal{H}_{1},\mathcal{H}_{2})\big{)}\leq\upsilon|\theta_{1}\neq\theta_{2}\big{)}\leq F(\upsilon;df,\psi)$. As mentioned in Section 3.2, the value of $\psi$ depends on the unknown parameters $\theta_{1}$ and $\theta_{2}$. From the definition of $\psi$, we know that $\theta_{1}=\theta_{2}$ is only a sufficient condition for $\psi=0$. The necessary and sufficient condition for $\psi=0$ is that $\left[\begin{matrix}\mathbf{X}_{1}\theta_{1}\\ \mathbf{X}_{2}\theta_{2}\end{matrix}\right]$ is in the column space of $\left[\begin{matrix}\mathbf{X}_{1}\\ \mathbf{X}_{2}\end{matrix}\right]$, e.g., there exists $\theta$ such that $\left[\begin{matrix}\mathbf{X}_{1}\theta_{1}\\ \mathbf{X}_{2}\theta_{2}\end{matrix}\right]=\left[\begin{matrix}\mathbf{X}_{1}\theta\\ \mathbf{X}_{2}\theta\end{matrix}\right]$. Only when both $\mathbf{X}_{1}$ and $\mathbf{X}_{2}$ have a full column rank, $\theta_{1}=\theta_{2}$ becomes the necessary and sufficient condition for $\psi=0$. This means when either $\mathbf{X}_{1}$ or $\mathbf{X}_{2}$ is rank-deficient, there always exists $\theta_{1}$ and $\theta_{2}$, and $\theta_{1}\neq\theta_{2}$, that make $\psi=0$. For example, assuming $\mathbf{X}_{1}$ is rank-sufficient and $\mathbf{X}_{2}$ is rank-deficient, then $\psi=0$ as long as $\theta_{1}-\theta_{2}$ is in the null space of $\mathbf{X}_{2}$.

To obtain a non-trivial upper bound of the type-II error probability, or equivalently a non-zero lower bound of the non-centrality parameter $\psi$, both $\mathbf{X}_{1}$ and $\mathbf{X}_{2}$ need to be rank-sufficient. Under this assumption, we can rewrite $\psi$ in the following way to derive its lower bound.

Denote $\epsilon=\theta_{2}-\theta_{1}$. Then $\theta_{2}=\theta_{1}+\epsilon$. We can decompose $\sigma^{2}\psi$ as:

Since $\left[\begin{matrix}\mathbf{X}_{1}\theta_{1}\\ \mathbf{X}_{2}\theta_{1}\end{matrix}\right]$ is in the column space of $\left[\begin{matrix}\mathbf{X}_{1}\\ \mathbf{X}_{2}\end{matrix}\right]$, the first term in the above result is zero. The second and third terms can be shown equal to zero as well using the property that matrix product is distributive w.r.t. matrix addition, which leaves us only the last term. Therefore, by substituting $\epsilon=\theta_{2}-\theta_{1}$ back, we obtain:

The first inequality uses the Rayleigh-Ritz theorem, and the second inequality uses the relation $\mathbf{Y}(\mathbf{X}+\mathbf{Y})^{-1}\mathbf{X}=(\mathbf{X}^{-1}+\mathbf{Y}^{-1})^{-1}$, where $\mathbf{X}$ and $\mathbf{Y}$ are both invertible matrices. This relation can be derived by taking inverse on both sides of the equation $\mathbf{X}^{-1}(\mathbf{X}+\mathbf{Y})\mathbf{Y}^{-1}=\mathbf{X}^{-1}\mathbf{X}\mathbf{Y}^{-1}+\mathbf{X}^{-1}\mathbf{Y}\mathbf{Y}^{-1}=\mathbf{Y}^{-1}+\mathbf{X}^{-1}$.

The results above show that given two datasets $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$, the type-I error probability of the homogeneity test only depends on the selection of threshold $\upsilon$, while the type-II error probability also depends on the ground-truth parameters ($\theta_{1}$, $\theta_{2}$) and variance of noise $\sigma^{2}$. If either $\mathbf{X}_{1}$ or $\mathbf{X}_{2}$ is rank-deficient, the type-II error probability will be trivially upper bounded by $F(\upsilon;df,0)$, which means for a smaller upper bound of type-I error probability (i.e., $1-F(\upsilon;df,0)$), the upper bound of type-II error probability (i.e., $F(\upsilon;df,0)$) will be large. Intuitively, for a certain level of type-I error, to ensure a smaller type-II error probability in the worst case, we at least need both $\mathbf{X}_{1}$ and $\mathbf{X}_{2}$ to be rank-sufficient and the value of $\frac{||\theta_{1}-\theta_{2}||^{2}/\sigma^{2}}{\frac{1}{\lambda_{\min}(\mathbf{X}_{1}^{\top}\mathbf{X}_{1})}+\frac{1}{\lambda_{\min}(\mathbf{X}_{2}^{\top}\mathbf{X}_{2})}}$ to be large. Similar idea is also found in Wu et al. (2018); Gentile et al. (2014, 2017), where they require the confidence bounds of the estimators (which correspond to the minimum eigenvalue of the correlation matrix in our case) to be small enough, w.r.t. $||\theta_{1}-\theta_{2}||^{2}$, to ensure their change detection or cluster identification is accurate. Here we unify the analysis of these two tasks with this homogeneity test.

These error probabilities are the key concerns in our problem: in change detection, they correspond to the early and late detection of change points Wu et al. (2018); and in cluster identification, they correspond to missing a user model in the neighborhood and placing a wrong user model in the neighborhood Gentile et al. (2014). Given it is impossible to completely eliminate these two types of errors in a non-deterministic algorithm, the uniformly most powerful property of the test defined in Eq (1) guarantees its sensitivity is optimal at any level of specificity.

In the environment specified in Section 3.1, the user’s reward mapping function is piecewise stationary (e.g., the line segments on each user’s interaction trace in Figure 1), which calls for the learner to actively detect changes and re-initialize the estimator to avoid distortion from outdated observations Yu and Mannor (2009); Cao et al. (2019); Besson and Kaufmann (2019); Wu et al. (2018). A limitation of these methods is that they do not attempt to reuse outdated observations because they implicitly assume each stationary period has an unique parameter. Our setting relaxes this by allowing for existence of identical reward mappings across users and time (e.g., the orange line segments in Figure 1), which urges the learner to take advantage of this situation by identifying and aggregating observations with the same parameter to obtain a more accurate reward estimation.

Since neither the change points nor the grouping structure is known, in order to reuse past observations while avoiding distortion, the learner needs to accurately detect change points, stores observations in the interval between two consecutive detection together, and then correctly identify intervals with the same parameter as the current one. In this paper, we propose to unify these two operations using the test in Section 3.2, which leads to the algorithm Dynamic Clustering of Bandits, or DyClu in short. DyClu forms a two-level hierarchy as shown in Figure 1: at the lower level, it stores observations in each interval and their sufficient statistics in a user model; at the upper level, it detects change in user’s reward function to decide when to create new user models and clusters individual user models for arm selection. Detailed steps of DyClu are explained in Algorithm 1.

The lower level of DyClu manages observations associated with each user $i\in\mathcal{U}$ in user models, denoted by $\mathbb{M}_{i,t}$. Each user model $\mathbb{M}_{i,t}=(\mathbf{A}_{i,t},\mathbf{b}_{i,t},\mathcal{H}_{i,t})$ stores:

1. 1.

   $\mathcal{H}_{i,t}$: a set of observations associated with user $i$ since the initialization of $\mathbb{M}_{i,t}$ up to time $t$, where each element is a context vector and reward pair $(\mathbf{x}_{k},y_{k})$.

2. 2.

   Sufficient statistics: $\mathbf{A}_{i,t}=\sum_{(\mathbf{x}_{k},\cdot)\in\mathcal{H}_{i,t}}\mathbf{x}_{k}\mathbf{x}_{k}^{\top}$ and $\mathbf{b}_{i,t}=\sum_{(\mathbf{x}_{k},y_{k})\in\mathcal{H}_{i,t}}\mathbf{x}_{k}y_{k}$.

Every time DyClu detects change in a user’s reward mapping function, a new user model is created to replace the previous one (line 15 in Algorithm 1). We refer to the replaced user models as outdated models and the others up-to-date ones. We denote the set of all outdated user models at time $t$ as $\mathbb{O}_{t}$ and the up-to-date ones as $\mathbb{U}_{t}$. In Figure 1, the row of circles next to $\mathbb{M}_{1,t-1}$ represents all the user models for user $1$, red ones denote outdated models and the blue one denotes up-to-date model.

The upper level of DyClu is responsible for managing the user models via change detection and model clustering. It replaces outdated models in each user and aggregates models across users and time for arm selection.

At any time step $t$, DyClu only updates $\mathbb{M}_{i_{t},t-1}$ when $e_{i_{t},t}=0$ (line 10-12 in Algorithm 1). This guarantees that if the underlying reward distribution has changed, with a high probability we have $e_{i_{t},t}=1$, and thus the user model $\mathbb{M}_{i_{t},t-1}$ will not be updated. This prevents any distortion in $\mathcal{H}_{i_{t},t}$ by observations from different reward distributions.

When $\hat{e}_{i_{t},t}$ exceeds the threshold specified by Lemma 4, DyClu will inform the lower level to move $\mathbb{M}_{i_{t},t-1}$ to the outdated model set $\mathbb{O}_{t}=\mathbb{O}_{t-1}\cup\left\{\mathbb{M}_{i_{t},t-1}\right\}$; and then create a new model $\mathbb{M}_{i_{t},t}=\left(A_{i_{t},t}=\textbf{0},b_{i_{t},t}=\textbf{0},\mathcal{H}_{i,t}=\emptyset\right)$ for user $i_{t}$ as shown in line 13-16 in Algorithm 1.

When selecting an arm for user $i_{t}$ at time $t$, DyClu aggregates the sufficient statistics of user models in neighborhood $\hat{V}_{i_{t},t-1}$. Then it adopts the popular UCB strategy Auer (2002); Li et al. (2010) to balance exploitation and exploration. Specifically, DyClu selects arm $\mathbf{x}_{t}$ that maximizes the UCB score computed by aggregated sufficient statistics as follows (line 5 in Algorithm 1),

In Eq (2), $\hat{\theta}_{\hat{V}_{i_{t},t-1}}=\mathbf{A}_{\hat{V}_{i_{t},t-1}}^{-1}\mathbf{b}_{\hat{V}_{i_{t},t-1}}$ is the ridge regression estimator using aggregated statistics $\mathbf{A}_{\hat{V}_{i_{t},t-1}}=\lambda\mathbf{I}_{d}+\sum_{(\mathbf{A}_{j},\mathbf{b}_{j},\mathcal{H}_{j})\in\hat{V}_{i_{t},t-1}}\mathbf{A}_{j}$ and $\mathbf{b}_{\hat{V}_{i_{t},t-1}}=\sum_{(\mathbf{A}_{j},\mathbf{b}_{j},\mathcal{H}_{j})\in\hat{V}_{i_{t},t-1}}\mathbf{b}_{j}$; the confidence bound of reward estimation for arm $\mathbf{x}$ is $CB_{\hat{V}_{i_{t},t-1}}(\mathbf{x})=\alpha_{\hat{V}_{i_{t},t-1}}\sqrt{\mathbf{x}^{\top}\mathbf{A}_{\hat{V}_{i_{t},t-1}}^{-1}\mathbf{x}}$, where $\alpha_{\hat{V}_{i_{t},t-1}}=\sigma\sqrt{d\log{(1+\frac{\sum_{(A_{j},b_{j},\mathcal{H}_{j})\in\hat{V}_{i_{t},t-1}}|\mathcal{H}_{j}|}{d\lambda})}+2\log{\frac{1}{\delta}}}+\sqrt{\lambda}$.

Denote $R_{T}=\sum_{t=1}^{T}\theta_{i_{t}}^{\top}\mathbf{x}_{t}^{*}-\theta_{i_{t}}^{\top}\mathbf{x}_{t}$ as the accumulative regret, where $\mathbf{x}_{t}^{*}=\operatorname*{arg\,max}_{x_{t,j}\in C_{t}}\theta_{i_{t}}^{\top}\mathbf{x}_{t,j}$ is the optimal arm at time $t$. Our regret analysis relies on the high probability results in Abbasi-Yadkori et al. (2011) and decomposition of "good" and "bad" events according to change detection and clustering results. Specifically, the "good" event corresponds to the case where the aggregated statistics in Eq 2 contains and only contains all the past observations associated with the ground truth unique bandit parameter at current time step. The "bad" event is simply the complement of "good" event that occurs as a result of errors in change detection and clustering, and we can analyze their occurrences based on our results in Lemma 2 and Lemma 3. Following this idea, the upper regret bound of DyClu given in Theorem 5 can be derived. The full proof, along with ancillary results, are provided in the appendix.

This upper regret bound depends both on the intrinsic clustering and non-stationary property of user set $\mathcal{U}$. Since the proposed DyClu algorithm addresses a bandit problem that generalizes environment settings in several existing studies, here we discuss how DyClu compares with these algorithms when reduced to their corresponding environment settings.

Case 1: Setting $m=1$, $n=1$ and $\Gamma_{1}(T)=1$ reduces the problem to the basic linear bandit setting, because the environment consists of only one user with a stationary reward distribution for the entire time of interaction. With only one user who has a stationary reward distribution, we have $\sum_{k=1}^{1}\sqrt{p_{k}}=1$ where $p_{k}$ is frequency of occurrences of $\phi_{k}$ in $T$ as defined in Section 3.1. In addition, since there is only one stationary period, the added regret caused by late detection does not exist; and the added regret due to the failure in clustering can be bounded by a constant as later shown in Lemma 7, which only depends on environment variables. The upper regret bound of DyClu then becomes $O\big{(}\sigma d\sqrt{T\log^{2}{T}}\big{)}$, which achieves the same order of regret as that in LinUCB Abbasi-Yadkori et al. (2011).

Case 2: Setting $\Gamma_{i}(T)=1,\forall i\in\mathcal{U}$ reduces the problem to the online clustering of bandit setting Gentile et al. (2014), because all users in the environment have a stationary reward distribution of their own. Similar to Case 1, the added regret caused by late detection becomes zero and the added regret due to the failure in clustering is bounded by a constant, which leads to the upper regret bound of $O\big{(}\sigma d\sqrt{T\log^{2}{T}}(\sum_{k=1}^{m}\sqrt{p_{k}}\big{)}$. DyClu achieves the same order of regret as that in CLUB Gentile et al. (2014).

Case 3: Setting $n=1$ reduces the problem to a piecewise stationary bandit setting, because the environment consists of only one user with piecewise stationary reward distributions. For the convenience of comparison, we can rewrite the upper regret bound of DyClu in the form of $O\big{(}\sum_{k\in[m]}R_{Lin}(|N^{\phi}_{k}(T)|)+\Gamma_{1}(T)\big{)}$, where $R_{Lin}(t)=O\Big{(}d\sqrt{t\log^{2}{t}}\Big{)}$ Abbasi-Yadkori et al. (2011) and $N^{\phi}_{k}(T)=\left\{1\leq t^{{}^{\prime}}\leq T:\theta_{i_{t^{{}^{\prime}}},t^{{}^{\prime}}}=\phi_{k}\right\}$ is the set of time steps up to time $T$ when the user being served has the bandit parameter equal to $\phi_{k}$. Detailed derivation of this is given in Section C.1. Note that the upper regret bound of dLinUCB Wu et al. (2018) for this setting is $O\big{(}\Gamma_{1}(T)R_{Lin}(S_{max})+\Gamma_{1}(T)\big{)}$, where $S_{max}$ denotes the maximum length of stationary periods. The regret of DyClu depends on the number of unique bandit parameters in the environment, instead of the number of stationary periods as in dLinUCB, because DyClu can reuse observations from previous stationary periods. This suggests DyClu has a lower regret bound if different stationary periods share the same unique bandit parameters; for example, in situations where a future reward mapping function switches back to a previous one.