Best Arm Identification in Restless Markov Multi–Armed Bandits

The continued evolution of the unselected arms in our work makes it necessary for the decision entity to keep a record of (a) the time elapsed since each arm was previously selected (the arm’s delay), and (b) the state of each arm recorded at its previous selection instant (the arm’s last observed state). The notion of arm delays is superfluous when the arms are rested because the unobserved arms remain frozen. They are also redundant when the arms yield i.i.d. observations because the observation from an arm at any given time is independent of all its previous observations. When the arms are restless, the arm delays are non-negative, integer-valued, and introduce a countably-infinite dimension to the problem. In a related problem of identifying an anomalous or odd arm in a restless multi-armed bandit, Karthik and Sundaresan [6, 7] demonstrated that the arm delays and the last observed states constitute a controlled Markov process. A key aspect of the works [6, 7] is the notion of a trembling hand for selecting the arms that is defined in the system model. Probabilistically, a trembling hand with parameter $\gamma\in[0,1]$ selects an arm uniformly at random with probability $\gamma$, and selects the intended arm with probability $1-\gamma$. As the authors note in [6], when $\gamma>0$, the controlled Markov process (of arm delays and last observed states) satisfies a key ergodicity property (see [6, Lemma 1]) which is pivotal in establishing matching upper and lower bounds on the expected time to find the odd arm. An understanding of whether the lower bound for the case $\gamma=0$ admits a matching upper bound remains open.

The trembling hand model of [6, 7] implies that at any given time, the probability of selecting any arm is at least $\gamma/K$, irrespective of the arm selection scheme used. In this work, we forgo the trembling hand assumption and allow for arm selection schemes to put zero mass on certain arms. We derive a lower bound on the expected time to find the best arm over all arm selection schemes. However, such a generic lower bound may not be achievable. In order to analyse the achievability of the lower bound, we restrict attention to those arm selection policies which, for an input parameter $R>K$, forcefully select an arm if its delay is equal to $R$. For this class of policies, we derive an upper bound in terms of $R$ and show that the sequence of upper bounds, one for each value of $R$, is monotonically non-increasing as $R$ increases. An advantage of our method over the methods of [6, 7] is that while the arm selection schemes in [6, 7] are not practically implementable, the arm selection schemes we propose in this paper are easy to implement.

The question of whether, in general, the limiting value of the upper bounds matches the generic lower bound appears to be a difficult problem and remains open. Notwithstanding this, we identify some special cases when the limiting value of the upper bounds matches with the generic lower bound.

The problem of minimising regret for multi-armed bandits was introduced in the seminal work of Lai and Robbins [8] when each arm yields i.i.d. observations. Anantharam et al. [9] extended the results of Lai and Robbins to the setting of rested arms. We refer the reader to [10] for an extensive survey of regret minimization problems. In [11, 3], the authors derived a lower bound on the sample complexity of best arm identification in multi-armed bandits with i.i.d. observations from arms. Many follow-up works proposed algorithms to achieve the lower bounds; notable among them are action-elimination algorithms [12, 13], upper confidence bound (UCB) algorithms [14, 15], and lower and upper confidence bound (LUCB) algorithms [16, 17]. Several extensions to the classical setup of best-arm identification in multi-armed bandits have appeared in the literature; notable among them are correlated bandits [18], cascading bandits [19], bandits with switching costs [20], and bandits with corrupted rewards [21].

For the problem of finding the best arm as quickly as possible subject to an upper bound on the error probability, the paper [3] provided a problem instance-dependent lower bound on the expected time required to find the best arm for the setting of i.i.d. observations from the arms, and a track-and-stop scheme that meets the lower bound asymptotically as the error probability vanishes. Moulos [4] extended the results of [3] to the setting of rested arms with hidden Markov observations. However, the lower and the upper bounds in [4] differ by a constant multiplicative factor; the achievability analysis therein uses novel concentration inequalities for Markov processes derived by the author. For a related problem of finding the anomalous (or odd) arm in multi-armed bandits, the works [22, 6, 7] obtain matching upper and lower bounds on the expected time required to find the odd arm subject to an upper bound on the error probability. While [22] studies the setting of rested arms, the works [6, 7] focus on the setting of restless arms. A key aspect of the works [6, 7] is a certain trembling hand model for selecting the arms, motivated from a certain visual science experiment. The paper [6] assumes that the TPMs of the odd arm and the non-odd arms are known beforehand to the decision entity, whereas the works [22, 7] deal with the case when the arm TPMs are unknown.

The recent works [23, 24, 25] study a more general problem of sequential hypothesis testing in multi-armed bandits, some special cases of which are the problems of best arm identification and odd arm identification, in the context of i.i.d. observations from each arm. An extension of the results of [23, 24, 25] to the settings of rested and restless arms is a possible direction of future work.

In this paper, we study the problem of finding the best arm in a restless multi-armed bandit as quickly as possible, subject to an upper bound on the error probability. Our technical contributions are as follows:

• –

  In Section V, we show that given any $\epsilon>0$, the expected time required to find the best arm with an error probability no more than $\epsilon$ grows as $O(\log(1/\epsilon))$ in the limit as $\epsilon\downarrow 0$. We explicitly characterise the problem instance-dependent constant multiplying $\log(1/\epsilon)$ that is a function of the arm TPMs.

• –

  For a problem instance $C$, the constant $T^{\star}(C)$ appearing in our lower bound is the solution of a sup-min optimisation problem, where the supremum is over all possible state-action occupancy measures associated with the countable-state controlled Markov process of arm delays and last observed states; see (21) for details. The question of whether the supremum in the expression for $T^{\star}(C)$ lower bound is attained is still open. The key difficulty in showing this is the presence of the countably-infinite valued arm delays appearing in the expression for $T^{\star}(C)$, which makes further simplifications of the expression for $T^{\star}(C)$ difficult. In the prior works [3, 5, 4], further simplification of the expression for the constants governing the lower bound is possible because the notion of arm delays is superfluous in the settings of those works.

  In a related problem of odd arm identification in restless multi-armed bandits, Karthik and Sundaresan [6, 7] show that under a trembling hand model for arms selection, the supremum in the expression for the lower bounds in these works may be further simplified by restricting the supremum to the class of all stationary arm selection policies; see [6, 7] for more details. However, it is unclear if such a simplification is possible in the absence of the trembling hand (as is the setting of this paper).

• –

  In our achievability analysis presented in Section VI, to ameliorate the difficulty arising from the countably infinite-valued arm delays, we constrain the maximum delay of each arm to be no more than $R$, and focus our attention on those policies which select an arm forcibly if its delay equals $R$. This constraint on the maximum delay of each arm makes the state-action space finite and amenable to further analyses. To the best of our knowledge, this is the first work to analyse delay-constrained policies for restless multi-armed bandits.

  It is worth noting that the achievability analyses in the works [6, 7] rely crucially on a key ergodicity property (see [6, Lemma 1]) for the controlled Markov process of arm delays and last observed states that is satisfied under a trembling hand model for arms selection. This ergodicity property is pivotal to the achievability analysis in [6, 7], and it is unclear if the same property holds in the absence of the trembling hand.

• –

  We devise a policy that, for an input parameter $R$, forcibly pulls an arm if its delay equals $R$, and achieves an upper bound of the order $\Theta(\log(1/\epsilon))$. We show that the best (smallest) constant multiplying $\log(1/\epsilon)$ is equal to $1/T_{R}^{\star}(C)$ under the problem instance $C$; see (27) for the exact expression of $T_{R}^{\star}(C)$. Our results imply that $1/T_{R}^{\star}(C)$ is a valid asymptotic upper bound on the expected stopping time for all integers $R$.

• –

  We show that $T_{R}^{\star}(C)$ is non-decreasing in $R$, and that $T_{R}^{*}(C)\leq T^{\star}(C)$ for all $R$, thus implying that $\lim_{R\to\infty}\ T_{R}^{\star}(C)$ exists. Thus, the lower bound of our work is given by $T^{\star}(C)^{-1}$, whereas the upper bound is governed by $\lim_{R\to\infty}T_{R}^{\star}(C)^{-1}$. While it is certainly true that $\lim_{R\to\infty}\ T_{R}^{\star}(C)\leq T^{\star}(C)$, showing that, in general, this inequality is an equality seems to be a difficult problem and remains open. In the special case when the TPM of each arm has identical rows, which is akin to obtaining i.i.d. observations from the arms, the above inequality is an equality, thus leading to matching upper and lower bounds in this special setting. We are thus able to recover the results of the prior works [5, 3] for the case when the i.i.d. observations from each arm come from a finite alphabet.

The rest of this paper is organised as follows. In Section II, we setup the notations and state our central goal. In Section III, we introduce the notions of arm delays, last observed states, and the Markov decision problem arising from the arm delays and the last observed states. We provide expressions for the log-likelihoods and the log-likelihood ratios, the basic quantities of analysis, in Section IV, and present the asymptotic lower bound on the growth rate of the expected time required to find the best arm in Section V. In Section VI, we constrain the maximum delay of each arm to be no more than $R$ for some $R\in\mathbb{N}\cap(K,\infty)$, and describe a policy that forcibly samples an arm whose delay equals $R$. We provide results on the performance of the policy in Section VII, and state the main result in Section VIII. We discuss the convergence of $T_{R}^{\star}(C)$ to $T^{\star}(C)$ as $R\to\infty$ in Section IX, where we show that the convergence takes place in the special case when the TPM of each arm has identical rows. We conclude the paper in Section X. The proofs of all the results are contained in Appendices A-I.

A policy $\pi$ is defined by a collection of functions $\{\pi_{t}:t\geq 0\}$. At each time $t$, $\pi_{t}$ does one of the following based on the history $\mathcal{F}_{t}$:

• •

  stop and declare the index of the best arm;

• •

  choose to pull arm $A_{t}$ according to a deterministic or a randomised rule.

Let $\pi$ denote a generic policy, and let $\tau(\pi)$ denote the stopping time of policy $\pi$ (defined with respect to the filtration (4)). Let $\theta(\tau(\pi))$ denote the index of the best arm declared by the policy $\pi$ at the stopping time.

Let $P_{C}^{\pi}(\cdot)$ denote the probability computed under the assignment of the TPMs $C$ and under the policy $\pi$. For $a\in\mathcal{A}$, let $\mathcal{C}_{a}\subset\mathcal{C}$ denote the collection of all permutations in which $P_{a}=P_{1}$. Clearly, the collection $\{\mathcal{C}_{a}:\ a\in\mathcal{A}\}$ is a partition of $\mathcal{C}$. Given an error probability threshold $\epsilon>0$, let

$$\Pi(\epsilon)\coloneqq\{\pi:\ \text{for all }a\in\mathcal{A},\ \ P_{C}^{\pi}(\theta(\tau(\pi))\neq a)\leq\epsilon\ \forall C\in\mathcal{C}_{a}\}$$

(5)

denote the collection of all policies whose error probability at the stopping time is no more than $\epsilon$ for all possible assignments of TPMs. We anticipate from similar results in the prior works [3, 5, 4] that

$$\inf_{\pi\in\Pi(\epsilon)}\mathbb{E}_{C}^{\pi}[\tau(\pi)]=\Theta(\log(1/\epsilon)).$$

Here, $\mathbb{E}_{C}^{\pi}[\cdot]$ denotes the expectation under the assignment of the TPMs $C$ and policy $\pi$. Our interest is in characterising, or at least bounding, the value of

$$\lim_{\epsilon\downarrow 0}\inf_{\pi\in\Pi(\epsilon)}\ \frac{\mathbb{E}_{C}^{\pi}[\tau(\pi)]}{\log(1/\epsilon)}.$$

(6)

For simplicity, we assume that every policy starts by selecting arm $1$ at time $t=0$, arm $2$ at time $t=1$, etc., and arm $K$ at time $t=K-1$. This ensures that the Markov process of each arm is observed at least once.

Notice that $T^{\star}(C)$ is the optimal value of an infinite-dimensional linear program (LP). The question of whether there exists $\nu$ attaining the supremum in (21) remains open. The key difficulty in showing this is that because the set $\mathbb{S}$ is countably infinite, it is not clear whether the set of all $\nu$ satisfying (22)-(23) (which is akin to the space of all probability distributions on $\mathbb{S}\times\mathcal{A}$) is compact. Also, because this supremum is over all probability distributions on $\mathbb{S}\times\mathcal{A}$ which is a large class of distributions, the lower bound in (20) may not be achievable. It seems necessary to introduce additional constraints on $\nu$ to render the lower bound achievable. Indeed, given $\delta>0$, suppose $\nu_{\delta}$ is a probability distribution on $\mathbb{S}\times\mathcal{A}$ such that

$$\min_{C^{\prime}\in\textsf{Alt}(C)}\ \sum_{(\underline{d},\underline{i})\in\mathbb{S}}\ \sum_{a=1}^{K}\ \nu_{\delta}(\underline{d},\underline{i},a)\ D_{\textsf{KL}}((P_{C}^{a})^{d_{a}}(\cdot|i_{a})\|(P_{C^{\prime}}^{a})^{d_{a}}(\cdot|i_{a}))\geq T^{\star}(C)-\delta.$$

One way to achieve the quantity on the left hand side of the above equation is to ensure that for all $(\underline{d},\underline{i},a)\in\mathbb{S}\times\mathcal{A}$, the value of the fraction $N(n,\underline{d},\underline{i},a)/n$ is close to $\nu_{\delta}(\underline{d},\underline{i},a)$ for all $n$ large (the regime of large $n$ is akin to the regime of vanishing error probabilities). It seems difficult to accomplish this in the absence of more structure on $\nu_{\delta}$. It is worth noting here that in a related problem of odd arm identification, the authors of [6] are confronted with a similar difficulty in showing the achievability of the lower bound therein. To ameliorate the difficulty, they introduce a version of the following flow constraint on $\nu$:

$$\textsf{flow constraint}:\quad\sum_{a=1}^{K}\nu(\underline{d}^{\prime},\underline{i}^{\prime},a)=\sum_{(\underline{d},\underline{i})\in\mathbb{S}}\ \sum_{a=1}^{K}\ \nu(\underline{d},\underline{i},a)\ Q_{C}(\underline{d}^{\prime},\underline{i}^{\prime}|\underline{d},\underline{i},a)\quad\text{for all }(\underline{d}^{\prime},\underline{i}^{\prime})\in\mathbb{S}.$$

(24)

In (24), $Q_{C}$ is the MDP transition matrix defined in (9). The flow constraint is, in fact, a global balance equation, and dictates that for any $(\underline{d}^{\prime},\underline{i}^{\prime})\in\mathbb{S}$, the long-term probability of a transition from $(\underline{d}^{\prime},\underline{i}^{\prime})$ (the flow out of $(\underline{d}^{\prime},\underline{i}^{\prime})$, captured by the left hand side of (24)) is equal to the probability of a transition to $(\underline{d}^{\prime},\underline{i}^{\prime})$ (the flow into $(\underline{d}^{\prime},\underline{i}^{\prime})$, captured by the right hand side of (24)). The authors of [6] show that their lower bound, after including the flow constraint, can be achieved by a certain trembling hand-based policy; see [6, Section V] for a description of the policy. However, the policy in [6] is not practically implementable.

With an end goal of showing achievability of our lower bound, we take (24) into consideration along with (22)-(23) when evaluating the supremum in (21), and wish to design a policy that (a) is computationally feasible/tractable and easy-to-implement, and (b) achieves the lower bound in (20). This forms the content of the next section.

Recall that $\{(\underline{d}(t),\underline{i}(t)):t\geq K\}$ and $\{A_{t}:t\geq 0\}$ together define a Markov decision problem (MDP) whose state space is $\mathbb{S}$, the action space is $\mathcal{A}$, the state at time $t$ is $(\underline{d}(t),\underline{i}(t))$, and the control (or action) at time $t$ is $A_{t}$. From Section III, we know that the transition probabilities of the MDP are given by (9). When the delay of each arm is constrained to be no more than $R$, the modified state space of the MDP is $\mathbb{S}_{R}$, and the modified transition probabilities for the MDP are as follows:

• •

  Case 1: $(\underline{d},\underline{i})\notin\bigcup_{a=1}^{K}\ \mathbb{S}_{R,a}$. In this case, the transition probabilities are as in (9).

• •

  Case 2: $(\underline{d},\underline{i})\in\mathbb{S}_{R,a}$ for some $a\in\mathcal{A}$. In this case, when $A_{t}=a$,

  	$\displaystyle P_{C}^{\pi}(\underline{d}(t+1)=\underline{d}^{\prime},\underline{i}(t+1)=\underline{i}^{\prime}\mid\underline{d}(t)=\underline{d},\underline{i}(t)=\underline{i},A_{t}=a)$
  	$\displaystyle=\begin{cases}(P_{C}^{a})^{R}(i_{a}^{\prime}|i_{a}),&\text{if }d_{a}^{\prime}=1\text{ and }d^{\prime}_{\tilde{a}}=d_{\tilde{a}}+1\text{ for all }\tilde{a}\neq a,\\ &i_{\tilde{a}}^{\prime}=i_{\tilde{a}}\text{ for all }\tilde{a}\neq a,\\ 0,&\text{otherwise},\end{cases}$		(25)

  and when $A_{t}\neq a$, the transition probabilities are undefined.

We write $Q_{C,R}(\underline{d}^{\prime},\underline{i}^{\prime}|\underline{d},\underline{i},a)$ to denote the transition probabilities in (25).

Recall that in the absence of any constraints on the maximum delay of each arm, the lower bound is as in (20), with the constant $T^{\star}(C)$ in (20) as given in (21). Further, the supremum in (21) is over all $\nu$ satisfying (22)-(24). When the delay of each arm is constrained to be no more than $R$, the following additional constraint on $\nu$ comes into play:

$$R\textsf{-max-delay-constraint}:\quad\nu(\underline{d},\underline{i},a)=\sum_{a^{\prime}=1}^{K}\ \nu(\underline{d},\underline{i},a^{\prime})\quad\text{for all }(\underline{d},\underline{i})\in\mathbb{S}_{R,a}.$$

(26)

The condition in (26) captures the observation that any occurrence of the state $(\underline{d},\underline{i})\in\mathbb{S}_{R,a}$ is followed by selecting arm $a$ forcibly (i.e., with probability $1$), thus implying that $\nu(\underline{d},\underline{i},a^{\prime})=0$ for all $a^{\prime}\neq a$ which is equivalent to (26). Let $T_{R}^{\star}(C)$ be the optimal value of the following optimisation problem:

	$\displaystyle\sup_{\nu}\ \min_{C^{\prime}\in\textsf{Alt}(C)}\ \sum_{(\underline{d},\underline{i})\in\mathbb{S}_{R}}\ \sum_{a=1}^{K}\ \nu(\underline{d},\underline{i},a)\ D_{\textsf{KL}}((P_{C}^{a})^{d_{a}}(\cdot|i_{a})\|(P_{C^{\prime}}^{a})^{d_{a}}(\cdot|i_{a})),$		(27)
	subject to
	$\displaystyle\sum_{a=1}^{K}\nu(\underline{d}^{\prime},\underline{i}^{\prime},a)=\sum_{(\underline{d},\underline{i})\in\mathbb{S}_{R}}\ \sum_{a=1}^{K}\,\nu(\underline{d},\underline{i},a)\ Q_{C,R}(\underline{d}^{\prime},\underline{i}^{\prime}|\underline{d},\underline{i},a)\quad\text{for all }(\underline{d}^{\prime},\underline{i}^{\prime})\in\mathbb{S}_{R},$		(28)
	$\displaystyle\sum_{(\underline{d},\underline{i})\in\mathbb{S}_{R}}\ \sum_{a=1}^{K}\ \nu(\underline{d},\underline{i},a)=1,$		(29)
	$\displaystyle\nu(\underline{d},\underline{i},a)\geq 0\quad\text{for all }(\underline{d},\underline{i},a)\in\mathbb{S}_{R}\times\mathcal{A},$		(30)
	$\displaystyle\nu(\underline{d},\underline{i},a)=\sum_{a^{\prime}=1}^{K}\ \nu(\underline{d},\underline{i},a^{\prime})\quad\text{for all }(\underline{d},\underline{i})\in\mathbb{S}_{R,a},\quad a\in\mathcal{A}.$		(31)

Notice that the above optimisation problem is a finite-dimensional LP, and is the analogue of the infinite-dimensional LP in (21) with the additional condition in (31) to account for the case when an arm is forcibly selected if its delay equals $R$. Because (a) $\mathbb{S}_{R}\times\mathcal{A}$ is finite, (b) the space of all probability distributions on $\mathbb{S}_{R}\times\mathcal{A}$ (say, $\mathscr{P}(\mathbb{S}_{R}\times\mathcal{A})$) is compact with respect to the topology arising from the Euclidean metric in $\mathbb{R}^{|\mathbb{S}_{R}\times\mathcal{A}|}$, (c) the set of all $\nu$ satisfying (28)-(31) is a closed subset of $\mathscr{P}(\mathbb{S}_{R}\times\mathcal{A})$ (and therefore compact), and (d) the expression

$$\min_{C^{\prime}\in\textsf{Alt}(C)}\ \sum_{(\underline{d},\underline{i})\in\mathbb{S}_{R}}\ \sum_{a=1}^{K}\ \nu(\underline{d},\underline{i},a)\ D_{\textsf{KL}}((P_{C}^{a})^{d_{a}}(\cdot|i_{a})\|(P_{C^{\prime}}^{a})^{d_{a}}(\cdot|i_{a}))$$

is continuous in $\nu$, it follows by Weierstrass extreme value theorem that there exists $\nu_{C,R}^{\star}=\{\nu_{C,R}^{\star}(\underline{d},\underline{i},a):(\underline{d},\underline{i},a)\in\mathbb{S}_{R}\times\mathcal{A}\}$ that attains the supremum in (27). Although a closed-form expression for $\nu_{C,R}^{\star}$ is not currently available, it can easily be evaluated numerically. In the next section, we fix $R\in\mathbb{N}\cap(K,\infty)$ and design a policy for finding the best arm that samples an arm forcibly if its delay is equal to $R$. Additionally, we demonstrate that our policy (a) stops in finite time almost surely, (b) satisfies the desired error probability, and (c) achieves an upper bound of $1/T_{R}^{\star}(C)$ asymptotically as the error probability vanishes. We shall see that our policy is easy to implement as it operates on the finite set $\mathbb{S}_{R}$ instead of the countable set $\mathbb{S}$.

Fix an $R\in\mathbb{N}\cap(K,\infty)$. In this section, we design a policy for finding the best arm when the delay of each arm is constrained to be no more than $R$. Towards this, we first analyse a uniform arm selection policy that, for all $t\geq K$, selects the arms uniformly whenever the delay of each arm is $<R$, and forcibly selects an arm whose delay is equal to $R$. Let this policy be denoted $\pi_{R}^{\textsf{unif}}$. It is clear that $\{(\underline{d}(t),\underline{i}(t)):t\geq K\}$ is a Markov process under $\pi_{R}^{\textsf{unif}}$ with $\mathbb{S}_{R}$ as its state space. The following result shows that this Markov process is, in fact, ergodic.

As a consequence of Lemma 1, the process $\{(\underline{d}(t),\underline{i}(t)):\ t\geq K\}$ has a unique stationary distribution, say $\mu_{C,R}^{\textsf{unif}}=\{\mu_{C,R}^{\textsf{unif}}(\underline{d},\underline{i}):(\underline{d},\underline{i})\in\mathbb{S}_{R}\}$, under the policy $\pi_{R}^{\textsf{unif}}$ and under the assignment of the TPMs $C$. We note that $\mu_{C,R}^{\textsf{unif}}(\underline{d},\underline{i})>0$ for all $(\underline{d},\underline{i})\in\mathbb{S}_{R}$. Let

$$\nu_{C,R}^{\textsf{unif}}(\underline{d},\underline{i},a)\coloneqq\begin{cases}\frac{\mu_{C,R}^{\textsf{unif}}(\underline{d},\underline{i})}{K},&(\underline{d},\underline{i})\notin\bigcup_{a^{\prime}=1}^{K}\ \mathbb{S}_{R,a^{\prime}},\\ \mu_{C,R}^{\textsf{unif}}(\underline{d},\underline{i}),&(\underline{d},\underline{i})\in\mathbb{S}_{R,a},\\ 0,&(\underline{d},\underline{i})\in\bigcup_{a^{\prime}\neq a}\ \mathbb{S}_{R,a^{\prime}}.\end{cases}$$

(32)

denote the corresponding ergodic state-action occupancy measure. Observe that $\nu_{C,R}^{\textsf{unif}}$ satisfies (28)-(31).

For $\eta\in(0,1]$ and $C\in\mathcal{C}$, let

	$\displaystyle\nu_{\eta,R,C}(\underline{d},\underline{i},a)$	$\displaystyle\coloneqq\eta\ \nu_{C,R}^{\textsf{unif}}(\underline{d},\underline{i},a)+(1-\eta)\ \nu_{C,R}^{\star}(\underline{d},\underline{i},a),$		(33)
	$\displaystyle\mu_{\eta,R,C}(\underline{d},\underline{i})$	$\displaystyle\coloneqq\sum_{a=1}^{K}\ \nu_{\eta,R,C}(\underline{d},\underline{i},a).$		(34)

Observe that $\mu_{\eta,R,C}(\underline{d},\underline{i})\geq\frac{\eta}{K}\ \mu_{C,R}^{\textsf{unif}}(\underline{d},\underline{i})>0$ for all $(\underline{d},\underline{i})\in\mathbb{S}_{R}$. Also, $\nu_{\eta,R,C}$ satisfies (28)-(31) by virtue of the fact that both $\nu_{C,R}^{\textsf{unif}}$ and $\nu_{C,R}^{\star}$ satisfy (28)-(31). Let $\lambda_{\eta,R,C}=\{\lambda_{\eta,R,C}(a|\underline{d},\underline{i}):a\in\mathcal{A},\ (\underline{d},\underline{i})\in\mathbb{S}_{R}\}$ be defined as

$$\lambda_{\eta,R,C}(a|\underline{d},\underline{i})=\frac{\nu_{\eta,R,C}(\underline{d},\underline{i},a)}{\mu_{\eta,R,C}(\underline{d},\underline{i})}\quad(\underline{d},\underline{i})\in\mathbb{S}_{R},\ a\in\mathcal{A}.$$

(35)

Notice that for all $(\underline{d},\underline{i})\notin\bigcup_{a^{\prime}=1}^{K}\ \mathbb{S}_{R,a^{\prime}}$,

$\displaystyle\lambda_{\eta,R,C}(a|\underline{d},\underline{i})\geq\frac{\eta}{K}\ \mu_{C,R}^{\textsf{unif}}(\underline{d},\underline{i})$

$\displaystyle\geq\frac{\eta}{K}\ \mu_{R}^{\textsf{min}},$

(36)

where

$$\mu_{R}^{\textsf{min}}\coloneqq\min_{C\in\mathcal{C}}\ \min_{(\underline{d},\underline{i})\in\mathbb{S}_{R}}\mu_{C,R}^{\textsf{unif}}(\underline{d},\underline{i})>0.$$

(37)

That is, whenever the delay of each arm is $<R$, the distribution $\lambda_{\eta,R,C}(\cdot|\underline{d},\underline{i})$ puts a strictly positive mass on each of the arms. Using the arm selection rule in (35) instead of the uniform sampling rule and following the proof template in Appendix B, it can be shown that the policy $\pi^{\lambda_{\eta,R,C}}$, which selects the arms at each time instant according to the rule in (35), renders the process $\{(\underline{d}(t),\underline{i}(t)):t\geq K\}$ ergodic. We now claim that the stationary distribution of the process $\{(\underline{d}(t),\underline{i}(t)):t\geq K\}$ under $\pi^{\lambda_{\eta,R,C}}$ is $\mu_{\eta,R,C}$. Indeed, suppose $Q^{C,R}=\{Q^{C,R}(\underline{d}^{\prime},\underline{i}^{\prime}|\underline{d},\underline{i}):(\underline{d}^{\prime},\underline{i}^{\prime}),(\underline{d},\underline{i})\in\mathbb{S}_{R}\}$ denotes the transition probability matrix of the Markov process $\{(\underline{d}(t),\underline{i}(t)):t\geq K\}$ under the policy $\pi^{\lambda_{\eta,R,C}}$. Then,

$\displaystyle Q^{C,R}(\underline{d}^{\prime},\underline{i}^{\prime}|\underline{d},\underline{i})$

$\displaystyle=\sum_{a=1}^{K}\ \lambda_{\eta,R,C}(a|\underline{d},\underline{i})\ Q_{C,R}(\underline{d}^{\prime},\underline{i}^{\prime}|\underline{d},\underline{i},a),$

(38)

from which it follows that for all $(\underline{d}^{\prime},\underline{i}^{\prime})\in\mathbb{S}_{R}$,

	$\displaystyle\sum_{(\underline{d},\underline{i})\in\mathbb{S}_{R}}\ \mu_{\eta,R,C}(\underline{d},\underline{i})\ Q^{C,R}(\underline{d}^{\prime},\underline{i}^{\prime}|\underline{d},\underline{i})$	$\displaystyle=\sum_{(\underline{d},\underline{i})\in\mathbb{S}_{R}}\ \sum_{a=1}^{K}\ \mu_{\eta,R,C}(\underline{d},\underline{i})\ \lambda_{\eta,R,C}(a|\underline{d},\underline{i})\ Q_{C,R}(\underline{d}^{\prime},\underline{i}^{\prime}|\underline{d},\underline{i},a)$
		$\displaystyle=\sum_{(\underline{d},\underline{i})\in\mathbb{S}_{R}}\ \sum_{a=1}^{K}\ \nu_{\eta,R,C}(\underline{d},\underline{i},a)\ Q_{C,R}(\underline{d}^{\prime},\underline{i}^{\prime}|\underline{d},\underline{i},a)$
		$\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\sum_{a=1}^{K}\ \nu_{\eta,R,C}(\underline{d}^{\prime},\underline{i}^{\prime},a)$
		$\displaystyle=\mu_{\eta,R,C}(\underline{d}^{\prime},\underline{i}^{\prime}),$		(39)

thus proving the claim. In the above set of equations, $(a)$ follows from the fact that $\nu_{\eta,R,C}$ satisfies (28).