When are Kalman-Filter Restless Bandits Indexable?

We study the problem of monitoring several time series so as to maintain a precise belief while minimising the cost of sensing. Such problems can be viewed as POMDPs with belief-dependent rewards [3] and their applications include active sensing [7], attention mechanisms for multiple-object tracking [22], as well as online summarisation of massive data from time-series [4]. Specifically, we discuss the restless bandit [24] associated with the discrete-time Kalman filter [19].

Restless bandits generalise bandit problems [6, 8] to situations where the state of each arm (project, site or target) continues to change even if the arm is not played. As with bandit problems, the states of the arms evolve independently given the actions taken, suggesting that there might be efficient algorithms for large-scale settings, based on calculating an index for each arm, which is a real number associated with the (belief-)state of that arm alone. However, while bandits always have an optimal index policy (select the arm with the largest index), it is known that no index policy can be optimal for some discrete-state restless bandits [17] and such problems are in general PSPACE-hard even to approximate to any non-trivial factor [10]. Further, in this paper we address restless bandits with real-valued rather than discrete states.

On the other hand, Whittle proposed a natural index policy for restless bandits [24], but this policy only makes sense when the restless bandit is indexable, as we now explain. Say we have $n$ restless bandits and we are constrained to play $m$ arms at each time. Whittle considered relaxing this constraint by only requiring that the time-average number of arms played is $m$. Now the optimal average cost for this relaxed problem is a lower bound on the optimal average cost for the original problem. Also, the relaxed problem can be separated into $n$ single-arm problems by the method of Lagrange multipliers, making it relatively easy to solve. In this separated version of the relaxed problem, each arm behaves identically to an arm in the original problem, except that an additional price $\lambda$ is charged each time the arm is played, where $\lambda$ corresponds to the Lagrange multiplier for the relaxed constraint. Now let us consider a family of optimal policies which achieves the optimal cost-to-go $Q_{i}(x,u;\lambda)$ for a single arm $i$ with price $\lambda$ and which takes actions $u=\pi_{i}(x;\lambda)$ when in state $x$ where $u=0$ means passive and $u=1$ means active. At first glance, we might intuitively suppose that it becomes less and less attractive to be active as the price $\lambda$ increases so that as the price is increased beyond some value $\lambda_{i}(x)$, the optimal action switches from active to passive. At this price we are ambivalent between being active and passive so that $Q_{i}(x,0;\lambda_{i}(x))=Q_{i}(x,1;\lambda_{i}(x))$. Such a value $\lambda_{i}(x)$ is called the Whittle index for arm $i$ in state $x$. Indeed if there is a family of optimal policies for which

$\displaystyle\pi_{i}(x;\lambda_{\text{hi}})\leq\pi_{i}(x;\lambda_{\text{lo}})\quad\text{for all states $x$ and all pairs of prices $\lambda_{\text{hi}}\geq\lambda_{\text{lo}}$}$

then an optimal solution to the relaxed problem for price $\lambda$ is to activate arm $i$ if and only if $\lambda<\lambda_{i}(x)$. If a restless bandit satisfies this condition, it is said to be indexable. It is important to note that some restless bandits are not indexable, so activating arm $i$ if and only if $\lambda<\lambda_{i}(x)$ does not correspond to an optimal solution to the relaxed problem. Indeed, in a study of small randomly-generated problems, Weber and Weiss [23] found that roughly 10% of problems were not indexable.

As a policy based on $\lambda_{i}(x)$ is so good for the relaxed problem when the arms are indexable, this motivates us to use $\lambda_{i}(x)$ as a heuristic for the original problem. This heuristic is called Whittle’s index policy and at each time it activates the $m$ arms with the highest indexes $\lambda_{i}(x)$. Further motivation for studying indexability is that for ordinary bandits the Whittle index reduces to the Gittins index, making the Whittle index policy optimal when only one arm may be active at each time, that is when $m=1$. More generally, Whittle’s index policy is not optimal for some restless bandit problems even when the arms are indexable, but indexability is still a rather useful concept, since if all arms are indexable and certain other conditions hold, Whittle’s policy is asymptotically optimal, as we now explain. Consider a sequence of restless bandit problems parameterised by the number of indexable arms $n$ and in which $m=\alpha n$ of the arms can be simultaneously active for some fixed $\alpha\in(0,1)$. Then as $n$ tends to infinity, the time-average cost per arm for Whittle’s index policy converges to the time-average cost per arm for an optimal policy, provided a certain fluid approximation has a unique fixed point. This result was first demonstrated by Weber and Weiss [23] who for simplicity of exposition only considered the symmetric case in which the $n$ arms have identical costs and transition probabilities. Recently, Verloop [20] extended this result to asymmetric cases involving multiple types of arms. Interestingly, this extension also covers cases where new arms arrive and old arms depart.

Restless bandits associated with scalar Kalman(-Bucy) filters in continuous time were recently shown to be indexable [12] and the corresponding discrete-time problem has attracted considerable attention over a long period [15, 11, 16, 21]. However, that attention has produced no satisfactory proof of indexability – even for scalar time-series and even if we assume that there is a monotone optimal policy for the single-arm problem, which is a policy that plays the arm if and only if the relevant belief-state exceeds some threshold (here the relevant belief-state is a posterior variance). Theorem 1 of this paper addresses that gap. After formalising the problem (Section 2), we describe the concepts and intuition (Section 3) behind the main result (Section 4). The main tools are mechanical words (which are not sufficiently well-known) and Schur convexity. As these tools are associated with rather general theorems, we believe that future work (Section 5) should enable substantial generalisation of our results.

We consider the problem of tracking $N$ time-series, which we call arms, in discrete time. The state $Z_{i,t}\in\mathbb{R}$ of arm $i$ at time $t\in\mathbb{Z}_{+}$ evolves as a standard-normal random walk independent of everything but its immediate past ($\mathbb{Z}_{+},\mathbb{R}_{-}$ and $\mathbb{R}_{+}$ all include zero). The action space is $\mathcal{U}:=\{1,\dots,N\}$. Action $u_{t}=i$ makes an expensive observation $Y_{i,t}$ of arm $i$ which is normally-distributed about $Z_{i,t}$ with precision $b_{i}\in\mathbb{R}_{+}$ and we receive cheap observations $Y_{j,t}$ of each other arm $j$ with precision $a_{j}\in\mathbb{R}_{+}$ where $a_{j}<b_{j}$ and $a_{j}=0$ means no observation at all.

Let $Z_{t},Y_{t},\mathcal{H}_{t},\mathcal{F}_{t}$ be the state, observation, history and observed history, so that $Z_{t}:=(Z_{1,t},\dots,Z_{N,t}),Y_{t}:=(Y_{1,t},\dots,Y_{N,t}),\mathcal{H}_{t}:=((Z_{0},u_{0},Y_{0}),\dots,(Z_{t},u_{t},Y_{t}))$ and $\mathcal{F}_{t}:=((u_{0},Y_{0}),\dots,(u_{t},Y_{t})).$ Then we formalise the above as (${\bf 1}_{\cdot}$ is the indicator function)

$\displaystyle Z_{i,0}$

$\displaystyle\sim\mathcal{N}(0,1),$

$\displaystyle Z_{i,t+1}\mid\mathcal{H}_{t}$

$\displaystyle\sim\mathcal{N}(Z_{i,t},1),$

$\displaystyle Y_{i,t}\mid\mathcal{H}_{t-1},Z_{t},u_{t}$

$\displaystyle\sim\mathcal{N}\left(Z_{i,t},\frac{{\bf 1}_{u_{t}\neq i}}{a_{i}}+\frac{{\bf 1}_{u_{t}=i}}{b_{i}}\right).$

Note that this setting is readily generalised to $\mathbb{E}[(Z_{i,t+1}-Z_{i,t})^{2}]\neq 1$ by a change of variables.

Thus the posterior belief is given by the Kalman filter as $Z_{i,t}\mid\mathcal{F}_{t}\sim\mathcal{N}(\hat{Z}_{i,t},x_{i,t})$ where the posterior mean is ${\hat{Z}}_{i,t}\in\mathbb{R}$ and the error variance $x_{i,t}\in\mathbb{R}_{+}$ satisfies

$\displaystyle x_{i,t+1}=\phi_{i,{\bf 1}_{u_{t+1}=i}}(x_{i,t})\quad\text{where}\quad\phi_{i,0}(x):=\frac{x+1}{a_{i}x+a_{i}+1}\ \ \text{and}\ \ \phi_{i,1}(x):=\frac{x+1}{b_{i}x+b_{i}+1}.$

(1)

Problem KF1. Let $\pi$ be a policy so that $u_{t}=\pi(\mathcal{F}_{t-1})$. Let $x^{\pi}_{i,t}$ be the error variance under $\pi$. The problem is to choose $\pi$ so as to minimise the following objective for discount factor $\beta\in[0,1)$. The objective consists of a weighted sum of error variances $x^{\pi}_{i,t}$ with weights $w_{i}\in\mathbb{R}_{+}$ plus observation costs $h_{i}\in\mathbb{R}_{+}$ for $i=1,\dots,N$:

$\displaystyle\mathbb{E}\left[\sum_{t=0}^{\infty}\sum_{i=1}^{N}\beta^{t}\left\{h_{i}{\bf 1}_{u_{t}=i}+w_{i}x_{i,t}^{\pi}\right\}\right]=\sum_{t=0}^{\infty}\sum_{i=1}^{N}\beta^{t}\left\{h_{i}{\bf 1}_{u_{t}=i}+w_{i}x_{i,t}^{\pi}\right\}$

where the equality follows as (1) is a deterministic mapping (and assuming $\pi$ is deterministic).

Single-Arm Problem and Whittle Index. Now fix an arm $i$ and write $x_{t}^{\pi},\phi_{0}(\cdot),\dots$ instead of $x_{t,i}^{\pi},\phi_{i,0}(\cdot),\dots$. Say there are now two actions $u_{t}=0,1$ corresponding to cheap and expensive observations respectively and the expensive observation now costs $h+\nu$ where $\nu\in\mathbb{R}$. The single-arm problem is to choose a policy, which here is an action sequence, $\pi:=(u_{0},u_{1},\dots)$

$\displaystyle\text{so as to minimise}\quad V^{\pi}(x|\nu):=\sum_{t=0}^{\infty}\beta^{t}\left\{(h+\nu)u_{t}+wx_{t}^{\pi}\right\}\quad\text{where $x_{0}=x$.}$

(2)

Let $Q(x,\alpha|\nu)$ be the optimal cost-to-go in this problem if the first action must be $\alpha$ and let $\pi^{*}$ be an optimal policy, so that

$\displaystyle Q(x,\alpha|\nu):=(h+\nu)\alpha+wx+\beta V^{\pi^{*}}(\phi_{\alpha}(x)|\nu).$

For any fixed $x\in\mathbb{R}_{+}$, the value of $\nu$ for which actions $u_{0}=0$ and $u_{0}=1$ are both optimal is known as the Whittle index $\lambda^{W}(x)$ assuming it exists and is unique. In other words

The Whittle index $\lambda^{W}(x)$ is the solution to $Q(x,0|\lambda^{W}(x))=Q(x,1|\lambda^{W}(x)).$

(3)

Let us consider a policy which takes action $u_{0}=\alpha$ then acts optimally producing actions $u_{t}^{\alpha*}(x)$ and error variances $x_{t}^{\alpha*}(x)$. Then (3) gives

$\displaystyle\sum_{t=0}^{\infty}\beta^{t}\left\{(h+\lambda^{W}(x))u^{0*}_{t}+wx_{t}^{0*}(x)\right\}=\sum_{t=0}^{\infty}\beta^{t}\left\{(h+\lambda^{W}(x))u^{1*}_{t}+wx_{t}^{1*}(x)\right\}.$

Solving this linear equation for the index $\lambda^{W}(x)$ gives

$\displaystyle\lambda^{W}(x)=w\frac{\sum_{t=1}^{\infty}\beta^{t}(x_{t}^{0*}(x)-x_{t}^{1*}(x))}{\sum_{t=0}^{\infty}\beta^{t}(u^{1*}_{t}(x)-u^{0*}_{t}(x))}-h.$

(4)

Whittle [24] recognised that for his index policy (play the arm with the largest $\lambda^{W}(x)$) to make sense, any arm which receives an expensive observation for added cost $\nu$, must also receive an expensive observation for added cost $\nu^{\prime}<\nu$. Such problems are said to be indexable. The question resolved by this paper is whether Problem KF1 is indexable. Equivalently, is $\lambda^{W}(x)$ non-decreasing in $x\in\mathbb{R}_{+}$?

We make the following intuitive assumption about threshold (monotone) policies.
A1. For some $x\in\mathbb{R}_{+}$ depending on $\nu\in\mathbb{R}$, the policy $u_{t}={\bf 1}_{x_{t}\geq x}$ is optimal for problem (2).

Note that under A1, definition (3) means the policy $u_{t}={\bf 1}_{x_{t}>x}$ is also optimal, so we can choose

$\displaystyle\left.\begin{aligned} u_{t}^{0*}(x)&:=\begin{cases}0&\text{if $x_{t-1}^{0*}(x)\leq x$}\\ 1&\text{otherwise}\end{cases}&\quad\text{and}\quad x_{t}^{0*}(x)&:=\begin{cases}\phi_{0}(x_{t-1}^{0*}(x))&\text{if $x_{t-1}^{0*}(x)\leq x$}\\ \phi_{1}(x_{t-1}^{0*}(x))&\text{otherwise}\end{cases}\\ u_{t}^{1*}(x)&:=\begin{cases}0&\text{if $x_{t-1}^{1*}(x)<x$}\\ 1&\text{otherwise}\end{cases}&\quad\text{and}\quad x_{t}^{1*}(x)&:=\begin{cases}\phi_{0}(x_{t-1}^{1*}(x))&\text{if $x_{t-1}^{1*}(x)<x$}\\ \phi_{1}(x_{t-1}^{1*}(x))&\text{otherwise}\end{cases}\end{aligned}\quad\right\}$

(5)

where $x_{0}^{0*}(x)=x_{0}^{1*}(x)=x$. We refer to $x_{t}^{0*}(x),x_{t}^{1*}(x)$ as the $x$-threshold orbits (Figure 1).

We are now ready to state our main result.

Theorem 1. Suppose a threshold policy (A1) is optimal for the single-arm problem (2). Then Problem KF1 is indexable. Specifically, for any $b>a\geq 0$ let

$\displaystyle\phi_{0}(x)$

$\displaystyle:=\frac{x+1}{ax+a+1},$

$\displaystyle\phi_{1}(x)$

$\displaystyle:=\frac{x+1}{bx+b+1}$

and for any $w\in\mathbb{R}_{+},h\in\mathbb{R}$ and $0<\beta<1$, let

$\displaystyle\lambda^{W}(x):=w\frac{\sum_{t=1}^{\infty}\beta^{t}(x_{t}^{0*}(x)-x_{t}^{1*}(x))}{\sum_{t=0}^{\infty}\beta^{t}(u^{1*}_{t}(x)-u^{0*}_{t}(x))}-h$

(6)

in which action sequences $u_{t}^{0*}(x),u_{t}^{1*}(x)$ and error variance sequences $x_{t}^{0*}(x),x_{t}^{1*}(x)$ are given in terms of $\phi_{0},\phi_{1}$ by (5). Then $\lambda^{W}(x)$ is a continuous and non-decreasing function of $x\in\mathbb{R}_{+}$.

We are now ready to describe the key concepts underlying this result.

Words. In this paper, a word $w$ is a string on $\{0,1\}^{*}$ with $k^{\rm th}$ letter $w_{k}$ and $w_{i:j}:=w_{i}w_{i+1}\dots w_{j}$. The empty word is $\epsilon$, the concatenation of words $u,v$ is $uv$, the word that is the $n$-fold repetition of $w$ is $w^{n}$, the infinite repetition of $w$ is $w^{\omega}$ and $\tilde{w}$ is the reverse of $w$, so $w=\tilde{w}$ means $w$ is a palindrome. The length of $w$ is ${\left|w\right|}$ and ${\left|w\right|}_{u}$ is the number of times that word $u$ appears in $w$, overlaps included.

Christoffel, Sturmian and Mechanical Words. It turns out that the action sequences in (5) are given by such words, so the following definitions are central to this paper.

The Christoffel tree (Figure 2) is an infinite complete binary tree [5] in which each node is labelled with a pair $(u,v)$ of words. The root is $(0,1)$ and the children of $(u,v)$ are $(u,uv)$ and $(uv,v)$. The Christoffel words are the words $0,1$ and the concatenations $uv$ for all $(u,v)$ in that tree. The fractions ${\left|uv\right|}_{1}/{\left|uv\right|}_{0}$ form the Stern-Brocot tree [9] which contains each positive rational number exactly once. Also, infinite paths in the Stern-Brocot tree converge to the positive irrational numbers. Analogously, Sturmian words could be thought of as infinitely-long Christoffel words.

Alternatively, among many known characterisations, the Christoffel words can be defined as the words $0,1$ and the words $0w1$ where $a:={\left|0w1\right|}_{1}/{\left|0w1\right|}$ and

$\displaystyle(01w)_{n}:=\lfloor(n+1)a\rfloor-\lfloor na\rfloor$

for any relatively prime natural numbers ${\left|0w1\right|}_{0}$ and ${\left|0w1\right|}_{1}$ and for $n=1,2,\dots,{\left|0w1\right|}$. The Sturmian words are then the infinite words $0w_{1}w_{2}\cdots$ where, for $n=1,2,\dots$ and $a\in(0,1)\backslash\mathbb{Q}$,

$\displaystyle(01w_{1}w_{2}\cdots)_{n}:=\lfloor(n+1)a\rfloor-\lfloor na\rfloor.$

We use the notation $0w1$ for Sturmian words although they are infinite.

The set of mechanical words is the union of the Christoffel and Sturmian words [13]. (Note that the mechanical words are sometimes defined in terms of infinite repetitions of the Christoffel words.)

Majorisation. As in [14], let $x,y\in\mathbb{R}^{m}$ and let $x_{(i)}$ and $y_{(i)}$ be their elements sorted in ascending order. We say $x$ is weakly supermajorised by $y$ and write $x\prec^{w}y$ if

$\displaystyle\sum_{k=1}^{j}x_{(k)}\geq\sum_{k=1}^{j}y_{(k)}\qquad\text{for all $j=1,\dots,m$.}$

If this is an equality for $j=m$ we say $x$ is majorised by $y$ and write $x\prec y$. It turns out that

	$\displaystyle x\prec y\qquad$	$\displaystyle\Leftrightarrow\qquad\sum_{k=1}^{j}x_{[k]}\leq\sum_{k=1}^{j}y_{[k]}\quad\text{for $j=1,\dots,m-1$ with equality for $j=m$}$
where $x_{[k]},y_{[k]}$ are the sequences sorted in descending order. For $x,y\in\mathbb{R}^{m}$ we have [14]
	$\displaystyle x\prec y\qquad$	$\displaystyle\Leftrightarrow\qquad\sum_{i=1}^{m}f(x_{i})\leq\sum_{i=1}^{m}f(y_{i})\quad\text{for all convex functions $f:\mathbb{R}\rightarrow\mathbb{R}$.}$

More generally, a real-valued function $\phi$ defined on a subset $\mathcal{A}$ of $\mathbb{R}^{m}$ is said to be Schur-convex on $\mathcal{A}$ if $x\prec y$ implies that $\phi(x)\leq\phi(y)$.

Möbius Transformations. Let $\mu_{A}(x)$ denote the Möbius transformation $\mu_{A}(x):=\frac{A_{11}x+A_{12}}{A_{21}x+A_{22}}$ where $A\in\mathbb{R}^{2\times 2}$. Möbius transformations such as $\phi_{0}(\cdot),\phi_{1}(\cdot)$ are closed under composition, so for any word $w$ we define $\phi_{w}(x):=\phi_{w_{{\left|w\right|}}}\circ\dots\circ\phi_{w_{2}}\circ\phi_{w_{1}}(x)$ and $\phi_{\epsilon}(x):=x.$

Intuition. Here is the intuition behind our main result.

For any $x\in\mathbb{R}_{+}$, the orbits in (5) correspond to a particular mechanical word $0,1$ or $0w1$ depending on the value of $x$ (Figure 1). Specifically, for any word $u$, let $y_{u}$ be the fixed point of the mapping $\phi_{u}$ on $\mathbb{R}_{+}$ so that $\phi_{u}(y_{u})=y_{u}$ and $y_{u}\in\mathbb{R}_{+}$. Then the word corresponding to $x$ is 1 for $0\leq x\leq y_{1}$, $0w1$ for $x\in[y_{01w},y_{10w}]$ and 0 for $y_{0}\leq x<\infty$. In passing we note that these fixed points are sorted in ascending order by the ratio $\rho:={\left|01w\right|}_{0}/{\left|01w\right|}_{1}$ of counts of 0s to counts of 1s, as illustrated by Figure 3. Interestingly, it turns out that ratio $\rho$ is a piecewise-constant yet continuous function of $x$, reminiscent of the Cantor function.

Also, composition of Möbius transformations is homeomorphic to matrix multiplication so that

$\displaystyle\mu_{A}\circ\mu_{B}(x)=\mu_{AB}(x)\qquad\text{for any $A,B\in\mathbb{R}^{2\times 2}.$}$

Thus, the index (6) can be written in terms of the orbits of a linear system (11) given by $0,1$ or $0w1.$ Further, if $A\in\mathbb{R}^{2\times 2}$ and $\det(A)=1$ then the gradient of the corresponding Möbius transformation is the convex function

$\displaystyle\frac{d\mu_{A}(x)}{dx}=\frac{1}{(A_{21}x+A_{22})^{2}}.$

So the gradient of the index is the difference of the sums of a convex function of the linear-system orbits. However, such sums are Schur-convex functions and it follows that the index is increasing because one orbit weakly supermajorises the other, as we now show for the case $0w1$ (noting that the proof is easier for words $0,1$). As $0w1$ is a mechanical word, $w$ is a palindrome. Further, if $w$ is a palindrome, it turns out that the difference between the linear-system orbits increases with $x$. So, we might define the majorisation point for $w$ as the $x$ for which one orbit majorises the other. Quite remarkably, if $w$ is a palindrome then the majorisation point is $\phi_{w}(0)$ (Proposition 7). Indeed the black circles and blue dots of Figure 3 coincide. Finally, $\phi_{w}(0)$ is less than or equal to $y_{01w}$ which is the least $x$ for which the orbits correspond to the word $0w1$. Indeed, the blue dots of Figure 3 are below the corresponding black dots. Thus one orbit does indeed supermajorise the other.