Indexed Minimum Empirical Divergence-Based Algorithms
for Linear Bandits

The $K$-armed bandit setting is a special case of the linear bandit. There exist several good algorithms such as UCB1 (Auer et al. (2002)), Thompson Sampling (Agrawal & Goyal (2012)), and the Indexed Minimum Empirical Divergence (IMED) algorithm (Honda & Takemura (2015)) for this setting. There are three main families of asymptotically optimal multi-armed bandit algorithms based on different principles (Baudry et al. (2023)). However, among these algorithms, only IMED lacks an extension for contextual bandits with linear payoff. In the context of the varying arm setting of the linear bandit problem, the LinUCB algorithm in Li et al. (2010) is frequently employed in practice. It has a theoretical guarantee on the regret in the order of $O(d\sqrt{T}\log(T))$ using the confidence width as in OFUL (Abbasi-Yadkori et al. (2011)). Although the SupLinUCB algorithm introduced by Chu et al. (2011) uses phases to decompose the reward dependence of each time step and achieves an $\widetilde{O}(\sqrt{dT})$ (the $\widetilde{O}(\cdot)$ notation omits logarithmic factors in $T$) regret upper bound, its empirical performance falls short of both the algorithm in Li et al. (2010) and the Linear Thompson Sampling algorithm (Agrawal & Goyal (2013)) as mentioned in Lattimore & Szepesvári (2020, Chapter 22).

On the other hand, the Optimism in the Face of Uncertainty Linear (OFUL) bandit algorithm in Abbasi-Yadkori et al. (2011) achieves a regret upper bound of $\widetilde{O}(d\sqrt{T})$ through an improved analysis of the confidence bound using a martingale technique. However, it involves a bilinear optimization problem over the action set and the confidence ellipsoid when choosing the arm at each time. This is computationally expensive, unless the confidence ellipsoid is a convex hull of a finite set.

For randomized algorithms designed for the linear bandit problem, Agrawal & Goyal (2013) proposed the LinTS algorithm, which is in the spirit of Thompson Sampling (Thompson (1933)) and the confidence ellipsoid similar to that of LinUCB-like algorithms. This algorithm performs efficiently and achieves a regret upper bound of ${O}(d^{\frac{3}{2}}\sqrt{T}\wedge d\sqrt{T\log K})$, where $K$ is the number of arms at each time step such that $|\mathcal{A}_{t}|=K$ for all $t$. Compared to LinUCB with OFUL’s confidence width, it has an extra $O(\sqrt{d}\wedge\sqrt{\log K})$ term for the minimax regret upper bound.

Recently, MED-like (minimum empirical divergence) algorithms have come to the fore since these randomized algorithms have the property that the probability of selecting each arm is in closed form, which benefits downstream work such as offline evaluation with the inverse propensity score. Both MED in the sub-Gaussian environment and its deterministic version IMED have demonstrated superior performances over Thompson Sampling (Bian & Jun (2021), Honda & Takemura (2015)). Baudry et al. (2023) also shows MED has a close relation to Thompson Sampling. In particular, it is argued that MED and TS can be interpreted as two variants of the same exploration strategy. Bian & Jun (2021) also shows that probability of selecting each arm of MED in the sub-Gaussian case can be viewed as a closed-form approximation of the same probability as in Thompson Sampling. We take inspiration from the extension of Thompson Sampling to linear bandits and thus are motivated to extend MED-like algorithms to the linear bandit setting and prove regret bounds that are competitive vis-à-vis the state-of-the-art bounds.

Thus, this paper aims to answer the question of whether it is possible to devise an extension of the IMED algorithm for the linear bandit problem the varying arm set setting (for both infinite and finite arm sets) with a regret upper bound of $O(d\sqrt{T}\log T)$ which matches LinUCB with OFUL’s confidence bound while being as efficient as LinUCB. The proposed family of algorithms, called LinIMED as well as SupLinIMED, can be viewed as generalizations of the IMED algorithm (Honda & Takemura (2015)) to the linear bandit setting. We prove that LinIMED and its variants achieve a regret upper bound of $\widetilde{O}(d\sqrt{T})$ and they perform efficiently, no worse than LinUCB. SupLinIMED has a regret bound of $\widetilde{O}(\sqrt{dT})$, but works only for instances with finite arm sets. In our empirical study, we found that the different variants of LinIMED perform better than LinUCB and LinTS for various synthetic and real-world instances under consideration.

Compared to OFUL, LinIMED works more efficiently. Compared to SupLinUCB, our LinIMED algorithm is significantly simpler, and compared to LinUCB with OFUL’s confidence bound, our empirical performance is better. This is because in our algorithm, the exploitation term and exploration term are decoupling and this leads to a finer control while tuning the hyperparameters in the empirical study.

Compared to LinTS, our algorithm’s (specifically LinIMED-3) regret bound is superior, by an order of $O(\sqrt{d}\wedge\sqrt{\log K})$. Since fixed arm setting is a special case of finite varying arm setting, our result is more general than other fixed-arm linear bandit algorithms like Spectral Eliminator (Valko et al. (2014)) and PEGOE (Lattimore & Szepesvári (2020, Chapter 22)). Finally, we observe that since the index used in LinIMED has a similar form to the index used in the Information Directed Sampling (IDS) procedure in Kirschner et al. (2021) (which is known to be asymptotically optimal but more difficult to compute), LinIMED performs significantly better on the “End of Optimism” example in Lattimore & Szepesvari (2017). We summarize the comparisons of LinIMED to other linear bandit algorithms in Table 1. We discussion comparisons to other linear bandit algorithms in Sections 3.2, 3.3, and Appendix B.

Now we consider the case in which the arm set $\mathcal{A}_{t}$ at each time $t$ is finite but still time-varying. In particular, $\mathcal{A}_{t}=\{a_{t,1},a_{t,2},\ldots,a_{t,K}\}\subseteq\mathbb{R}$ are sets of constant size $K$ such that $|\mathcal{A}_{t}|=K<\infty$. In the pseudocode of Algorithm 2, we apply the SupLinUCB framework (Chu et al., 2011), leveraging Algorithm 3 (in Appendix A) as a subroutine within each phase. This ensures the independence of the choice of the arm from past observations of rewards, thereby yielding a concentration inequality in the estimated reward (see Lemma 1 in Chu et al. (2011)) that converges to within $\sqrt{d}$ proximity of the unknown expected reward in a finite arm setting. As a result, the regret yields an improvement of $\sqrt{d}$ ignoring the logarithmic factor. At each time step $t$ and phase $s$, in Line 5, we utilize the BaseLinUCB Algorithm as a subroutine to calculate the sample mean and confidence width since we also need these terms to calculate the IMED-style indices of each arm. In Lines 6–9 (Case 1), if the width of each arm is less than $\frac{1}{\sqrt{T}}$, we choose the arm with the smaller IMED-style index. In Lines 10–12 (Case 2), the framework is the same as in SupLinUCB (Chu et al. (2011)), if the width of each arm is smaller than $2^{-s}$ but there exist arms with widths larger than $\frac{1}{\sqrt{T}}$, then in Line 11 the “unpromising” arms will be eliminated until the width of each arm is smaller enough to satisfy the condition in Line 6. Otherwise, if there exist any arms with widths that are larger than $2^{-s}$, in Lines 14–15 (Case 3), we choose one such arm and record the context and reward of this arm to the next layer $\Psi^{s}_{t+1}$.

The IMED algorithm is a deterministic algorithm for the $K$-armed bandit problem. At each time step $t$, it chooses the arm $a$ with the minimum index, i.e.,

where $T_{i}(t)=\sum_{s=1}^{t-1}\mathbbm{1}\left\{A_{t}=a\right\}$ is the total arm pulls of the arm $i$ until time $t$ and $D_{\text{inf}}(\hat{F}_{i}(t),\hat{\mu}^{*}(t))$ is some divergence measure between the empirical distribution of the sample mean for arm $i$ and the arm with the highest sample mean. More precisely, $D_{\text{inf}}(F,\mu):=\inf_{G\in\mathcal{G}:\mathbb{E}(G)\leq\mu}D(F\|G)$ and $\mathcal{G}$ is the family of distributions supported on $(-\infty,1]$. As shown in Honda & Takemura (2015), its asymptotic bound is even better than KL-UCB (Garivier & Cappé (2011)) algorithm and can be extended to semi-bounded support models such as $\mathcal{G}$. Also, this algorithm empirically outperforms the Thompson Sampling algorithm as shown in Honda & Takemura (2015). However, an extension of IMED algorithm with minimax regret bound of $\widetilde{O}(d\sqrt{T})$ has not been derived. In our design of LinIMED algorithm, we replace the optimized KL-divergence measure in IMED in Eqn. equation 3 with the squared gap between the sample mean of the arm $i$ and the arm with the maximum sample mean. This choice simplifies our analysis and does not adversely affect the regret bound. On the other hand, we view the term $1/T_{i}(t)$ as the variance of the sample mean of arm $i$ at time $t$; then in this spirit, we use $\beta_{t-1}(\gamma)\lVert x_{t,a}\rVert^{2}_{V_{t-1}^{-1}}$ as the variance of the sample mean (which is $\langle\hat{\theta}_{t-1},x_{t,a}\rangle$) of arm $a$ at time $t$. We choose ${\hat{\Delta}_{t,a}^{2}}/{(\beta_{t-1}(\gamma)\lVert x_{t,a}\rVert_{V_{t-1}^{-1}}^{2})}$ instead of the KL-divergence approximation for the index since in the classical linear bandit setting, the noise is sub-Gaussian and it is known that the KL-divergence of two Gaussian random variables with the same variance ($\mathrm{KL}(\mathcal{N}(\mu_{1},\sigma^{2}),\mathcal{N}(\mu_{2},\sigma^{2}))=\frac{(\mu_{1}-\mu_{2})^{2}}{2\sigma^{2}}$) has a closed form expression similar to ${\hat{\Delta}_{t,a}^{2}}/{(\beta_{t-1}(\gamma)\lVert x_{t,a}\rVert_{V_{t-1}^{-1}}^{2})}$ ignoring the constant $\frac{1}{2}$.

Information Directed Sampling (IDS), introduced by Russo & Van Roy (2014), serves as a good principle for regret minimization in linear bandits to achieve the asymptotic optimality. The intuition behind IDS is to balance between the information gain on the best arm and the expected reward at each time step. This goal is realized by optimizing the distribution of selecting each arm $\pi\in\mathcal{D}(\mathcal{A})$ (where $\mathcal{A}$ is the fixed finite arm set) with the minimum information ratio such that:

where $\hat{\Delta}_{t}$ is the empirical gap and $g_{t}$ is the so-called information gain (defined later). Kirschner & Krause (2018), Kirschner et al. (2020) and Kirschner et al. (2021) apply the IDS principle to the linear bandit setting, The first two works propose both randomized and deterministic versions of IDS for linear bandit. They showed a near-optimal minimax regret bound of the order of $\widetilde{O}(d\sqrt{T})$. Kirschner et al. (2021) designed an asymptotically optimal linear bandit algorithm retaining its near-optimal minimax regret properties. Comparing these algorithms with our LinIMED algorithms, we observe that the first term of the index of non-greedy actions in our algorithms is $\hat{\Delta}^{2}_{t,a}/(\beta_{t-1}(\gamma)\lVert x_{t,a}\rVert^{2}_{V^{-1}_{t-1}})$, which is similar to the choice of information ratio in IDS with the estimated gap $\Delta_{t}(a):=\hat{\Delta}_{t,a}$ as defined in Algorithm 1 and the information ratio $g_{t}(a):=\beta_{t-1}(\gamma)\lVert x_{t,a}\rVert^{2}_{V^{-1}_{t-1}}$. As mentioned in Kirschner & Krause (2018), when the noise is 1-subgaussian and $\lVert x_{t,a}\rVert^{2}_{V^{-1}_{t-1}}\ll 1$, the information gain in deterministic IDS algorithm is approximately $\lVert x_{t,a}\rVert^{2}_{V^{-1}_{t-1}}$, which is similar to our choice $\beta_{t-1}(\gamma)\lVert x_{t,a}\rVert^{2}_{V^{-1}_{t-1}}$. However, our LinIMED algorithms are different from the deterministic IDS algorithm in Kirschner & Krause (2018) since the estimated gap defined in our algorithm $\hat{\Delta}_{t,a}$ is different from that in deterministic IDS. Furthermore, as discussed in Kirschner et al. (2020), when the noise is 1-subgaussian and $\lVert x_{t,a}\rVert^{2}_{V^{-1}_{t-1}}\ll 1$, the action chosen by UCB minimizes the deterministic information ratio. However, this is not the case for our algorithm since we have the second term $-\log(\beta_{t-1}(\gamma)\lVert x_{t,a}\rVert^{2}_{V^{-1}_{t-1}})$ in LinIMED-1 which balances information and optimism. Compared to IDS in Kirschner et al. (2021), their algorithm is a randomized version of the deterministic IDS algorithm, which is more computationally expensive than our algorithm since our LinIMED algorithms are fully deterministic (the support of the allocation in Kirschner et al. (2021) is two). IDS defines a more complicated version of information gain to achieve asymptotically optimality. Finally, to the best of our knowledge, all these IDS algorithms are designed for linear bandits under the setting that the arm set is fixed and finite, while in our setting we assume the arm set is finite and can change over time. We discuss comparisons to other related work in Appendix B.