Adversarial Bandits against Arbitrary Strategies

We can easily obtain the regret lower bound of this problem from the well-known regret lower bound of adversarial bandits. Let $t_{s}$ be the time when the $s$-th switch of the best arm in hindsight happens for $s\in[S]$ and $t_{S+1}-1=T$, $t_{0}=1$. We consider that $t_{s}$’s are equally distributed over $T$. Then we have $T_{s}:=t_{s+1}-t_{s}=\Theta(T/S)$ for $s\in[0,S]$. Then from Theorem 5.1 in Auer et al., (2002), for the best arm in hindsight over $T_{s}$ time steps, we get the regret lower bound of $\Omega(\sqrt{KT_{s}})$. We can obtain that

However, determining the feasibility of a tighter regret lower bound under undetermined $S$ remains an unresolved challenge.

In the master-base framework, at each time, a master algorithm selects a base and the selected base selects an arm. For the undetermined switch value $S$ in advance, we suggest tuning each base algorithm using a candidate of $S$ as follows.

Let $\mathcal{H}$ represent the set of candidates of the switch parameter $S\in[T-1]$ for the bases such that:

Then, each base adopts one of the candidate parameters in $\mathcal{H}$ for tuning its learning rate. For simplicity, let $H=|\mathcal{H}|$ such that $H=O(\log(T))$ and let base $h$ represent the base having the candidate parameter $h\in\mathcal{H}$ when there is no confusion. Also, let $h^{\dagger}$ be the largest value $h\leq S$ among $h\in\mathcal{H}$, which indicates the near-optimal parameter for $S$. Then we can observe that

Here we describe the OMD method Lattimore and Szepesvári, (2020). For a regularizer function $F:\mathbb{R}^{d}\rightarrow\mathbb{R}$ and $\textit{{p}},\textit{{q}}\in\mathbb{R}^{d}$, we define Bregman divergence as

Let $\textit{{p}}_{t}$ be the distribution for selecting an action at time $t$ and $\mathcal{P}_{d}$ be the probability simplex with dimension $d$. Then with a loss vector l, using the online mirror descent we can get $\textit{{p}}_{t+1}$ as follows:

The solution of (1) can be found using the following two-step procedure:

We use a regularizer $F$ that contains a learning rate (to be specified later). We note that, in the bandit setting, we cannot observe full information of loss at time $t$, but get partial feedback based on a selected action. Therefore, it is required to use an estimated loss vector for OMD.

We first provide a simple master-base OMD algorithm (Algorithm 1) with the negative entropy regularizer defined as

where $\textit{{p}}\in\mathbb{R}^{d}$, $p(i)$ denotes the $i$-index entry for p, and $\eta$ is a learning rate. We note that well-known EXP3 Auer et al., (2002) for the adversarial bandits is also based on the negative entropy function.

In Algorithm 1, at each time, the master selects a base $h_{t}$ from distribution $\textit{{p}}_{t}$. Then following distribution $\textit{{p}}_{t,h_{t}}$ for selecting an arm, the base $h_{t}$ selects an arm $a_{t}$ and receive a corresponding loss $l_{t}(a_{t})$. Using the loss feedback, it gets unbiased estimators $l_{t}^{\prime}(h)$ and $l_{t,h}^{\prime\prime}(a)$ for a loss from selecting each base $h\in\mathcal{H}$ and each arm $a\in[K]$, respectively. Then using OMD with the estimators, it updates the distributions $\textit{{p}}_{t+1}$ and $\textit{{p}}_{t+1,h}$ for selecting a base and an arm from base $h$, respectively.

For getting $\textit{{p}}_{t+1}$, it uses the negative entropy regularizer with learning rate $\eta$. The domain for updating the distribution for selecting a base is defined as a clipped probability simplex such that $\mathcal{P}_{H}^{\alpha}=\mathcal{P}_{H}\cap[\alpha,1]^{H}$ for $\alpha>0$. By introducing $\alpha$, it can control the variance of estimator $l_{t}^{\prime}(h)=l_{t}(a_{t})\mathbbm{1}(h=h_{t})/p_{t}(h)$ by restricting the minimum value for $p_{t}(h)$. For getting $\textit{{p}}_{t+1,h}$, it also uses the negative entropy regularizer with learning rates depending on a value of $h$ for each base. The learning rate $\eta(h)$ is tuned by using a candidate value $h$ for $S$ in the base $h$ to control adaptation for switching such that

The domain for the distribution is also defined as a clipped probability simplex such that $\mathcal{P}_{K}^{\beta}=\mathcal{P}_{K}\cap[\beta,1]^{K}$ for $\beta>0$. The purpose of $\beta$ is to introduce some regularization in learning $\textit{{p}}_{t+1,h}$ for dealing with switching best arms in hindsight, which is slightly different from the purpose of $\alpha$.

Now we provide a regret bound for the algorithm in the following theorem.

From Theorem 1, the regret bound of Algoritm 1 is tight with respect to $S$ compared to that of EXP3.S Auer et al., (2002) which has a linear dependency on $S$. Therefore, when $S$ is large (specifically $S=\omega((T/K)^{1/3})$), Algorithm 1 performs better than EXP3.S. Also, compared with previous bandit-over-bandit approach (BOB) Cheung et al., (2019) having a loose dependency on $T$ as $T^{3/4}$, our algorithm has a tighter regret bound with respect to $T$. Therefore, when $T$ is large (specifically $T=\omega(S^{6}K^{4})$), Algorithm 1 achieves a better regret bound than BOB.

However, the achieved regret bound from Algorithm 1 still has $O(T^{2/3})$ rather than $O(\sqrt{T})$ due to the large variance of loss estimators from sampling twice at each time for a base and an arm. In the following, we provide an algorithm utilizing adaptive learning rates to control the variance of estimators.

Here we propose Algorithm 2, which utilizes adaptive learning rates to control the variance of estimators. We first explain the base algorithm. For the base algorithm, we propose to use the negative entropy regularizer with adaptive learning rate $\eta_{t}(h)$ such that

The adaptive learning rate $\eta_{t}(h)$ is optimized using variance information for loss estimators at each time $t$ to control the variance such that

where $\rho_{t}(h)$ is a variance threshold term (to be specified later). This implies that if the variance of the estimators is small, then the learning rate becomes large.

For the master algorithm, we adopt the method of Corral Agarwal et al., (2017), in which, by using a log-barrier regularizer with increasing learning rates, it introduces a negative bias term to cancel a variance term from bases by handling the worst case with respect to $\rho_{t}(h^{\dagger})$. The log-barrier regularizer is defined as:

with learning rates $\bm{\xi}_{t}$ for the master algorithm.

Here we describe the update learning rates procedure for the master and bases in Algorithm 2; the other parts are similar with Algorithm 1. The variance of the loss estimator $l_{t}^{\prime}(h)$ for base $h$ is $1/p_{t+1}(h)$. If the variance $1/p_{t+1}(h)$ for base $h$ is larger than a threshold $\rho_{t}(h)$, then it increases learning rate as $\xi_{t+1}(h)=\gamma\xi_{t}(h)$ with $\gamma>1$ and the threshold is updated as $\rho_{t+1}(h)=2/p_{t+1}(h)$, which is also used for tuning the learning rate $\eta_{t}(h)$. Otherwise, it keeps the learning rate and threshold the same with the previous time step.

In the following theorem, we provide a regret bound of Algorithm 2.