Efficient and Optimal Fixed-Time Regret with Two Experts

Of course, a natural question is whether regret smaller than $\sqrt{(T/2)\ln n}$ is even possible as $T$ grows even if $n$ is fixed. Cover (1967) showed that for two experts (that is, for $n=2$) a regret of $\sqrt{T/2\pi}+O(1)$ is the best possible in the worst-case by showing an algorithm for the case with costs in $\{0,1\}$. In related work, the attainable worst-case regret bounds for three (Abbasi-Yadkori et al., 2017; Kobzar et al., 2020) and four (Bayraktar et al., 2020) experts were recently improved, respectively, to $\sqrt{8T/(9\pi)}+O(\ln T)$ and $\sqrt{T\pi/8}$ up to lower-order terms. Although optimal, Cover’s algorithm is based in a dynamic programming approach that takes $O(T^{2})$ time in total. In comparison, MWU takes $O(1)$ time per round to compute the probabilities to assign to the two experts at each round. Finally, one can adapt Cover’s algorithm for costs in $[0,1]$, but the standard approach is to randomly round to costs to either $0$ or $1$. In this case, the regret guarantees only hold in expectation.

In a related line of work, Harvey et al. (2020b) study the 2-experts problem in the anytime setting, that is, in the case where the player/algorithm does not know the total number of rounds $T$ ahead of time. They showed an optimal strategy for the player whose regret on any round $t\in\mathbb{N}$ is at most $(\gamma/2)\sqrt{t}$, where the constant $\gamma\approx 1.30693$ arises naturally in the study of Brownian motion (see Mörters and Peres, 2010 for an introduction to the field and historical references). Moreover, their algorithm can be computed (up to machine precision) in $O(1)$ time since it boils down to the evaluation of well-known mathematical functions such as the exponential function and the imaginary error function. In this work, we combine similar ideas based on stochastic calculus together with Cover’s algorithm to propose an efficient and optimal fixed-time algorithm for 2-experts.

In (1), the supremum ranges over all the possible adversaries for a game with $T$ rounds. However, we need only consider in the supremum oblivious adversaries (Karlin and Peres, 2017, Section 18.5.4), that is, adversaries $\mathcal{B}$ whose choice on each round depends only on the round number and not on the choices of the player. For any $\ell=(\ell_{1},\dotsc,\ell_{T})^{\mathsf{T}}\in(\mathbb{R}^{n})^{T}$, we denote by $\mathcal{B}_{\ell}$ the oblivious adversary that plays $\ell_{t}$ on round $t\in[T]$.

In fact, we may restrict our attention to even smaller sets of adversaries (for details on these reductions, see Gravin et al., 2016 and Karlin and Peres, 2017, Section 18.5.3). First, in (1) we need only to consider binary adversaries, that is, adversaries which can assign only costs in $\{0,1\}$ to the experts. Furthermore, to obtain the value of the optimal regret for two experts we only need to consider adversaries that pick vector costs in $\mathcal{L}\coloneqq\{(1,0)^{\mathsf{T}},(0,1)^{\mathsf{T}}\}$, which we call restricted binary adversaries. Intuitively, the adversary can do no better by placing equal costs on both experts at any given round. The optimal algorithm for two experts proposed by Cover (1967) heavily relies on the assumption that the adversary is a restricted binary one and does not extend to general costs in $[0,1]$ without resorting to randomly rounding the costs — which makes the regret guarantees hold only in expectation.

In this work we design an algorithm the suffers at most $\sqrt{T/(2\pi)}+O(1)$ regret for arbitrary $[0,1]$ costs. Our initial analysis handles only restricted binary adversaries, but simple concavity arguments extend the upper bound to general adversaries. Throughout this text we fix a time horizon $T\in\mathbb{N}$.

The case where we have only 2 experts admits a simplification that aids us greatly in the design of upper- and lower-bounds on the optimal regret. Namely, the gap (between experts) at round $t\in[T]$ is given by $\lvert L_{t}(1)-L_{t}(2)\rvert$, where $L_{t}$ is the cumulative loss vector at round $t$ as defined in Section 2. Furthermore, we denote by lagging expert (on round $t\in[T]$) an expert with maximum cumulative loss on round $t$ among both experts. Similarly, we denote by leading expert (on round $t\in[T]$) an expert with minimum cumulative loss on round $t$. The following proposition from Harvey et al. (2020b) shows that, for the restricted binary adversaries described earlier, the regret can be almost fully characterized by the expert gaps and the player’s choices of distributions on the experts. In the next proposition (and throughout the remainder of the text), for any predicate $P$ we define $[{P}]$ to be 1 if $P$ is true and $0$ otherwise.

Let us begin by further connecting Cover’s algorithm to random walks. Let $\mathcal{A}_{p}$ be a player strategy induced by some function $p\colon[T]\times\{0,\dotsc,T-1\}\to\mathbb{R}$. If $p(t,0)=1/2$ for all $t\in\{0,\dotsc,T-1\}$, then Proposition 2.1 tells us that, for any restricted binary adversary sequence of gaps $g_{1},\dotsc,g_{T}\in\mathbb{R}_{\geq 0}$ and for $g_{0}\coloneqq 0$ we have

The right-hand side of the above equation is a discrete analog of the Riemman-Stieltjes integral of $p$ with respect to $g$. In fact, if $(g_{t})_{t=0}^{T}$ is a random sequence³³3We usually also require some kind of restriction on $(g_{t})_{t=0}^{T}$, such as requiring it to be a martingale or a local martingale., the above is also known as a discrete stochastic integral. In particular, consider the case where $(g_{t})_{t=0}^{T}$ is a length $T$ reflected (i.e., absolute value of a) symmetric random walk. Then, any possible sequence of deterministic gaps has a positive probability of being realized by $(g_{t})_{t=0}^{T}$. In other words, any sequence of gaps is in the support of $(g_{t})_{t=0}^{T}$. Thus, bounding the worst-case regret of $\mathcal{A}_{p}$ is equivalent to bounding almost surely the value of (4) when $(g_{t})_{t=0}^{T}$ is a reflected symmetric random walk. This idea will prove itself powerful in continuous-time even though it is not very insightful for the discrete time problem.

A stochastic process that can be seen as the continuous-time analogue of symmetric random walks is Brownian motion (Revuz and Yor, 1999; Mörters and Peres, 2010). We fix a Brownian motion $(B_{t})_{t\geq 0}$ throughout the remainder of this text. Inspired by the observation that the discrete regret boils down to a discrete stochastic integral, Harvey et al. (2020b) define a continuous analogue of regret as a continuous stochastic integral. More specifically, given a function $p\colon[0,T)\times\mathbb{R}\to[0,1]$ such that $p(t,0)=1/2$ for all $t\in\mathbb{R}_{\geq 0}$, define the continuous regret at time $T$ by

where the term in the limit of the right-hand above is the stochastic integral (from $0$ to $T-\varepsilon$) of $p$ with respect to the process $(\lvert B_{t}\rvert)_{t\geq 0}$. We take the limit as a mere technicality: $p$ need not be defined at time $T$ and we want to ensure left-continuity of the continuous regret (the limit is well-defined since a stochastic integral with respect to a reflected Brownian motion is guaranteed to have limits from the left and to be continuous from the right). It is worth noting that usually stochastic integrals are defined with respect to martingales or local martingales, but $(\lvert B_{t}\rvert)_{t\geq 0}$ is neither. Still, $(\lvert B_{t}\rvert)_{t\geq 0}$ happens to be a semi-martingale, which roughly means that it can be written as a sum of two processes: a local-martingale and a process of bounded variation. In this case one can still define stochastic integrals in a way that foundational results such as Itô’s formula still hold and details can be found in Revuz and Yor (1999). We do not give the precise definition of a stochastic integral since we shall not deal with its definition directly. Still, one may think intuitively of such integrals as random Riemann-Stieltjes integrals, although the precise definition of stochastic integral is more delicate.

Let us now look for a continuous function $p\colon[0,T)\times\mathbb{R}\to[0,1]$ with $p(t,0)=1/2$ for all $t\geq 0$ with small continuous regret. Note that without the conditions of continuity or the requirement of $p(t,0)=1/2$ for $t\geq 0$, the problem would be trivial. If we did not require $p(t,0)=1/2$ for all $t\in[0,T)$, then taking $p(t,g)\coloneqq 0$ everywhere would yield $0$ continuous regret. Moreover, dropping this requirement would go against the analogous conditions needed in the discrete case, where regret could be written as a “discrete stochastic integral” on Proposition 2.1 only when the player chooses $(1/2,1/2)^{\mathsf{T}}$ in rounds with $0$ gap. Finally, requiring continuity of the $p$ is a way to avoid technicalities and “unfair” player strategies.

When working with Riemann integrals, instead of manipulating the definitions directly we use powerful and general results such as the Fundamental Theorem of Calculus (FTC). Analogously, the following result, known as Itô’s formula, is one of the main tools we use when manipulating stochastic integrals and which can be seen as an analogue of the FTC (and shows how stochastic integrals do not always follow the classical rules of calculus). We denote by $C^{1,2}$ the class of bivariate functions that are continuously differentiable with respect to their first argument and twice continuously differentiable with respect to their second argument. Moreover, for any function $f\in C^{1,2}$ we denote by $\partial_{t}f$ the partial derivative of $f$ with respect to its first argument, and we denote by $\partial_{g}f$ and $\partial_{gg}f$, respectively, the first and second derivatives of $f$ with respect to its second argument.

Note that the first integral in the equation of the above theorem resembles the definition of the continuous regret. In fact, the above result shows an alternative way to write the continuous regret at time $T$ of a function $p\colon[0,T)\times\mathbb{R}\to[0,1]$ such that there is $R\in C^{1,2}$ with $\partial_{g}R=p$. However, it might be hard to compute (or even to bound) the second integral on (5). A straightforward way to circumvent this problem is to look for functions such that the second integral in (5) is 0. For that, it suffices to consider functions $R\in C^{1,2}$ that satisfy the backwards heat equation on $[0,T)\times\mathbb{R}$, that is,

We summarize the above discussion and its implications in the following lemma.

In the remainder of this text we will make extensive use of a well-known function related to the Gaussian distribution known as complementary error function, defined by

In Section 4.4 we will show that the function $Q\colon(-\infty,T)\times\mathbb{R}\to[0,1]$ in $C^{1,2}$ given by

satisfies $\operatorname{ContRegret}(T,Q)=\sqrt{T/(2\pi)}$ almost surely. Before bounding the continuous regret, it is enlightening to see how $Q$ is related to Cover’s algorithm.

Specifically, let $p^{*}$ be as in (3). Due to the Central Limit Theorem, $Q$ can be seen as an approximation of $p^{*}$. To see why, let $(S_{t})_{t=0}^{\infty}$ be a symmetric random walk, and define $X_{t}\coloneqq S_{t}-S_{t-1}$ and $Y_{t}\coloneqq(X_{t}+1)/2$ for each $t\geq 1$. Note that $Y_{t}$ follows a Bernoulli distribution with parameter $1/2$ for any $t\geq 1$. Moreover, let $Z$ be a Gaussian random variable with mean 0 and variance 1. Then, by setting $\mu\coloneqq\bm{\mathrm{E}}[2Y_{1}]=1$ and $\sigma^{2}\coloneqq\bm{\mathrm{E}}[(2Y_{1}-\mu)^{2}]=1$, the Central Limit Theorem guarantees

where the limit holds in distribution. Thus, we roughly have that $S_{L}$ and $\sqrt{L}Z$ have similar distributions. Then,

One may already presume that using $Q$ in place of $p^{*}$ in the discrete experts’ problem should yield a regret bound close enough to the guarantees on the regret of Cover’s algorithm. Indeed, using Berry-Esseen’s Theorem (Durrett, 2019, Section 3.4.4) to more precisely bound the difference between $p^{*}$ and $Q$ yields a $O(\sqrt{T})$ regret bound with suboptimal constants against binary adversaries. However, it is not clear if the approximation error would yield the optimal constant in the regret bound. Additionally, these guarantees do not naturally extend to arbitrary experts’ costs in $[0,1]$. In Section 5 we will show how to use an algorithm closely related to $Q$ that enjoys a clean bound on the discrete-time regret.

Interestingly, not only is $Q$ in $C^{1,2}$, but it also satisfies the backwards heat equation, even though we have never explicitly required such a condition to hold. Since the proof of this fact boils down to technical but otherwise straightforward computations, we defer it to Appendix C.

However, recall that to use Lemma 4.2 we need a function $R\in C^{1,2}$ with $\partial_{g}R=Q$ that satisfies the backwards heat equation, not necessarily $Q$ itself needs to satisfy the backwards heat equation. Luckily enough, the following lemma shows how to obtain such a function $R$ based on $Q$.

In light of the above proposition, for all $(t,g)\in(-\infty,T)\times\mathbb{R}$ define

In the case above, we can evaluate these integrals and obtain a formula for $R$ that is easier to analyze. Although we defer a complete proof of the next equation to Appendix C, using that $\int_{0}^{y}\operatorname*{erfc}(x)\mathop{}\!\mathrm{d}x=y\operatorname*{erfc}(y)-\frac{1}{\sqrt{\pi}}e^{-y^{2}}+\frac{1}{\sqrt{\pi}}$ (Olver et al., 2010, Section 7.7(i)) and that $\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}x}\operatorname*{erfc}(x)=-\frac{2}{\sqrt{\pi}}e^{-x^{2}}$ we can show for every $t\in(-\infty,T)$ and $g\in\mathbb{R}$ that

Since $R$ satisfies (BHE), Lemma 4.2 shows the continuous regret of $Q$ is given exactly by $R$. The following lemma shows a bound on $R$ and, thus, a bound on the continuous regret of $Q$.

Combining these results we get the desired bound on the continuous regret of $Q$, which we summarize in the following theorem.

In Section 4, the main tool to relate the continuous regret to the function $R$ was Itô’s formula. Similarly, one of the main tools for the analysis of the discretized continuous-time algorithm will be a discrete version of Itô’s formula. In order to state such a formula and to describe the algorithm, some standard notation to denote discrete derivative will be useful. Namely, for any function $f\colon\mathbb{R}^{2}\to\mathbb{R}$ and any $t,g\in\mathbb{R}$, define

We are now in place to state a discrete analogue of Itô’s formula. One important assumption of the next theorem is that $g_{0},\dotsc,g_{T}\in\mathbb{R}$ are such that successive values have absolute difference equal to $1$. In the case where $g_{0},\dotsc,g_{T}$ are gaps in a 2-experts problem, this means that the adversary needs to be a restricted binary adversary. The version of the next theorem as stated — including the dependence on $t$ — can be found in Harvey et al. (2020b, Lemma 3.7). Yet, this theorem is a slight generalization of earlier results such as the ones due to Fujita (2008, Section 2) and Kudzhma (1982, Theorem 2)

The first summation in the right-hand side of discrete Itô’s formula can be seen as a discrete stochastic integral when $(g_{t})_{t=0}^{T}$ is a random sequence. Remarkably, this term is extremely similar to the regret formula from Proposition 2.1. Thus, if we were to use discrete Itô’s formula to bound the regret, it would be desirable for the second term to (approximately) satisfy an analogue of (BHE). In fact, the potential $V^{*}$ from Cover’s algorithm satisfies the discrete BHE (with some care needed when the gap is zero, see Appendix A.3). Furthermore, the connection between BHE seems to extend to other problems in online learning: in recent work, Zhang et al. (2022) showed how coin-betting with potentials that satisfy the BHE yield optimal algorithms for unconstrained online learning.

Since $R$ satisfies (BHE), one might hope that $R$ would also satisfy such a discrete backwards-heat inequality, yielding an upper-bound on the regret of the strategy given by $R_{g}$. In the work of Harvey et al. (2020b) in the anytime setting, it was the case that the terms in the second sum were non-negative, which in a sense means that the discretized algorithm suffers negative discretization error. In the fixed-time setting we are not as lucky.

Based on the discussion at the beginning of this section, a natural way to discretize the algorithm from Section 4 is to define the function $q\colon[T]\times\{0,\dotsc,T-1\}\to\mathbb{R}$ by

where we need to treat the case at the very last step differently since $R$ is not defined on $\{T\}\times\mathbb{R}$. It is not clear from its definition, but we indeed have $q(t,0)=1/2$ for all $t\in[T]$. We defer the (relatively technical) proof of the next result to Appendix D.

Our goal now is to combine Proposition 2.1 and the discrete Itô’s formula to bound the regret of $\mathcal{A}_{q}$. Since $R$ satisfies the (BHE), one might hope that $R$ is close to satisfying the discrete version of this equation. To formalize this idea, for all $t\in(-\infty,T)$ and $g\in\mathbb{R}$ define

The above terms measure how well the first derivative with respect to the first variable and the second derivative with respect to the second variable are each approximated by their discrete analogues. That is, these are basically the discretization errors on the derivatives of $R$. Then, combining the fact that $R$ satisfies (BHE) together with Proposition 2.1 yields the following theorem.

In light of Theorem 5.3, it suffices to bound the accumulated discretization error of the derivatives to obtain potentially good bounds on the regret of $\mathcal{A}_{q}$. The next two lemmas show that both $r_{t}(t,g)$ and $r_{gg}(t,g)$ are in $O((T-t)^{-3/2})$. Since

this will show that $\mathcal{A}_{q}$ suffers at most $\sqrt{T/2\pi}+O(1)$ regret.⁶⁶6This together with Prop. 2.1 also shows that the difference between in the regret of $\mathcal{A}_{q}$ and $\mathcal{A}_{Q}$ is in $O(1)$. Since the proof of these bounds are relatively technical but otherwise not considerably insightful, we defer them to Appendix D.

Combining the above lemmas together with Theorem 5.3 yields the following regret bound.