Online Learning with Optimism and Delay

Online learning is a sequential decision-making paradigm in which a learner is pitted against a potentially adversarial environment (Shalev-Shwartz, 2007; Orabona, 2019). At time $t$, the learner must select a play $\mathbf{w}_{t}$ from some set of possible plays $\mathbf{W}$. The environment then reveals the loss function $\ell_{t}$ and the learner pays the cost $\ell_{t}(\mathbf{w}_{t})$. The learner uses information collected in previous rounds to improve its plays in subsequent rounds. Optimistic online learners additionally make use of side-information or “hints” about expected future losses to improve their plays. Over a period of length $T$, the goal of the learner is to minimize regret, an objective that quantifies the performance gap between the learner and the best possible constant play in retrospect in some competitor set $\mathbf{U}$: $\textup{Regret}_{T}=\sup_{\mathbf{u}\in\mathbf{U}}\sum_{t=1}^{T}\ell_{t}(\mathbf{w}_{t})-\ell_{t}(\mathbf{u})$. Adversarial online learning algorithms provide robust performance in many complex real-world online prediction problems such as climate or weather forecasting.

In traditional online learning paradigms, the loss for round $t$ is revealed to the learner immediately at the end of round $t$. However, many real-world applications produce delayed feedback, i.e., the loss for round $t$ is not available until round $t+D$ for some delay period $D.$¹¹1Our initial presentation will assume constant delay $D$, but we provide extensions to variable and unbounded delays in App. O. Existing delayed online learning algorithms achieve optimal worst-case regret rates against adversarial loss sequences, but each has drawbacks when deployed for real applications with short horizons $T$. Some use only a small fraction of the data to train each learner (Weinberger & Ordentlich, 2002; Joulani et al., 2013); others tune their parameters using uniform bounds on future gradients that are often challenging to obtain or overly conservative in applications (McMahan & Streeter, 2014; Quanrud & Khashabi, 2015; Joulani et al., 2016; Korotin et al., 2020; Hsieh et al., 2020). Only the concurrent work of Hsieh et al. (2020, Thm. 13) can make use of optimistic hints and only for the special case of unconstrained online gradient descent.

In this work, we aim to develop robust and practical algorithms for real-world delayed online learning. To this end, we introduce three novel algorithms—DORM, DORM+, and AdaHedgeD—that use every observation to train the learner, have no parameters to tune, exhibit optimal worst-case regret rates under delay, and enjoy improved performance when accurate hints for unobserved losses are available. We begin by formulating delayed online learning as a special case of optimistic online learning and use this “delay-as-optimism” perspective to develop:

1. 1.

   A formal reduction of delayed online learning to optimistic online learning (Lems. 1 and 2),

2. 2.

   The first optimistic tuning-free and self-tuning algorithms with optimal regret guarantees under delay (DORM, DORM+, and AdaHedgeD),

3. 3.

   A tightening of standard optimistic online learning regret bounds that reveals the robustness of optimistic algorithms to inaccurate hints (Thms. 3 and 4),

4. 4.

   The first general analysis of follow-the-regularized-leader (Thms. 5 and 10) and online mirror descent algorithms (Thm. 6) with optimism and delay, and

5. 5.

   The first meta-algorithm for learning a low-regret optimism strategy under delay (Thm. 13).

We validate our algorithms on the problem of subseasonal forecasting in Sec. 7. Subseasonal forecasting—predicting precipitation and temperature 2-6 weeks in advance—is a crucial task for allocating water resources and preparing for weather extremes (White et al., 2017). Subseasonal forecasting presents several challenges for online learning algorithms. First, real-time subseasonal forecasting suffers from delayed feedback: multiple forecasts are issued before receiving feedback on the first. Second, the regret horizons are short: a common evaluation period for semimonthly forecasting is one year, resulting in 26 total forecasts. Third, forecasters cannot have difficult-to-tune parameters in real-time, practical deployments. We demonstrate that our algorithms DORM, DORM+, and AdaHedgeD sucessfully overcome these challenges and achieve consistently low regret compared to the best forecasting models.

Our Python library for Optimistic Online Learning under Delay (PoolD) and experiment code are available at
https://github.com/geflaspohler/poold.

Notation  For integers $a,b$, we use the shorthand $[b]\triangleq\{1,\dots,b\}$ and $\mathbf{g}_{a:b}\triangleq\sum_{i=a}^{b}\mathbf{g}_{i}$. We say a function $f$ is proper if it is somewhere finite and never $-\infty$. We let $\partial f(\mathbf{w})=\{\mathbf{g}\in\mathbb{R}^{d}:f(\mathbf{u})\geq f(\mathbf{w})+\langle\mathbf{g},\mathbf{u}-\mathbf{w}\rangle,\ \forall\mathbf{u}\in\mathbb{R}^{d}\}$ denote the set of subgradients of $f$ at $\mathbf{w}\in\mathbb{R}^{d}$ and say $f$ is $\mu$-strongly convex over a convex set $\mathbf{W}\subseteq\mathop{\mathrm{int}}\mathop{\mathrm{dom}}f$ with respect to $\|{\cdot}\|$ with dual norm $\|{\cdot}\|_{*}$ if $\forall\mathbf{w},\mathbf{u}\in\mathbf{W}$ and $\mathbf{g}\in\partial f(\mathbf{w})$, we have $f(\mathbf{u})\geq f(\mathbf{w})+\langle\mathbf{g},\mathbf{u}-\mathbf{w}\rangle+\frac{\mu}{2}\|\mathbf{w}-\mathbf{u}\|^{2}$. For differentiable $\psi$, we define the Bregman divergence $\mathcal{B}_{\psi}(\mathbf{w},\mathbf{u})\triangleq\psi(\mathbf{w})-\psi(\mathbf{u})-\langle{\nabla\psi(\mathbf{u})},{\mathbf{w}-\mathbf{u}}\rangle$. We define $\operatorname{diam}({\mathbf{W}})=\inf_{\mathbf{w},\mathbf{w}^{\prime}\in\mathbf{W}}\|{\mathbf{w}-\mathbf{w}^{\prime}}\|$, $(r)_{+}\triangleq\max(r,0)$, and $\min(r,s)_{+}\triangleq(\min(r,s))_{+}$.

Standard online learning algorithms, such as follow the regularized leader (FTRL) and online mirror descent (OMD) achieve optimal worst-case regret against adversarial loss sequences (Orabona, 2019). However, many loss sequences encountered in applications are not truly adversarial. Optimistic online learning algorithms aim to improve performance when loss sequences are partially predictable, while remaining robust to adversarial sequences (see, e.g., Azoury & Warmuth, 2001; Chiang et al., 2012; Rakhlin & Sridharan, 2013b; Steinhardt & Liang, 2014). In optimistic online learning, the learner is provided with a “hint” in the form of a pseudo-loss $\tilde{\ell}_{t}$ at the start of round $t$ that represents a guess for the true unknown loss. The online learner can incorporate this hint before making play $\mathbf{w}_{t}$.

In standard formulations of optimistic online learning, the convex pseudo-loss $\tilde{\ell}_{t}(\mathbf{w}_{t})$ is added to the standard FTRL or OMD regularized objective function and leads to optimistic variants of these algorithms: optimistic FTRL (OFTRL, Rakhlin & Sridharan, 2013a) and single-step optimistic OMD (SOOMD, Joulani et al., 2017, Sec. 7.2). Let $\tilde{\mathbf{g}}_{t}\in\partial\tilde{\ell}_{t}(\mathbf{w}_{t-1})$ and $\mathbf{g}_{t}\in\partial\ell_{t}(\mathbf{w}_{t})$ denote subgradients of the pseudo-loss and true loss respectively. The inclusion of an optimistic hint leads to the following linearized update rules for play $\mathbf{w}_{t+1}$:

where $\tilde{\mathbf{g}}_{t+1}\in\mathbb{R}^{d}$ is the hint subgradient, $\lambda\geq 0$ is a regularization parameter, and $\psi$ is proper regularization function that is $1$-strongly convex with respect to a norm $\|{\cdot}\|$. The optimistic learner enjoys reduced regret whenever the hinting error $\|{\mathbf{g}_{t+1}-\tilde{\mathbf{g}}_{t+1}}\|_{*}$ is small (Rakhlin & Sridharan, 2013a; Joulani et al., 2017). Common choices of optimistic hints include the last observed subgradient or average of previously observed subgradients (Rakhlin & Sridharan, 2013a). We note that the standard FTRL and OMD updates can be recovered by setting the optimistic hints to zero.

In the delayed feedback setting with constant delay of length $D$, the learner only observes $(\ell_{i})_{i=1}^{t-D}$ before making play $\mathbf{w}_{t+1}$. In this setting, we propose counterparts of the OFTRL and SOOMD online learning algorithms, which we call optimistic delayed FTRL (ODFTRL) and delayed optimistic online mirror descent (DOOMD) respectively:

for hint vector $\mathbf{h}_{t+1}$. Our use of the notation $\mathbf{h}_{t+1}$ instead of $\tilde{\mathbf{g}}_{t+1}$ for the optimistic hint here is suggestive. Our regret analysis in Thms. 5 and 6 reveals that, instead of hinting only for the “future“ missing loss $\mathbf{g}_{t+1}$, delayed online learners should uses hints $\mathbf{h}_{t}$ that guess at the summed subgradients of all delayed and future losses: $\mathbf{h}_{t}=\sum_{s=t-D}^{t}\tilde{\mathbf{g}}_{s}$.

Regret matching (RM) (Blackwell, 1956; Hart & Mas-Colell, 2000) and regret matching+ (RM+) (Tammelin et al., 2015) are online learning algorithms that have strong empirical performance. RM was developed to find correlated equilibria in two-player games and is commonly used to minimize regret over the simplex. RM+ is a modification of RM designed to accelerate convergence and used to effectively solve the game of Heads-up Limit Texas Hold’em poker (Bowling et al., 2015). RM and RM+ support neither optimistic hints nor delayed feedback, and known regret bounds have a suboptimal scaling with respect to the problem dimension $d$ (Cesa-Bianchi & Lugosi, 2006; Orabona & Pál, 2015). To extend these algorithms to the delayed and optimistic setting and recover the optimal regret rate, we introduce our generalizations, delayed optimistic regret matching (DORM)

and delayed optimistic regret matching+ (DORM+)

Each algorithm makes use of an instantaneous regret vector $\mathbf{r}_{t}\triangleq\mathbf{1}\langle{\mathbf{g}_{t}},{\mathbf{w}_{t}}\rangle-\mathbf{g}_{t}$ that quantifies the relative performance of each expert with respect to the play $\mathbf{w}_{t}$ and the linearized loss subgradient $\mathbf{g}_{t}$. The updates also include a parameter $q\geq 2$ and its conjugate exponent $p=q/(q-1)$ that is set to recover the minimax optimal scaling of regret with the number of experts (see Cor. 9). We note that DORM and DORM+ recover the standard RM and RM+ algorithms when $D=0$, $\lambda=1$, $q=2$, and $\mathbf{h}_{t}=\mathbf{0},\ \forall t$.

In this section, we analyze an adaptive version of ODFTRL with time-varying regularization $\lambda_{t}\psi$ and develop strategies for setting $\lambda_{t}$ appropriately in the presence of optimism and delay. We begin with a new general regret analysis of optimistic delayed adaptive FTRL (ODAFTRL)

where $\mathbf{h}_{t+1}\in\mathbb{R}^{d}$ is an arbitrary hint vector revealed before $\mathbf{w}_{t+1}$ is generated, $\psi$ is $1$-strongly convex with respect to a norm $\|{\cdot}\|$, and $\lambda_{t}\geq 0$ is a regularization parameter.

The proof of this result in App. G builds on a new regret bound for undelayed optimistic adaptive FTRL (OAFTRL). In the absence of delay ($D=0$), Thm. 10 strictly improves existing regret bounds (Rakhlin & Sridharan, 2013a; Mohri & Yang, 2016; Joulani et al., 2017) for OAFTRL by providing tighter guarantees whenever the hinting error $\|{\mathbf{h}_{t}-\sum_{s=t-D}^{t}\mathbf{g}_{t}}\|_{*}$ is larger than the subgradient magnitude $\|{\mathbf{g}_{t}}\|_{*}$. In the presence of delay, Thm. 10 benefits both from robustness to hinting error in the worst case and the ability to exploit accurate hints in the best case. The bounded-domain factors $\mathbf{a}_{t,F}$ strengthen both standard OAFTRL regret bounds and the concurrent bound of Hsieh et al. (2020, Thm. 1) when $\operatorname{diam}({\mathbf{W}})$ is small and will enable us to design practical $\lambda_{t}$-tuning strategies under delay without any prior knowledge of unobserved subgradients. We now turn to these self-tuning protocols.

As we have seen, optimistic hints play an important role in online learning under delay: effective hinting can counteract the increase in regret under delay. In this section, we consider the problem of choosing amongst several competing hinting strategies. We show that this problem can again be treated as a delayed online learning problem. In the following, we will call the original online learning problem the “base problem” and the learning-to-hint problem the “hinting problem.”

Suppose that, at time $t$, we observe the hints $\tilde{\mathbf{g}}_{t}$ of $m$ different hinters arranged into a $d\times m$ matrix $H_{t}$. Each column of $H_{t}$ is one hinter’s best estimate of the sum of missing loss subgradients $\mathbf{g}_{t-D:t}$. Our aim is to output a sequence of combined hints $\mathbf{h}_{t}(\omega_{t})\triangleq H_{t}\omega_{t}$ with low regret relative to the best constant combination strategy $\omega\in\Omega\triangleq\triangle_{m-1}$ in hindsight. To achieve this using delayed online learning, we make use of a convex loss function $l_{t}(\omega)$ for the hint learner that upper bounds the base learner regret.

As we detail in App. K, Assump. 1 holds for all of the learning algorithms introduced in this paper. For example, by Cor. 9, if the base learner is DORM, we may choose $C_{0}(\mathbf{u})=0$, $C_{1}(\mathbf{u})=\sqrt{\frac{\|{\mathbf{u}}\|_{p}^{2}}{2(p-1)}},$ and the $\mathcal{O}(D+1)$ convex function $f_{t}(\mathbf{h}_{t})=\|{\mathbf{r}_{t}}\|_{q}\|{\mathbf{h}_{t}-\sum_{s=t-D}^{t}\mathbf{r}_{s}}\|_{q}\geq\mathbf{b}_{t,q}$.³³3The alternative choice $f_{t}(\mathbf{h}_{t})=\frac{1}{2}\|{\mathbf{h}_{t}-\sum_{s=t-D}^{t}\mathbf{r}_{s}}\|^{2}_{q}$ also bounds regret but may have size $\Theta((D+1)^{2})$.

For any base learner satisfying Assump. 1, we choose $l_{t}(\omega)=f_{t}(H_{t}\omega)$ as our hinting loss, use the tuning-free DORM+ algorithm to output the combination weights $\omega_{t}$ on each round, and provide the hint $\mathbf{h}_{t}(\omega_{t})=H_{t}\omega_{t}$ to the base learner. The following result, proved in App. J, shows that this learning to hint strategy performs nearly as well as the best constant hint combination strategy in restrospect.

To quantify the size of this regret bound, consider again the DORM base learner with $f_{t}(\mathbf{h}_{t})=\|{\mathbf{r}_{t}}\|_{q}\|{\mathbf{h}_{t}-\sum_{s=t-D}^{t}\mathbf{r}_{s}}\|_{q}$. By Lem. 26 in App. K, $\|{\gamma_{t}}\|_{\infty}\leq d^{1/q}\|{H_{t}}\|_{\infty}\|{\mathbf{r}_{t}}\|_{q}$ for $\|{H_{t}}\|_{\infty}$ the maximum absolute entry of $H_{t}$. Each column of $H_{t}$ is a sum $D+1$ subgradient hints, so $\|{H_{t}}\|_{\infty}$ is $\mathcal{O}(D+1)$. Thus, for this choice of hinter loss, the $\textup{huber}(\xi_{t},\zeta_{t})$ term is $\mathcal{O}((D+1)^{3})$, and the hint learner suffers only $\mathcal{O}(T^{1/4}(D+1)^{3/4})$ additional regret from learning to hint. Notably, this additive regret penalty is $\mathcal{O}(\sqrt{(D+1)T})$ if $D=\mathcal{O}(T)$ (and $o(\sqrt{(D+1)T})$ when $D=o(T)$), so the learning to hint strategy of Thm. 13 preserves minimax optimal regret rates.

Related work  Rakhlin & Sridharan (2013a, Sec. 4.1) propose and analyze a method to learn optimism strategies for a two-step OMD base learner. Unlike Thm. 13, the approach does not accommodate delay, and the analyzed regret is only with respect to single hinting strategies $\omega\in\{\mathbf{e}_{j}\}_{j\in[m]}$ rather than combination strategies, $\omega\in\triangle_{m-1}$.

We now apply the online learning techniques developed in this paper to the problem of adaptive ensembling for subseasonal forecasting. Our experiments are based on the subseasonal forecasting data of Flaspohler et al. (2021) that provides the forecasts of $d=6$ machine learning and physics-based models for both temperature and precipitation at two forecast horizons: 3-4 weeks and 5-6 weeks. In operational subseasonal forecasting, feedback is delayed; models make $D=2$ or $3$ forecasts (depending on the forecast horizon) before receiving feedback. We use delayed, optimistic online learning to play a time-varying convex combination of input models and compete with the best input model over a year-long prediction period ($T=26$ semimonthly dates). The loss function is the geographic root-mean squared error (RMSE) across $514$ locations in the Western United States.

We evaluate the relative merits of the delayed online learning techniques presented by computing yearly regret and mean RMSE for the ensemble plays made by the online leaner in each year from 2011-2020. Unless otherwise specified, all online learning algorithms use the recent_g hint $\tilde{\mathbf{g}}_{s}$, which approximates each unobserved subgradient at time $t$ with the most recent observed subgradient $\mathbf{g}_{t-D-1}$. See App. L for full experimental details, App. N for algorithmic details, and App. M for extended experimental results.

Competing with the best input model  The primary benefit of online learning in this setting is its ability to achieve small average regret, i.e., to perform nearly as well as the best input model in the competitor set $\mathbf{U}$ without knowing which is best in advance. We run our three delayed online learners—DORM, DORM+, and AdaHedgeD—on all four subseasonal prediction tasks and measure their average loss.

The average yearly RMSE for the three online learning algorithms and the six input models is shown in Table 1. The DORM+ algorithm tracks the performance of the best input model for all tasks except Temp. 5-6w. All online learning algorithms achieve negative regret for both precipitation tasks. Fig. 1 shows the yearly cumulative regret (in terms of the RMSE loss) of the online learning algorithms over the $10$-year evaluation period. There are several years (e.g., 2012, 2014, 2020) in which all online learning algorithms substantially outperform the best input forecasting model. The consistently low regret year-to-year of DORM+ compared to DORM and AdaHedgeD makes it a promising candidate for real-world delayed subseasonal forecasting. Notably, RM+ (a special case of DORM+) is known to have small tracking regret, i.e., it competes well even with strategies that switch between input models a bounded number of times (Tammelin et al., 2015, Thm. 2). We suspect that this is one source of DORM+’s superior performance. We also note that the self-tuned AdaHedgeD performs comparably to the the optimally-tuned DORM, demonstrating the effectiveness of our self-tuning strategy.

Impact of regularization  We evaluate the impact of the three regularization strategies developed in this paper: 1) the upper bound DUB strategy, 2) the tighter AdaHedgeD strategy, and 3) the DORM+ algorithm that is tuning-free. This tuning-free property has evident practical benefits, as this section demonstrates.

Fig. 2 shows the yearly regret of the DUB, AdaHedgeD, and DORM+ algorithms. A consistent pattern appears in the yearly regret: DUB has moderate positive regret, AdaHedgeD has both the largest positive and negative regret values, and DORM+ sits between these two extremes. If we examine the weights played by each algorithm (Fig. 3), the weights of DUB and AdaHedgeD appear respectively over- and under-regularized compared to DORM+ (the top model for this task). DUB’s use of the upper bound $\mathbf{b}_{t,F}$ results in a very large regularization setting ($\lambda_{T}=142.881$) and a virtually uniform weight setting. AdaHedgeD’s tighter bound $\delta_{t}$ produces a value for $\lambda_{T}=3.005$ that is two orders of magnitude smaller. However, in this short-horizon forecasting setting, AdaHedgeD’s aggressive plays result in higher average RMSE. By nature of it’s $\lambda_{t}$-free updates, DORM+ produces more moderately regularized plays $\mathbf{w}_{t}$ and negative regret.

To replicate or not to replicate  In this section, we compare the performance of replicated and non-replicated variants of our DORM+ algorithm. Both algorithms perform well (see Sec. M.3), but in all tasks, DORM+ outperforms replicated DORM+ (in which $D+1$ independent copies of DORM+ make staggered predictions). Fig. 4 provides an example of the weight plots produced by the replication strategy in the Temp. 5-6w task with $D=3$. The separate nature of the replicated learner’s plays is evident in the weight plots and leads to an average RMSE of $2.315$, versus $2.303$ for DORM+ in the Temp. 5-6w task.

Learning to hint  Finally, we examine the effect of optimism on the DORM+ algorithms and the ability of our “learning to hint” strategy to recover the performance of the best optimism strategy in retrospect. Following the hint construction protocol in Sec. N.2, we run the DORM+ base algorithm with $m=4$ subgradient hinting strategies: $\tilde{\mathbf{g}}_{s}=\mathbf{g}_{t-D-1}$ (recent_g), $\tilde{\mathbf{g}}_{s}=\mathbf{g}_{s-D-1}$ (prev_g), $\tilde{\mathbf{g}}_{s}=\frac{D+1}{t-D-1}\mathbf{g}_{1:t-D-1}$ (mean_g), or $\tilde{\mathbf{g}}_{s}=\mathbf{0}$ (none). We also use DORM+ as the meta-algorithm for hint learning to produce the learned optimism strategy that plays a convex combination of the four hinters. In Fig. 5, we first note that several optimism strategies outperform the none hinter, confirming the value of optimism in reducing regret. The learned variant of DORM+ avoids the worst-case performance of the individual hinters in any given year (e.g., 2015), while staying competitive with the best strategy (although it does not outperform the dominant recent_g strategy overall). We believe the performance of the online hinter could be further improved by developing tighter convex bounds on the regret of the base problem in the spirit of Assump. 1.