Beyond $\mathcal{O}(\sqrt{T})$ Regret: Decoupling Learning and Decision-making in Online Linear Programming

• •

  We show that first-order methods achieve $o(\sqrt{T})$ regret under weaker assumptions than LP-based methods. In particular, we identify a dual error bound condition that is sufficient to guarantee lower regret of first-order methods when the dual LP problem has a unique optimal solution. In the continuous-support setting, we establish an $\mathcal{O}(T^{1/3})$ regret result under weaker assumptions than the existing methods; In the finite-support setting, we establish an $\mathcal{O}(\log T)$ result, which significantly improves on the state-of-the-art $\mathcal{O}(T^{3/8})$ result and almost matches the $\mathcal{O}(1)$ regret of LP-based methods. For problems with $\gamma$-Hölder growth condition, we establish a general ${\mathcal{O}(T^{\frac{\gamma-1}{2\gamma-1}}\log T)}$ regret result, which interpolates among the settings of no growth ($\gamma=\infty$), continuous support with quadratic growth ($\gamma=2$) and finite support with sharpness ($\gamma=1$). Our results show that first-order methods perform well under strictly weaker conditions than LP-based methods and still achieve competitive performance, significantly advancing their applicability in practice.

• •

  We design a general exploration-exploitation framework to exploit the dual error bound condition. The idea is to learn a good approximation of the distribution dual optimal solution. Then, the online decision-making algorithm can be localized around a neighborhood of the approximate dual solution and makes decisions in an effective domain of size $o(1)$, thereby achieving improved regret guarantees. We reveal an important dilemma in simultaneously using a single first-order method as both learning and decision-making algorithms: a good learning algorithm can perform poorly in decision-making. This dilemma implies an important discrepancy between stochastic optimization and online decision-making, and it is addressed by decoupling learning and decision-making: two different first-order methods are adopted for learning and decision-making. This simple idea yields a highly flexible framework for online sequential decision-making. Our analysis can be of independent interest in the broader context of online convex optimization.

An online resource allocation problem with linear inventory and rewards can be modeled as an OLP problem: given time horizon $T\geq 1$ and $m\geq 1$ resources represented by $\mathbf{b}\in\mathbb{R}^{m}_{+}$, at time $t$, a customer with $(c_{t},\mathbf{a}_{t})\in\mathbb{R}\times\mathbb{R}^{m}$ arrives and requests resources $\mathbf{a}_{t}$ at price $c_{t}$. Decision $x^{t}\in[0,1]$ is made to (partially) accept the order or reject it. With compact notation $\mathbf{c}=(c_{1},\ldots,c_{T})^{\top}\in\mathbb{R}^{T}$ and $\mathbf{A}:=(\mathbf{a}_{1},\ldots,\mathbf{a}_{T})\in\mathbb{R}^{m\times T}$, the problem can be written as

where $\mathbf{0}$ and $\mathbf{1}$ are vectors of all zeros and ones. The dual problem

can be transformed into the following finite sum form

where $\mathbf{d}=T^{-1}\mathbf{b}$ is the average resource. When $(c_{t},\mathbf{a}_{t})$ are i.i.d. distributed, $f_{T}(\mathbf{y})$ can be viewed as a sample approximation of the expected dual function $f(\mathbf{y})$, where

Define the sets of dual optimal solutions, and let $\mathbf{y}_{T}^{\star},\mathbf{y}^{\star}$ be some dual optimal solutions, respectively:

and we can determine the primal optimal solution $\mathbf{x}_{T}^{\star}=(x_{1}^{\star},\ldots,x_{T}^{\star})\in\mathbb{R}^{T}$ by the LP optimality conditions:

This connection between primal and dual solutions motivates dual-based online learning algorithms: a dual-based learning algorithm maintains a dual sequence $\{\mathbf{y}^{t}\}_{t=1}^{T}$ in the learning process, while primal decisions are made based on the optimality condition. Given the sample approximation interpretation, first-order methods are natural candidate learning algorithms.

Given online algorithm output $\hat{\mathbf{x}}_{T}=(x^{1},\ldots,x^{T})$, its regret and constraint violation are defined as

These metrics are widely used in the OLP literature [18, 10].

We make the following assumptions throughout the paper.

1. A1.

   (Stochastic input) $\{(c_{t},\mathbf{a}_{t})\}_{t=1}^{T}$ are generated i.i.d. from some distribution $\mathcal{P}$.

2. A2.

   (Bounded data) There exist constants $\bar{a},\bar{c}>0$ such that $\|\mathbf{a}\|_{\infty}\leq\bar{a}$ and $|c|\leq\bar{c}$ almost surely.

3. A3.

   (Linear resource) The average resource $\mathbf{d}=T^{-1}\mathbf{b}$ satisfies $\underline{d}\cdot\mathbf{1}\leq\mathbf{d}\leq\bar{d}\cdot\mathbf{1}$, where $0<\underline{d}\leq\bar{d}$.

A1 to A3 are standard and minimal in the OLP literature [4, 18, 10], and it is known that online subgradient method (Algorithm 1) with constant stepsize $\alpha_{t}\equiv\mathcal{O}{(1/\sqrt{T})}$ achieves $\mathcal{O}(\sqrt{T})$ regret.

Theorem 2.1 will be used as a benchmark for our results. Under A1 to A3, $\mathcal{O}(\sqrt{T})$ regret has been shown to achieve the lower bound [2]. Under further assumptions such as non-degeneracy, LP-based OLP algorithms can efficiently leverage this structure to achieve $\mathcal{O}(\log T)$ and $\mathcal{O}(1)$ regret, respectively, in the continuous [19] and finite support settings [9]. However, how first-order methods can efficiently exploit these structures remains less explored. This paper establishes a new online learning framework to resolve this issue. In particular, we consider the $\gamma$-error bound condition from the optimization literature and summarize our main results below:

It turns out the dual error bound is key to improved regret for first-order methods. In the next section, we formally define the dual error bound condition and introduce its consequences.

Our key assumption, also known in the literature as the Hölder error bound condition [14], is stated as follows.

1. A4.

   (Dual error bound) $f(\mathbf{y})-f(\mathbf{y}^{\star})\geq\mu\cdot\mathrm{dist}(\mathbf{y},\mathcal{Y}^{\star})^{\gamma}$ for all $\mathbf{y}\in\mathcal{Y}=\{\mathbf{y}:\mathbf{y}\geq\mathbf{0},\|\mathbf{y}\|\leq\tfrac{\bar{c}+\underline{d}}{\underline{d}}\},\gamma\in[1,\infty)$.

The assumption A4 states a growth condition in terms of the expected dual function: as $\mathbf{y}$ leaves the distribution dual optimal set $\mathcal{Y}^{\star}$, the objective will grow at least at rate $\mathrm{dist}(\mathbf{y},\mathcal{Y}^{\star})^{\gamma}$. It is implied by the assumptions used in the analysis of LP-based OLP algorithms, which we summarize below.

We leave the detailed verification of the results in the appendix Section A.3.

While A4 is implied by the non-degeneracy assumptions in the literature, A4 does not rule out degenerate LPs. An LP can be degenerate but still satisfy the error bound. Therefore, A4 is weaker than the existing assumptions in the OLP literature. The error bound has several important consequences on our algorithm design, which we summarize next.

Lemma 3.1 shows that there is an efficient learning algorithm (in particular, Algorithm 4 in the appendix) that learns an approximate dual optimal solution $\hat{\mathbf{y}}$ with suboptimality $\varepsilon$ at sample complexity $\mathcal{O}(\varepsilon^{-2(1-\gamma^{-1})}\log(\tfrac{1}{\varepsilon}))$. The sample complexity increases as the growth parameter $\gamma$ becomes larger. Moreover, A4 allows us to transform the dual suboptimality into the distance to optimality: $\mathrm{dist}(\hat{\mathbf{y}},\mathcal{Y}^{\star})^{\gamma}\leq\varepsilon$. Back to the context of OLP, when the growth parameter $\gamma$ is small, it is possible to learn the distribution optimal solution with a small amount of customer data. For example, with $\gamma=1$ and $\varepsilon=\delta=1/T$, we only need the information of $\mathcal{O}(\log^{2}(T))$ customers to learn a highly accurate approximate dual solution satisfying $\mathrm{dist}(\hat{\mathbf{y}},\mathcal{Y}^{\star})\leq T^{-1}$. In other words, $\gamma$ characterizes the complexity or difficulty of the distribution of $(c,\mathbf{a})$; smaller $\gamma$ implies that the distribution is easier to learn.

The second implication of A4 comes from the stochastic optimization literature [20]: suppose the subgradient method (Algorithm 1) runs with constant stepsize $\alpha$, then the last iterate will end up in a noise ball around the optimal set, whose radius is determined by the initial distance to optimality $\mathrm{dist}(\mathbf{y}^{1},\mathcal{Y}^{\star})$ and the subgradient stepsize $\alpha$.

To demonstrate the role of Lemma 3.2 in our analysis. Suppose $\Delta$ is sufficiently small and $\alpha=\mathcal{O}(\Delta)$ is fixed. Then applying Lemma 3.2 with $T=1,\ldots$ shows that all the iterates generated by Algorithm 1 will satisfy

In other words, if $\mathbf{y}^{1}$ is close to the optimal set $\mathcal{Y}^{\star}$, then a proper choice of subgradient stepsize will keep all the iterates in a noise ball around $\mathcal{Y}^{\star}$. This noise ball is key to our improved reget guarantee.

The last implication, which connects the behavior of the subgradient method and OLP, states that the hindsight optimal dual solution $\mathbf{y}^{\star}_{T}$ will be close to $\mathcal{Y}^{\star}$.

Lemma 3.3 states a standard dual convergence result when A4 is present. This type of result is key to the analysis of the LP-based methods [19, 7, 22]. Although our analysis will not explicitly invoke Lemma 3.3, it provides sufficient intuition for our algorithm design: suppose $\mathrm{diam}(\mathcal{Y}^{\star})=0$, then the hindsight $\mathbf{y}^{\star}_{T}$, which has no regret, will be in an $o(1)$ neighborhood around $\mathbf{y}^{\star}$. In other words, if we have prior knowledge of the customer distribution (thereby, $\mathbf{y}^{\star}$), we can localize around $\mathbf{y}^{\star}$ since we know $\mathbf{y}^{\star}_{T}$ will not be far off. Moreover, according to Lemma 3.1 and Lemma 3.2, the subgradient method has the ability to get close to, and more importantly, to stay in proximity (the noise ball) around $\mathcal{Y}^{\star}$. Intuitively, if $\mathbf{y}^{1}$ is in an $o(1)$ neighborhood of $\mathcal{Y}^{\star}$, then we can adjust the stepsize of the subgradient method so that the online decision-making happens in an $o(1)$ neighborhood around $\mathcal{Y}^{\star}$, and better performance is naturally expected. Even if $\mathrm{diam}(\mathcal{Y}^{\star})>0$, the same argument still applies and can improve performance by a constant. In the next section, we formalize the aforementioned intuitions and establish a general framework for first-order methods to achieve better performance.

We start by formalizing the intuition that if $\mathbf{y}^{1}$ is sufficiently close to $\mathbf{y}^{\star}$, then adjusting the stepsize of the subgradient method allows us to make decisions in a noise ball around $\mathbf{y}^{\star}$ and achieve improved performance.

Putting Lemma 4.1 and Lemma 4.2 together, the performance of Algorithm 1 is characterized by

In the standard OLP analysis, it is only possible to ensure boundedness of $\|\mathbf{y}^{1}-\mathbf{y}^{\star}\|$ and $\|\mathbf{y}^{T+1}-\mathbf{y}^{\star}\|$. In other words,

and the optimal trade-off at $\alpha=\mathcal{O}(1/\sqrt{T})$ gives $\mathcal{O}(\sqrt{T})$ performance in Theorem 2.1. However, under A4, our analysis more accurately characterizes the behavior of the subgradient method, and we can do much better when $\mathbf{y}^{1}$ is close to $\mathbf{y}^{\star}$: suppose for now that $\mathrm{diam}(\mathcal{Y}^{\star})=0$ ($\mathcal{Y}^{\star}$ is a singleton) and that $\|\mathbf{y}^{1}-\mathbf{y}^{\star}\|=0$. Lemma 3.2 with $\Delta=0$ ensures that

Plugging (6) back into (5),

and taking $\alpha=T^{-\frac{\gamma}{2\gamma-1}}$ gives

This simple argument provides two important observations.

• •

  When $\gamma<\infty$ and $\mathrm{diam}(\mathcal{Y}^{\star})=0$, the knowledge of $\mathbf{y}^{\star}$ significantly improves the performance of first-order methods by shrinking the stepsize of the subgradient method from $\mathcal{O}(1/\sqrt{T})$ to $\mathcal{O}(T^{-\frac{\gamma}{2\gamma-1}})$: small stepsize implies localization around $\mathbf{y}^{\star}$. If $\gamma=1$, we achieve $\mathcal{O}(\log T)$ regret; if $\gamma\rightarrow\infty$, we recover $\mathcal{O}(\sqrt{T})$ regret.

• •

  Even if $\mathbf{y}^{\star}$ is known, the optimal strategy is not taking $\alpha=0$ and staying at $\mathbf{y}^{\star}$. Instead, $\alpha$ should be chosen according to $\gamma$, the strength of the error bound.

In summary, when $\mathbf{y}^{1}$ is close to $\mathbf{y}^{\star}$, we achieve improved performance guarantees through localization. The smaller $\gamma$ is, the smaller stepsize we take, and finally, the better regret we achieve. This observation matches Lemma 3.1: when a distribution is “easy”, we can trust $\mathbf{y}^{\star}$ and stay close to it.

Although it is sometimes reasonable to assume prior knowledge of $\mathbf{y}^{\star}$ beforehand, it is not always a practical assumption. Therefore, a natural strategy is learning it online from the customers. It is where the efficient learning algorithm from Lemma 3.1 comes into play and leads to an exploration-exploitation framework.

When $\mathbf{y}^{\star}$ is not known beforehand, Lemma 3.1 shows first-order methods can learn it from data and an exploration-exploitation strategy (Algorithm 2) is easily applicable: specify a target accuracy $\Delta$ and define

where $T_{e}$ is obtained by taking $\varepsilon=\Delta^{\gamma}$ and $\delta=T^{-2\gamma}$ in Lemma 3.1. Without loss of generality, we assume that $T_{e}$ is an integer and that $T\gg T_{e}$. Then Lemma 3.1 guarantees $\mathrm{dist}(\bar{\mathbf{y}}^{T_{e}+1},\mathcal{Y}^{\star})\leq\Delta$ with probability at least $1-T^{-2\gamma}$. In the exploitation phase, we use the subgradient method (Algorithm 1) with a properly configured stepsize to localize around $\mathcal{Y}^{\star}$ and achieve better performance. Lemma 4.3 characterizes the behavior of this two-phase algorithm Algorithm 2.

Lemma 4.3 presents two trade-offs:

• •

  Trade-off between exploration and exploitation.

  A high accuracy approximate dual solution $\mathrm{dist}(\bar{\mathbf{y}}^{T_{e}+1},\mathcal{Y}^{\star})=\Delta\approx 0$ allows localization and improves the performance in exploitation. However, reducing $\Delta$ requires a longer exploration phase and larger $V_{e}(T_{e})$.

• •

  Trade-off of stepsize within the exploitation phase.

  As in (7), the following terms dominate the performance in the exploitation phase

  $$\alpha T_{p}+\tfrac{\Delta}{\alpha}+\tfrac{\Delta^{2/\gamma}}{\alpha^{1/\gamma+1}T_{p}^{1/\gamma}}+\alpha^{1/\gamma-1}(\log T)^{1/\gamma}+\tfrac{{\mathrm{diam}}(\mathcal{Y}^{\star})}{\alpha}$$

  and we need to set the optimal $\alpha$ based on $(T_{p},\Delta,\gamma,{\mathrm{diam}}(\mathcal{Y}^{\star}))$.

Note that we haven’t specified the expression of $V(T_{e})$, since it depends on the dual sequence used for decision-making in the exploration phase. Ideally, $V(T_{e})$ should grow slowly in $T_{e}$ so that exploration provides a high-quality solution without compromising the overall algorithm performance. One natural idea is to make decisions based on the dual solutions produced by the efficient learning algorithm in Lemma 3.1. This is exactly what LP-based methods do [19]. However, as we will demonstrate in the next section, a good first-order learning algorithm can be inferior for decision-making. This counter-intuitive observation motivates the idea of decoupling learning and decision-making, and finally provides a general framework for first-order methods to go beyond $\mathcal{O}(\sqrt{T})$ regret.

Lemma 4.3 requires controlling $V(T_{e})$, the performance metric during exploration, by specifying $\{\mathbf{y}^{t}\}_{t=1}^{T_{e}}$ used for decision-making. It seems natural to adopt $\{\mathbf{y}^{t}_{L}\}_{t=1}^{T_{e}}$, the dual iterates produced by $\mathcal{A}_{L}$ for decision-making, and one may also wonder whether running $\mathcal{A}_{L}$ for decision-making over the whole horizon $T$ leads to further improved performance guarantees. However, this is not the case: using a good learning algorithm for decision-making leads to worse performance guarantees. To demonstrate this issue, we give a concrete example and consider the following one-dimensional multi-secretary online LP:

where $\{c_{t}\}_{t=1}^{T}$ are sampled uniformly from $[0,1]$. For this problem, $\mu=\tfrac{1}{2},\gamma=2$, and $y^{\star}=\tfrac{1}{2}$ is unique. Subgradient method with stepsize $\alpha_{t}=1/(\mu t)$ satisfies the convergence result of Lemma 3.1 is a suitable candidate for $\mathcal{A}_{L}$:

Lemma 4.4 suggests that using $\mathcal{A}_{L}$, we indeed approximate $y^{\star}$ efficiently. However, to approximate $y^{\star}$ to high accuracy, the algorithm will inevitably take small stepsize $\alpha_{t}$ when $t\geq\Omega(T)$. Following our discussion in Section 4.1, even with perfect information of $y^{\star}$, the online algorithm for $\gamma=2$ should remain adaptive to the environment by taking stepsize $\mathcal{O}(T^{-2/3})$. Taking $\mathcal{O}(1/T)$ stepsize nullifies this adaptivity, and the most direct consequence of lack of adaptivity is that, when the learning algorithm deviates from $y^{\star}$ due to noise, the overly small stepsize will take the algorithm a long time to get back. From an optimization perspective, this does not necessarily affect the quality of the final output $y^{T+1}$, since we only care about the quality of the final output. However, as a decision-making algorithm, the regret will accumulate when the algorithm tries to get back. This observation shows a clear distinction between stochastic optimization and online decision-making. Lemma 4.5 formalizes the aforementioned consequence:

As a consequence, a good learning algorithm, due to its lack of adaptivity, is a bad decision-making algorithm:

Although our example only covers $\gamma=2$, similar issues happen for other values of $\gamma$: the stepsize used by learning algorithms (Lemma 3.1) near convergence are much smaller than the optimal choice dictated by (7). This argument reveals a dilemma between learning and decision-making: a learning algorithm needs a small stepsize to achieve high accuracy, while a decision-making algorithm needs a larger stepsize to maintain adaptivity to the environment. This dilemma is inevitable for a single first-order method. However, the low computation cost of first-order methods opens up another way: it is feasible to use two separate algorithms for learning and decision-making.