Multi-point Feedback of Bandit Convex Optimization with Hard Constraints

Bandit Convex Optimization (BCO) is a fundamental framework of sequential decision-making under uncertain environments and with limited feedback, which can be regarded as a structured repeated game between a learner and an environment (Hazan et al. 2016, Lattimore and Szepesvári 2020). In this framework, a learner is given a convex feasible region $\mathcal{X}\subseteq\mathbb{R}^{d}$ and the total number $T$ of rounds. At each round, $t=1,2,\dots,T$, the learner makes decision $\bm{x}_{t}\in\mathcal{X}$, and then a convex loss function $f_{t}:\mathcal{X}\to\mathbb{R}$ is revealed. The learner cannot access the loss function $f_{t}$, but only the bandit feedback is available, i.e., the learner can only observe the value of the loss at the point she committed to, i.e., $f_{t}(\bm{x}_{t})$. The objective of the learner is to generate a sequence of decisions $\{\bm{x}_{t}\}_{t=1}^{T}\subseteq\mathcal{X}$ that minimizes cumulative loss $\sum_{t=1}^{T}f_{t}(\bm{x}_{t})$ under bandit feedback. The performance of the learner is evaluated in terms of regret, which is defined by

This regret measures the difference between the cumulative loss of the learner’s strategy and the minimum possible cumulative loss where the sequence of loss functions $\{f_{t}(\bm{x})\}_{t=1}^{T}$ had been known in advance and the learner could choose the best fixed optimal decision in hindsight.

In many real-world scenarios, the decisions are often subject to some constraints such as budget or resources. In the context of Online Convex Optimization (OCO), where the learner has access to the complete information about the loss functions, a projection operator is typically applied in each round so that the decisions belong to constraints (Zinkevich 2003, Hazan et al. 2016). However, such a projection step is typically a computational bottleneck when the feasible region is complex.

To address the issue of the projection step, Mahdavi et al. (2012) considers online convex optimization with long-term constraints, where the learner aims to generate a sequence of decisions that the decisions satisfy constraints in the long run, instead of requiring to satisfy the constraints in all rounds. They introduce the cumulative soft constraint violation metric defined by $V^{\text{soft}}_{T}\coloneqq\sum_{t=1}^{T}g_{t}(\bm{x}_{t})$, where $g_{t}(\bm{x})\leq 0$ is the functional constraint to be satisfied. Later, Yuan and Lamperski (2018) consideres strict notion of constraint violation reffered to as cumulative hard constraint violation, which is defined by $V^{\text{hard}}_{T}\coloneqq\sum_{t=1}^{T}\max\{g_{t}(\bm{x}_{t}),0\}$. This metric overcomes the drawback of cumulative soft constraint violation, and it is suitable for safety-critical systems, in which the failure of constraint violation may result in catastrophic consequences.

To see that the notion of cumulative hard constraint violation is a stronger metric, let us consider the example discussed in Guo et al. (2023). Given a sequence of decisions whose constraint functions are $\{g_{t}(\bm{x}_{t})\}_{t=1}^{T}$ with $T=1000$ such that $g_{t}(\bm{x}_{t})=-1$ if $t$ is odd; otherwise $g_{t}(\bm{x}_{t})=1$, we have $\sum_{t=1}^{\tau}g_{t}(\bm{x}_{t})\leq 0$ for any $\tau\in\{1,2,\dots,T\}$, however, the constraint $g_{t}(\bm{x})\leq 0$ is violated at half of rounds. On the other hand, the notion of hard constraint violation can capture the constraint violation since we have $V^{\text{hard}}_{T}=500$. Thus, the conventional definition of cumulative soft constraint violation $V_{T}^{\text{soft}}$ cannot accurately measure the constraint violation but cumulative hard constraint violation $V_{T}^{\text{hard}}$ can.

Many existing algorithms for BCO with constraints proposed in prior works typically involve projection operators as well as algorithms for OCO with constraints (Agarwal et al. 2010, Zhao et al. 2021), and are generally limited to the simple convex set. Chen et al. (2019), Garber and Kretzu (2020) consider a projection-free algorithm for BCO, but the constraint violation bound has not been reported. Some studies have extended the algorithm for OCO with soft constraints to the bandit setting (Mahdavi et al. 2012, Cao and Liu 2018), however, these algorithms cannot be directly extended to BCO with hard constraints. In other words, there has been no algorithm that can simultaneously achieve sub-linear bound both regret and cumulative hard constraints violation.

The present study focuses on the particular case of multi-point feedback of BCO with constraints, in which the loss functions are convex or strongly convex, and constraint violation is evaluated in terms of hard constraints. This kind of problem widely appears in real-world scenarios such as portfolio management problems, in which the manager has concrete constraints to be satisfied but only has access to the loss function $f_{t}(\cdot)$ at several points close to the decision $\bm{x}_{t}$. We present a penalty-based proximal gradient descent method which attains both $O(d^{2}T^{\max\{c,1-c\}})$ regret bound and $O(d^{2}T^{1-\frac{c}{2}})$ cumulative hard constraint violation bound, where $d$ is the dimensionality of the feasible region and $c\in[\frac{1}{2},1)$ is a user-determined parameter. Our proposed algorithm is inspired by a gradient estimation in the BCO literature (Flaxman et al. 2005, Agarwal et al. 2010) and an algorithm for OCO with hard constraints (Guo et al. 2022).

This section presents the proposed algorithm for solving the constrained BCO with two-point feedback. The procedure of the algorithm is shown in Algorithm 1, and this algorithm is motivated by the work in Guo et al. (2022) and the design of the algorithm is related to penalty-based proximal gradient descent method (Cheung and Lou 2017). At round $t\in[T]$, Algorithm 1 finds the decision vector $\bm{x}_{t+1}$ by solving the following strongly convex optimization problem:

where $\lambda_{t}$ is the penalty variable for controlling the quality of the decision, $\widehat{g}^{+}_{t}(\bm{x})\coloneqq\gamma_{t}[g_{t}(\bm{x})]_{+}$, $\xi>0$ is the shrinkage constant, and $\alpha_{t}>0,\,\gamma_{t}>0$ are predetermined learning rate. Note that the optimization problem in the right-hand side (r.h.s) of Eq. (6) is strongly convex optimization due to the $\ell_{2}$ regularizer term, and hence the optimal solution $\bm{x}_{t+1}$ does exist and unique. As is the case with Mahdavi et al. (2012), we optimize the r.h.s. of Eq. (6) on the domain $(1-\xi)\mathcal{X}$ to ensure that randomized two points around $\bm{x}_{t}$ are inside the feasible region $\mathcal{X}$. As shown in Flaxman et al. (2005), for any $\bm{x}\in(1-\xi)\mathcal{X}$ and for any unit vector $\bm{u}\in\mathbb{S}^{d}$, it holds $\bm{x}\pm\delta\bm{u}\in\mathcal{X}$.

At round $t$, where we find the decision $\bm{x}_{t+1}\in\mathcal{X}$, since we do not have the prior knowledge of the loss function $f_{t+1}(\bm{x})$ to be minimized, we estimate the loss by the first-order approximation at the previous decision $\bm{x}_{t}$ as $\widetilde{f}_{t+1}(\bm{x})=f_{t}(\bm{x}_{t})+\left<\nabla f_{t}(\bm{x}_{t}),\bm{x}-\bm{x}_{t}\right>$. Simultaneously, we have no full information of the loss function $f_{t}(\bm{x})$ and hence we cannot access its graient $\nabla f_{t}(\bm{x})$, so we estimate gradient by $\widetilde{\nabla}f_{t}$ with two points (line 5). To prevent the constraint from being severely violated, we also introduce the rectified Lagrange multiplier $\lambda_{t}$ associated with the functional constraint $g_{t}(\bm{x})\leq 0$, and add the penalty term $\lambda_{t}\widehat{g}_{t}^{+}(\bm{x})$ to the objective function (6), which is an approximator of the original penalty term $\theta_{t}g_{t}(\bm{x})$, where $\theta_{t}$ is the Lagrangian multiplier associated with the constraint $g_{t}(\bm{x})\leq 0$. We also add $\ell_{2}$ regularization term $\frac{\alpha_{t}}{2}\norm{\bm{x}-\bm{x}_{t}}^{2}_{2}$ to stabilize the optimization problem.

We will describe more in detail the role of penalty parameter $\lambda_{t}$ and its update rule. The penalty parameter $\lambda_{t}$ is related to the Lagrangian multiplier (denoted by $\theta_{t}$) associated with the functional constraint $g_{t}(\bm{x})\leq 0$, but slightly different because we have no prior knowledge of the constraint functions when making-decision. Instead, we take place the original Lagrangian multiplier $\theta_{t+1}$ with $\lambda_{t}$ such that $\lambda_{t}\widehat{g}^{+}_{t}(\bm{x})$ is an approximator of $\theta_{t}g_{t}(\bm{x})$. We update the penalty parameter (line 9) as $\lambda_{t+1}=\max\{\lambda_{t}+\gamma_{t+1}[g_{t+1}(\bm{x}_{t})]_{+},\eta_{t+1}\}$, where the first coordinate of maximum operator is the sum of the old $\lambda_{t}$ and the rectified constraint function value $\gamma_{t+1}[g_{t+1}(\bm{x}_{t})]_{+}$; and the second coordinate is the user-determined constant $\eta_{t+1}$ to impose a minimum penalty. This update rule for the penalty parameter prevents the decision determined by solving the problem (6) from being overly aggressive which leads to large constraint violation.

This section provides the theoretical analysis for the Algorithm 1. To facilitate the analysis, let $h_{t}:\mathcal{X}\to\mathbb{R}$ be a function defined by

where $\widehat{f}_{t}(\bm{x})=\mathbb{E}_{\bm{v}\in\mathbb{B}^{d}}[f_{t}(\bm{x}+\delta\bm{v})]$ and $\widetilde{\nabla}f_{t}$ is defined as Eq. (5). It is easily seen that $\nabla h_{t}(\bm{x}_{t})=\widetilde{\nabla}f_{t}$ holds, and hence we have $\norm{\nabla h_{t}(\bm{x})}_{2}=\|\widehat{\nabla}f_{t}\|_{2}\leq d\text{lip}(f_{t})$ for any $\bm{x}\in\mathcal{X}$. Moreover, the function $h_{t}$ defined as Eq. (7) is convex and Lipschitz continuous with Lipschitz constant $\operatorname{lip}(h_{t})=3d\operatorname{lip}(f_{t})$ on $\mathcal{X}$, because for any $\bm{x},\bm{y}\in\mathcal{X}$, we have

where the first inequality follows from the triangle inequality, the second inequality follows from the Cauchy-Schwarz inequality, the third inequality follows from $\norm{\nabla f(\bm{x})}_{2}\leq\operatorname{lip}(f)$ for any Lipshitz continuous function $f$ and for any $\bm{x}\in\mathcal{X}$, and the last inequality follows from $\operatorname{lip}(\widehat{f}_{t})=\operatorname{lip}(f_{t})$.

To prove Algorithm 1 attains sub-linear bound for both regret and cumulative hard constraint violation, we first show the following result which is a well-known property of a strongly convex function.

The following two lemmas play an important role in proving the main theorem (Theorem 1 and Theorem 2). The first one (Lemma 3) is an inequality involving the update rule of Algorithm 1, and the second one (Lemma 4) characterizes the relationship between the current solution $\bm{x}_{t}$ in Algorithm 1 and the optimal solution of the offline optimization problem formulated as Eq. (1).

We are now ready to prove the main results, which state Algorithm 1 achieves a sub-linear bound for both regret (2) and cumulative hard constraint violation (3). We first show the case where the loss functions are convex and constraint functions are fixed throughout the whole round.