Pseudonorm Approachability and
Applications to Regret Minimization

In this paper, we present a framework for converting high-dimensional $\ell_{\infty}$ approachability problems to low-dimensional “pseudonorm” approachability problems, in turn resolving many of these issues. To be precise, recall that the setting of approachability can be thought of as a $T$-round repeated game, where in round $t$ the learner chooses an action $p_{t}$ from some convex “action” set ${\mathscr{P}}\subseteq\mathbb{R}^{n}$, the adversary simultaneously chooses an action $\ell_{t}$ from some convex “loss” set ${\mathscr{L}}\subseteq\mathbb{R}^{m}$, and the learner receives a vector-valued payoff $u(p_{t},\ell_{t})$, where $u\colon{\mathscr{P}}\times{\mathscr{L}}\rightarrow\mathbb{R}$ is a $d$-dimensional bilinear function. The learner would like the $\ell_{\infty}$ distance between their average payoff $\frac{1}{T}\sum u(p_{t},\ell_{t})$ and some convex set¹¹1For simplicity, throughout this paper we assume ${\mathscr{S}}$ to be the negative orthant $(-\infty,0]^{d}$, as this is the case most relevant to regret minimization. However, much of our results extend straightforwardly to arbitrary convex sets. ${\mathscr{S}}$ to be as small as possible.

We first demonstrate how to construct a new $d^{\prime}$-dimensional bilinear function $\widetilde{u}\colon{\mathscr{P}}\times{\mathscr{L}}\rightarrow\mathbb{R}^{{}^{\prime}}$, a new convex set ${\mathscr{S}}^{\prime}\subseteq\mathbb{R}^{{}^{\prime}}$, and a pseudonorm²²2In this paper a pseudonorm is a function $f\colon\mathbb{R}^{{}^{\prime}}\rightarrow\mathbb{R}_{\geq 0}$ which satisfies most of the properties of a norm (e.g. positive homogeneity, triangle inequality), but may be asymmetric (there may exist $x$ where $f(x)\neq f(-x)$) and may not be definite (there may exist $x\neq 0$ where $f(x)=0$). Just as a norm $\lVert\cdot\rVert$ defines a distance between $x,y\in\mathbb{R}^{{}^{\prime}}$ via $\lVert x-y\rVert$, a pseudonorm $f$ defines the pseudodistance $f(x-y)$. $f$ such that the “pseudodistance” between the average modified payoff $\frac{1}{T}\sum\widetilde{u}(p_{t},\ell_{t})$ and ${\mathscr{S}}^{\prime}$ is equal to the $\ell_{\infty}$ distance between the original average payoff and ${\mathscr{S}}$. Importantly, the new dimension $d^{\prime}$ is equal to $mn$ and is independent of the original dimension $d$.

We then develop an algorithmic theory of pseudonorm approachability analogous to that developed in (Abernethy et al., 2011) for the $\ell_{2}$ norm and (Shimkin, 2016; Kwon, 2021) for other norms, showing that, in order to perform pseudonorm approachability, it suffices to be able to perform online linear optimization (OLO) over a convex set given by the Fenchel dual of the unit pseudonorm ball (and that the rate of approachability is directly related to the regret guarantees of this OLO subalgorithm). This has the following consequences for approachability:

• •

  First, by solving this OLO problem with a quadratic regularized Follow-The-Regularized-Leader (FTRL) algorithm, we show (modulo mild normalization assumptions on the sizes of ${\mathscr{P}}$, ${\mathscr{L}}$, and $u$) that there exists a pseudonorm approachability algorithm (and hence an $\ell_{\infty}$ approachability algorithm for the original problem) which converges at a rate of $O(nm/\sqrt{T})$. We additionally provide a stronger bound on the rate which scales as $O(D_{u}D_{p}D_{\ell}/\sqrt{T})$ where $D_{p}=\operatorname*{diam}{\mathscr{P}}$, $D_{\ell}=\operatorname*{diam}{\mathscr{L}}$, and $D_{u}$ is the maximum $\ell_{2}$ norm of the set of vectors formed by taking the coefficients of components of $u$ (Theorem 3.9). In comparison, the best-known generic guarantee for $\ell_{\infty}$-approachability prior to this work converged at a $d$-dependent rate of $O(\sqrt{(\log d)/T})$.

• •

  Second, we show that as long as we can evaluate the original $\ell_{\infty}$-distance between $\frac{1}{T}\sum u(p_{t},\ell_{t})$ and $(-\infty,0]^{d}$ efficiently, we can implement the above algorithm in $\operatorname*{poly}(m,n)$ time per round (Theorem 3.15). This has the following natural consequence for the class of regret minimization problems that can be written as $\ell_{\infty}$-approachability problems: if it is possible to efficiently compute some notion of regret for a sequence of losses and actions, then there is an efficient (in the dimensions of the actions and losses) learner that minimizes this regret.

• •

  Finally, in some cases, the $O(\sqrt{(\log d)/T})$ approachability rate from (inefficient) $\ell_{\infty}$-approachability outperforms the rate obtained by the quadratic regularized FTRL algorithm. We define a new regularizer whose value is given by finding the maximum entropy distribution of a subset of distributions of support $d$, and show that, by using this regularizer, we recover this $O(\sqrt{(\log d)/T})$ rate. In particular, whenever we can efficiently compute this maxent regularizer, there is an efficient learning algorithm with a $O(\sqrt{(\log d)/T})$ approachability rate.

• •

  We show that our framework straightforwardly recovers a regret-optimal and efficient algorithm for swap regret minimization. Doing so requires computing the above maximum entropy regularizer for this specific case, where we show that it has a nice closed form. In particular, to our knowledge. this is the first approachability-based algorithm for swap regret that both is efficient and has the optimal minimax regret.

• •

  In Section 4.3, we apply our framework to develop the first efficient contextual learning algorithms with low Bayesian swap regret. Such algorithms have the property that if learners employ them in a repeated Bayesian game, the time-average of their strategies will converge to a Bayesian correlated equilibrium, a well-studied equilibrium notion in game theory (see e.g. Bergemann and Morris (2016)).

  This notion of Bayesian swap regret was recently introduced by Mansour et al. (2022), who also provided an algorithm with low Bayesian swap regret, albeit one that is not computationally efficient. By applying our framework, we easily obtain an efficient contextual learning algorithm with $O(CK\sqrt{T})$ Bayesian swap regret, resolving an open question of Mansour et al. (2022) (here $C$ is the number of “contexts” / “types” of the learner).

• •

  In Section 4.4, we further analyze the application of our general $\ell_{\infty}$-approachability theory and algorithm to the analysis of reinforcement learning (RL) with vectorial losses. We point out how our framework can provide a general solution in the full information setting with known transition probabilities and how we can recover the best known solution for standard regret minimization in episodic RL. More importantly, we show how our framework and algorithm can lead to an algorithm for constrained MDPs with a significantly more favorable regret guarantee, logarithmic in the number of constraints $k$, in contrast with the $\sqrt{k}$-dependency of the results of Miryoosefi et al. (2019).

There is a wide literature dealing with various aspects of Blackwell’s approachability, including its applications to game theory, regret minimization, reinforcement learning, and multiple extensions.

Hart and Mas-Colell (2000) described an adaptive procedure for players in a game based on Blackwell’s approachability, which guarantees that the empirical distributions of the plays converges to the set of a correlated equilibrium. This procedure is related to internal regret minimization, for which, as shown by Foster and Vohra (1999), the existence of an algorithm follows the proof of Hart and Mas-Colell (2000). Hart and Mas-Colell (2001) further gave a general class of adaptive strategies based on approachability. Approachability has been widely used for calibration (Dawid, 1982), see Foster and Hart (2018) for a recent work on the topic. Approachability and partial monitoring were studied in a series of publications by Perchet (2010); Mannor et al. (2014a, b); Perchet and Quincampoix (2015, 2018); Kwon and Perchet (2017). More recently, approachability has also been used in the analysis of fairness in machine learning (Chzhen et al., 2021).

Approachability has also been extensively used in the context of reinforcement learning. Mannor and Shimkin (2003) discussed an extension of regret minimization in competitive Markov decision processes (MDPs) whose analysis is based on Blackwell’s approachability theory. Mannor and Shimkin (2004) presented a geometric approach to multiple-criteria reinforcement learning formulated as approachability conditions. Kalathil et al. (2014) presented strategies for approachability for MDPs and Stackelberg stochastic games based on Blackwell’s approachability theory. More recently, Miryoosefi et al. (2019) used approachability to derive solutions for reinforcement learning with convex constraints.

The notion of approachability was further extended in several studies. Vieille (1992) used differential games with a fixed duration to study weak approachability in finite dimensional spaces. Spinat (2002) formulated a necessary and sufficient condition for approachability of non-necessary convex sets. Lehrer (2003) extended Blackwell’s approachability theory to infinite-dimensional spaces.

The most closely related work to this paper, which we build upon, is that of Abernethy et al. (2011) who showed that, remarkably, any algorithm for Blackwell’s approachability could be converted into one for online convex optimization and vice-versa. Bernstein and Shimkin (2015) also discussed a related response-based approachability algorithm.

Perchet (2015) presented a specific study of $\ell_{\infty}$ approachability, for which they gave an exponential weight algorithm. Shimkin (2016, Section 5) studied approachability for an arbitrary norm and gave a general duality result for an arbitrary norm using Sion’s minmax theorem. The pseudonorm duality theorem we prove, using Fenchel duality, can be viewed as a generalization. Kwon (2016, 2021) also presented a duality theorem similar to that of Shimkin (2016) which they used to derive a FTRL algorithm for general norm approachability. They further treated the special case of internal and swap regret. However, unlike the algorithms derived in this work, the computational complexity of their swap regret algorithm is in $O(K^{K})$. This is also true of the paper of Perchet (2015) which also analyzes the swap regret problem.

It is known that if all players follow a swap regret minimization algorithm, then the empirical distribution of their play converges to a correlated equilibrium (Blum and Mansour, 2007). Hazan and Kale (2008) showed a result generalizing this property to the case of $\Phi$-regret and $\Phi$-equilibrium, where the $\Phi$-regret is the difference between the cumulative expected loss suffered by the learner and that of the best $\Phi$-modification of the sequence in hindsight. Gordon et al. (2008) further generalized the results of Hazan and Kale (2008) to a more general class of $\Phi$-modification regrets. The algorithms discussed in (Gordon et al., 2008) are distinct from those discussed in this paper (they do not clearly extend to the general approachability setting, and they require significantly different computational assumptions than ours). Nevertheless, they bare some similarity with our work.

We begin by illustrating the theory of Blackwell approachability for the specific case of the $\ell_{\infty}$-distance; this case is both particularly suited to the application of regret minimization, and will play an important role in the results (e.g. reductions to pseudonorm approachability) that follow.

We consider a repeated game setting, where every round $t$ a learner chooses an action $p_{t}$ belonging to a bounded³³3We bound the entries of ${\mathscr{P}}$, ${\mathscr{L}}$, and $u$ within $[-1,1]$ for convenience, but it is generally easy to translate between different boundedness assumptions (since almost all relevant quantities are linear). We express the majority of our theorem statements (with the notable exception of Theorem 3.9) in a way that is independent of the choice of bounds. convex set ${\mathscr{P}}\subseteq[-1,1]^{n}$, and an adversary simultaneously chooses a loss $\ell_{t}$ belonging to a bounded convex set ${\mathscr{L}}\subseteq[-1,1]^{m}$. Let $u\colon{\mathscr{P}}\times{\mathscr{L}}\rightarrow[-1,1]^{d}$ be a bounded bilinear⁴⁴4We briefly note that all our results also hold for biaffine functions $u_{i}$; in particular, extending the loss and action sets slightly (by replacing ${\mathscr{P}}$ and ${\mathscr{L}}$ with ${\mathscr{P}}\times\{1\}$ and ${\mathscr{L}}\times\{1\}$) allows us to write any biaffine function over the original sets as a bilinear function over the extended sets. vector-valued payoff function, and let ${\mathscr{S}}\subseteq\mathbb{R}$ be a closed convex set with the property that for every $\ell\in{\mathscr{L}}$, there exists a $p\in{\mathscr{P}}$ such that $u(p,\ell)\in{\mathscr{S}}$ (we say that such a set is “separable”). When $d=1$, the minimax theorem implies that there exists a single $p\in{\mathscr{P}}$ such that for all $\ell$, $u(p,\ell)\in{\mathscr{S}}$.

This is not true for $d>1$, but the theory of Blackwell approachability provides the following algorithmic analogue of this statement. Define a learning algorithm $\mathcal{A}$ to be a collection of functions $\mathcal{A}_{t}\colon{\mathscr{L}}^{t-1}\rightarrow{\mathscr{P}}$ for each $t\in[T]$, where $\mathcal{A}_{t}$ describes how the learner decides their action $p_{t}$ as a function of the observed losses $\ell_{1},\ell_{2},\dots,\ell_{t-1}$ up until time $t-1$. Blackwell approachability guarantees that there exists a learning algorithm $\mathcal{A}$ such that when $\mathcal{A}$ is run on any loss sequence $\bm{\ell}$ the resulting action sequence $\bm{p}$ has the property that:

As mentioned, one of the main motivations for studying Blackwell approachability is its connections to regret minimization. In particular, for a fixed choice of $u$, define

Note first that this definition of “regret” is exactly $T$ times the $\ell_{\infty}$ approachability distance in the case where ${\mathscr{S}}=(\infty,0]^{d}$ is the negative orthant; that is,

But secondly, note that by choosing ${\mathscr{P}}$, ${\mathscr{L}}$, and $u$ carefully, the definition $\eqref{eq:regret_def}$ can capture a wide variety of forms of regret studied in regret minimization. For example:

• •

  When ${\mathscr{P}}=\Delta_{K}$, ${\mathscr{L}}=[0,1]^{K}$, $d=K$, and $u(p,\ell)_{i}=\langle p,\ell\rangle-\ell_{i}$, $\operatorname{\mathsf{Reg}}(\bm{p},\bm{\ell})$ is the external regret of playing action sequence $\bm{p}$ against loss sequence $\bm{\ell}$; i.e., it measures the regret compared to the best single action.

• •

  When ${\mathscr{P}}=\Delta_{K}$, ${\mathscr{L}}=[0,1]^{K}$, $d=K^{K}$, and (for each function $\pi\colon[K]\rightarrow[K]$, $u(p,\ell)_{\pi}=\sum_{i=1}^{K}p_{i}(\ell_{i}-\ell_{\pi(i)})$, $\operatorname{\mathsf{Reg}}(\bm{p},\bm{\ell})$ is the swap regret of playing action sequence $\bm{p}$ against loss sequence $\bm{\ell}$; i.e., it measures the regret compared to the best action sequence $\bm{p^{\prime}}$ obtained by applying a fixed swap function to sequence $\bm{p}$.

• •

  When ${\mathscr{P}}\subseteq\mathbb{R}^{n}$ is a convex polytope, ${\mathscr{L}}=[0,1]^{n}$, $d=|V({\mathscr{P}})|$ (where $V({\mathscr{P}})$ is the set of vertices of ${\mathscr{P}}$), and (for each vertex $v\in V({\mathscr{P}})$), $u(p,\ell)_{v}=\langle p,\ell\rangle-\langle v,\ell\rangle$, this captures the (external) regret from performing online linear optimization over the polytope ${\mathscr{P}}$ (see Section 2.2).

• •

  Finally, to illustrate the power of this framework, we present an unusual swap regret minimization that we call “Procrustean swap regret minimization” (after the orthogonal Procrustes problem, see (Gower and Dijksterhuis, 2004)). Let ${\mathscr{P}}=\{x\in\mathbb{R}^{n}\,\colon||x||_{2}\leq 1\}$ be the unit ball in $n$ dimensions, ${\mathscr{L}}=[-1,1]^{n}$, and, for each orthogonal matrix⁵⁵5Technically, this leads to an infinite dimensional $u$ (since the group of orthogonal matrices is infinite), but one can instead take an arbitrarily fine discrete approximation of the set. Indeed, one of the advantages of the results we present is that they are largely independent of the dimension $d$ of $u$. $Q\in O(n)$, let $u(p,\ell)_{Q}=\langle p,\ell\rangle-\langle Qp,\ell\rangle$.

When the negative orthant $(-\infty,0]^{d}$ is separable with respect to $u$, which is true in all of the above examples, the theory of Blackwell approachability immediately guarantees the existence of a sublinear regret learning algorithm $\mathcal{A}$ for the corresponding notion of regret. Specifically, define the regret of a learning algorithm $\mathcal{A}$ to be the worst-case regret over all possible loss vectors $\bm{\ell}\in{\mathscr{L}}^{T}$; i.e.,

In fact, the theory of $\ell_{\infty}$-approachability is constructive and allows us to write down explicit algorithms $\mathcal{A}$ along with explicit (and in many cases, near optimal) regret bounds. However, before we introduce these algorithms, we will need to introduce the problem of online linear optimization.

Here we discuss algorithms for online linear optimization (OLO), a special case of online convex optimization where all the loss functions are linear functions of the learner’s action. These will form an important primitive of our algorithms for approachability.

Let ${\mathscr{X}},{\mathscr{Y}}$ be two bounded convex subsets of $\mathbb{R}^{r}$ and consider the following learning problem. Every round $t$ (for $T$ rounds) our learning algorithm must choose an element $x_{t}\in{\mathscr{X}}$ as a function of $y_{1},y_{2},\dots,y_{t-1}$. The adversary simultaneously chooses a loss “function” $y_{t}\in{\mathscr{Y}}$. This causes the learner to incur a loss of $\langle x_{t},y_{t}\rangle$ in round $t$. The goal of the learner is to minimize their regret, defined as the difference between their total loss and the loss they would have incurred by playing the best fixed action in hindsight, i.e.,

There are many similarities between this problem and the approachability and regret minimization problems discussed in Section 2.1. For example, if we take ${\mathscr{X}}={\mathscr{P}}=\Delta_{K}$ and ${\mathscr{Y}}={\mathscr{L}}=[0,1]^{K}$, then OLO is equivalent to the problem of external regret minimization. However, not all regret minimization problems can be written directly as an instance of OLO – for example, there is no clear way to write swap regret minimization as an OLO instance. Eventually we will demonstrate how to apply OLO as a subroutine to solve any regret minimization problem, but this will involve a reduction to Blackwell approachability and will require running OLO on different spaces than the action/loss sets ${\mathscr{P}}$ and ${\mathscr{L}}$ directly (which is why we distinguish the action/loss sets for OLO as ${\mathscr{X}}$ and ${\mathscr{Y}}$ respectively).

There is an important subclass of algorithms for OLO known as Follow the Regularized Leader (FTRL) algorithms (Shalev-Shwartz, 2007; Abernethy et al., 2008). An FTRL algorithm is completely specified by a strongly convex function $R\colon{\mathscr{X}}\rightarrow\mathbb{R}$, and plays the action

In words, this algorithm plays the action that minimizes the total loss on the rounds until the present (“following the leader”), subject to an additional regularization term. It is possible to characterize the worst-case regret of an FTRL algorithm in terms of properties of the regularizer $R$ and the sets ${\mathscr{X}}$ and ${\mathscr{Y}}$ (see e.g. Theorem 15 of Hazan (2016)). For our purposes, we will only need the following two results for specific regularizers.

We can now write down an explicit description of our algorithm for $\ell_{\infty}$-approachability (in terms of a blackbox OLO algorithm) and get quantitative bounds on the rate of convergence in the LHS of (1). Let $\mathcal{F}$ be an OLO algorithm for the sets ${\mathscr{X}}={\overline{\Delta}}_{d}$ and ${\mathscr{Y}}=[-1,1]^{d}$. Then we can describe our algorithm $\mathcal{A}$ for $\ell_{\infty}$-approachability as Algorithm 1.

It turns out that we can relate the $\ell_{\infty}$ approachability distance to $(-\infty,0]^{d}$ (and hence the regret of this algorithm $\mathcal{A}$) to the regret of our OLO algorithm $\mathcal{F}$.

If we let $\mathcal{F}$ be the negative entropy FTRL algorithm (Lemma 2.2)⁶⁶6There is a slight technical difference between the sets $\mathcal{X}={\overline{\Delta}}_{d}$ and $\mathcal{Y}=[-1,1]^{d}$ in Algorithm 1 and the sets $\mathcal{X}^{\prime}=\Delta_{d}$ and $\mathcal{Y}^{\prime}=[0,1]^{d}$ in Lemma 2.2. However, note that if we map $x\in\mathcal{X}$ to $(x_{1},x_{2},\dots,x_{n},1-\sum x_{i})\in\Delta_{d+1}$ and $y\in[-1,1]^{d}$ to $((1+y_{1})/2,(1+y_{2})/2,\dots,(1+y_{d})/2,0)\in[0,1]^{d+1}$, we preserve the inner product of $x$ and $y$ up to an additive constant, which disappears when computing regret, and a factor of $1/2$., we obtain the following regret guarantee for $\mathcal{A}$.

Equivalently, Corollary 2.4 can be interpreted as saying that there exists an $\ell_{\infty}$-approachability algorithm which approaches the negative orthant at an average rate of $O(\sqrt{(\log d)/T})$. In general, (3) lets us straightforwardly convert between results for approachability and results for regret minimization. Throughout the remainder of the paper we will primarily phrase our results in terms of $\operatorname{\mathsf{Reg}}(\mathcal{A})$, but switch between quantities of interest when convenient.

In Section 2.3, we described the theory of Blackwell approachability for a distance metric defined by the $\ell_{\infty}$ norm (i.e., $||z||_{\infty}=\max_{i}|z_{i}|$). We begin here by describing a generalization of this approachability theory to functions we refer to as pseudonorms and pseudodistances. A function $f\colon\mathbb{R}\to\mathbb{R}_{\geq 0}$ is a pseudonorm if $f(0)=0$, $f$ is positive homogeneous (for all $z\in\mathbb{R}$, $\alpha\in\mathbb{R}_{\geq 0}$, $f(\alpha z)=\alpha f(z)$), and $f$ satisfies the triangle inequality ($f(z+z^{\prime})\leq f(z)+f(z^{\prime})$ for all $z,z^{\prime}\in\mathbb{R}$). Note that unlike norms, pseudonorms may not satisfy definiteness and are not necessarily symmetric; it may be the case that $f(z)\neq f(-z)$. However, all norms are pseudonorms. Note also that by the positive homogeneity and the triangle inequality, a pseudonorm is a convex function. A pseudonorm $f$ defines a pseudodistance function $d_{f}$ via: $\forall z,z^{\prime}\in\mathbb{R},{}_{f}(z,z^{\prime})=f(z-z^{\prime})$.

In order to effectively work with pseudodistances, it will be useful to define the dual set ${\mathscr{T}}^{*}_{f}$ associated to $f$ as follows: ${\mathscr{T}}^{*}_{f}=\left\{\theta\colon\forall z\in\mathbb{R},\left\langle\theta,z\right\rangle\leq f(z)\right\}$. This coincides with the traditional notion of duality in convex analysis; for example, when $f$ is a norm, ${\mathscr{T}}^{*}_{f}$ coincides with the dual ball of radius one: ${\mathscr{T}}^{*}_{f}=\left\{\theta\colon\left\lVert\theta\right\rVert_{*}\leq 1\right\}$ (e.g., when $f$ is the $d$-dimensional $\ell_{\infty}$ norm, ${\mathscr{T}}_{f}^{*}$ is the $d$-dimensional $\ell_{1}$-ball). The following theorem relates the pseudodistance between $z$ and a convex set ${\mathscr{S}}$ to a convex optimization problem over the dual set.

We will primarily be concerned with the case where ${\mathscr{S}}$ is a convex cone. A convex cone is a set ${\mathscr{C}}$ such that if $x\in{\mathscr{C}}$, $\alpha x\in{\mathscr{C}}$ for all $\alpha\geq 0$. In this case, Theorem 3.1 can be simplified to write $d_{f}(z,{\mathscr{C}})$ as a linear optimization problem over the intersection of ${\mathscr{T}}^{*}_{f}$ and the polar cone of ${\mathscr{C}}$.

In Section 2.3, we expressed the regret for a general regret minimization problem as the $\ell_{\infty}$ distance

Consider the bilinear “basis map” $\widetilde{u}:{\mathscr{P}}\times{\mathscr{L}}\to\mathbb{R}^{nm}$ given by $\widetilde{u}(p,\ell)_{i,j}=p_{i}\ell_{j}$. Note that every bilinear function $b\colon{\mathscr{P}}\times{\mathscr{L}}\to\mathbb{R}$ can be written in the form $b(p,\ell)=\langle\widetilde{u}(p,\ell),v\rangle$ for some vector $v\in\mathbb{R}^{nm}$ (i.e., the monomials in $\widetilde{u}$ form a basis for the set of biaffine functions on ${\mathscr{P}}\times{\mathscr{L}}$)⁷⁷7If it is helpful, one can think of $\widetilde{u}$ as the natural map from ${\mathscr{P}}\times{\mathscr{L}}$ to the tensor product ${\mathscr{P}}\,\otimes\,{\mathscr{L}}$. The vectors $v_{i}$ can then be thought of as elements of the dual space, each representing a linear functional on ${\mathscr{P}}\,\otimes\,{\mathscr{L}}$..

Let $v_{i}\in\mathbb{R}^{nm}$ be the vector of coefficients corresponding to the component $u_{i}$ of $u$ and consider the function $f(x)=\max(\max_{i\in[d]}\langle v_{i},x\rangle,0)$. Note that $f$ is a pseudonorm on $\mathbb{R}^{nm}$; indeed, it’s straightforward to see that $f$ is non-negative and $f(\alpha x)=\alpha f(x)$ for any $\alpha\geq 0$. In addition, let ${\mathscr{C}}$ be the convex cone defined by ${\mathscr{C}}=\{x\,\mid\,\langle x,v_{i}\rangle\leq 0\,\forall i\in[d]\}$. We claim that that we can rewrite the $\ell_{\infty}$ distance in (5) in the following way.

The dual set ${\mathscr{T}}^{*}_{f}$ associated to this pseudonorm is given by:

The following two properties of this dual set will be useful in the sections that follow. First, we show that we can alternately think of ${\mathscr{T}}^{*}_{f}$ as the convex hull of the $v_{i}$.

Secondly, we show that the dual set ${\mathscr{T}}_{f}^{*}$ is contained within the polar cone ${\mathscr{C}}^{\circ}$ of ${\mathscr{C}}$.

This allows us to simplify Corollary 3.2 even further.

Finally, we prove the following “separability” conditions for this approachability problem.

We now present an algorithm for pseudonorm approachability in the setting of Section 3.2 (i.e., for the bilinear function $\widetilde{u}$ and the convex cone ${\mathscr{C}}$). Just as the algorithm in Section 2.3 for $\ell_{\infty}$-approachability required an OLO algorithm for the simplex, this algorithm will assume we have access to an OLO algorithm $\widetilde{\mathcal{F}}$ for the sets ${\mathscr{X}}={\mathscr{T}}_{f}^{*}$, and ${\mathscr{Y}}=-\operatorname*{conv}(\widetilde{u}({\mathscr{P}},{\mathscr{L}}))$ (recall that $\operatorname*{conv}(\widetilde{u}({\mathscr{P}},{\mathscr{L}}))$ denotes the convex hull of the points $\widetilde{u}(p,\ell)\in\mathbb{R}^{nm}$ for $p\in{\mathscr{P}}$ and $\ell\in{\mathscr{L}}$).

Our algorithm $\widetilde{\mathcal{A}}$ is summarized above in Algorithm 2. We now have the following analogue of Theorem 2.3.

If we choose $\widetilde{\mathcal{F}}$ to be FTRL with a quadratic regularizer, Lemma 2.1 implies the following result.

Theorem 3.9 shows that in settings where $\lambda$ is constant (which is true in all the settings we consider), there exists an algorithm where $\operatorname{\mathsf{Reg}}(\widetilde{\mathcal{A}})=\operatorname*{poly}(n,m)\sqrt{T}$; notably, $\operatorname{\mathsf{Reg}}(\widetilde{\mathcal{A}})$ does not depend on the dimension $d$, which in many cases can be thought of as the number of benchmarks of comparison for our learning algorithm.

In the following two sections, we will show how to strengthen this result in two different ways. First, we will show that (modulo some fairly mild computational assumptions) it is possible to efficiently implement the algorithm of Theorem 3.9. Second, we will show that by using a different choice of regularizer, we can recover exactly the regret obtained in Corollary 2.4.