Sharp bounds on the price of bandit feedback for several models of mistake-bounded online learning

Raymond Feng rfeng2004@gmail.com Jesse Geneson geneson@gmail.com Andrew Lee leeandrew1029@gmail.com Espen Slettnes espen@slett.net

Abstract

We determine sharp bounds on the price of bandit feedback for several variants of the mistake-bound model. The first part of the paper presents bounds on the $r$ -input weak reinforcement model and the $r$ -input delayed, ambiguous reinforcement model. In both models, the adversary gives $r$ inputs in each round and only indicates a correct answer if all $r$ guesses are correct. The only difference between the two models is that in the delayed, ambiguous model, the learner must answer each input before receiving the next input of the round, while the learner receives all $r$ inputs at once in the weak reinforcement model.

In the second part of the paper, we introduce models for online learning with permutation patterns, in which a learner attempts to learn a permutation from a set of permutations by guessing statistics related to sub-permutations. For these permutation models, we prove sharp bounds on the price of bandit feedback.

1 Introduction

We investigate several variants of the mistake-bound model [11]. In the standard model [1, 12] (also called the standard strong reinforcement learning model [1]), a learner attempts to classify inputs (in the set $X$ ) with labels (in the set $Y$ ) based on a set of possible functions $f:X\to Y$ in $F$ . The learning proceeds in rounds, and in each round the adversary gives the learner an input, and the learner must then guess the corresponding label. After each round, the adversary informs the learner of the correct answer (and therefore whether the learner was right or wrong). A variant of this model is the standard weak reinforcement learning model [1, 2], where the adversary only tells the learner yes if they were correct and no otherwise. This variant is also commonly called the bandit model [5, 6, 7, 10, 12].

For any learning scenario, we generally let $\operatorname{opt}_{\operatorname{scenario}}(F)$ represent the optimal worst case number of mistakes that a learning algorithm can achieve [1]. For example, for the bandit model and the standard model, the optimal worst case performances of learning algorithms would be denoted $\operatorname{opt}_{\operatorname{weak}}(F)=\operatorname{opt}_{\operatorname{bandit}}(F)$ and $\operatorname{opt}_{\operatorname{std}}(F)=\operatorname{opt}_{\operatorname{strong}}(F)$ respectively. There are some obvious inequalities that follow by definition, such as $\operatorname{opt}_{\operatorname{strong}}(F)\leq\operatorname{opt}_{\operatorname{weak}}(F)$ , just from the fact that the learner has strictly more information in one scenario compared to the other.

In [1], Auer and Long defined the $r$ -input delayed, ambiguous reinforcement model and compare it to a modified version of the standard weak reinforcement model (henceforth called the $r$ -input weak reinforcement model). The delayed, ambiguous reinforcement model is a situation where the learner receives a fixed number ( $r$ ) of inputs each round, and each input is given to the learner after they have answered the previous one. On the other hand, the learner receives all $r$ inputs at once for each round in the modified weak reinforcement model. In both situations, at the end of every round of $r$ inputs, the adversary says yes if the learner answered all $r$ inputs correctly and no otherwise. To compare the two situations, Auer and Long defined $\operatorname{CART}_{r}(F)$ (where $F$ is a set of functions $f:X\to Y$ ) to be a set of functions $f^{\prime}:X^{r}\to Y^{r}$ where each $f\in F$ has a corresponding $f^{\prime}\in\operatorname{CART}_{r}(F)$ such that for any $x_{1},x_{2},\dots,x_{r}\in X$ , we have $f^{\prime}((x_{1},x_{2},\dots,x_{r}))=(f(x_{1}),f(x_{2}),\dots,f(x_{r}))$ . They used $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ to represent the optimal worst-case performance in the modified weak reinforcement setting and $\operatorname{opt}_{\operatorname{amb},r}(F)$ to represent the optimal worst-case performance in the $r$ -input delayed, ambiguous reinforcement model. Auer and Long proved that the two situations are not equivalent for the learner in [1] by showing that that there is some input set $X$ and set $F$ of functions from $X$ to $\{0,1\}$ such that $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{2}(F))<\operatorname{opt}_{\operatorname{amb},2}(F)$ .

In Sections 2 and 3, we obtain sharp bounds on the maximum possible multiplicative gap between $\operatorname{opt}_{\operatorname{amb},r}(F)$ and $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ . In particular, we show that the multiplicative gap between $\operatorname{opt}_{\operatorname{amb},r}(F)$ and $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ can grow exponentially with $r$ , generalizing Auer and Long’s result from [1] for $r=2$ . Combined with a bound from [1], our new result shows that the maximum possible multiplicative gap between $\operatorname{opt}_{\operatorname{amb},r}(F)$ and $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ is $2^{r(1\pm o(1))}$ . Moreover, we give sharp bounds on $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ (in Section 2) and $\operatorname{opt}_{\operatorname{amb},r}(F)$ (in Section 3) for all sets $F$ which are a subset of the non-decreasing functions from $X$ to $\{0,1\}$ .

In a different paper, Long [12] also determined sharp bounds comparing the standard model and the standard weak reinforcement model for multi-class functions. Long proved the upper bound $\operatorname{opt}_{\operatorname{bandit}}(F)\leq(1+o(1))(|Y|\ln|Y|)\operatorname{opt}_{\operatorname{std}}(F)$ and constructed infinitely many $F$ for which $\operatorname{opt}_{\operatorname{bandit}}(F)\geq(1-o(1))(|Y|\ln|Y|)\operatorname{opt}_{\operatorname{std}}(F)$ as a lower bound. Geneson corrected an error in the proof of the lower bound [9].

In Section 4, we generalize this result to determine a lower bound and upper bound on the maximum factor gap between $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ and $\operatorname{opt}_{\operatorname{std}}(F)$ for multi-class functions using probabilistic methods and linear algebra techniques. The proof uses techniques previously used for experimental design [15, 14] and hashing, derandomization, and cryptography [4, 13]. We also determine a lower bound and upper bound on the maximum factor gap between $\operatorname{opt}_{\operatorname{amb,r}}(F)$ and $\operatorname{opt}_{\operatorname{std}}(F)$ . The bounds in this section are sharp up to a factor of $r(1+o(1))$ .

In Section 5, we define several new models where the learner is trying to guess a function from a set of permutations of length $n$ . In the order model, the adversary chooses $r$ inputs, and the learner attempts to guess the corresponding sub-permutation. In the comparison model, the adversary instead chooses $r$ pairs of inputs, and the learner attempts to guess for each pair whether or not it is an inversion. In the selection model, the adversary chooses $r$ inputs, and the learner attempts to guess which has the maximum evaluation. In the relative position model, the adversary chooses $r$ inputs and an additional distinguished input $x$ , and the learner attempts to guess the relative position of $x$ in the sub-permutation corresponding to all $r+1$ of these inputs. Finally, in the delayed relative position model, the adversary instead gives the $r$ elements to be compared to $x$ one at a time. We first establish general upper bounds, and then we discuss adversary strategies for a few special families of permutations that resemble sorting algorithms.

Finally, in Section 6, we discuss some future directions based on the results in this paper.

2 Bounds on $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$

In this section, we establish upper and lower bounds on $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ for families $F$ of non-decreasing functions that are within a constant factor of each other. We show for such families $F$ that $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))=(1\pm o(1))r\ln(|F|)$ .

Definition 1.

Without loss of generality impose an ordering on the set $X$ , and call its elements $\{1,2,\dots,|X|\}$ . Let $F=\{f_{1},f_{2}.\dots,f_{|F|}\}$ be a subset of the functions from $X$ to $\{0,1\}$ . We say that $F$ is non-decreasing if every function in $F$ is non-decreasing.

In other words, there are integers $1\leq a_{1}<a_{2}<\dots<a_{|F|}\leq|X|+1$ which are the minimum numbers such that $f_{i}(a_{i})=1$ (with the convention that $a_{|F|}=|X|+1$ if $f_{|F|}$ is identically 0) and satisfy the following property for each $1\leq i\leq|F|$ :

•

If $x>a_{i}$ , then $f_{i}(x)=1$ .
•

If $x<a_{i}$ , then $f_{i}(x)=0$ .

Remark 2.

Let $F$ be non-decreasing. For a function $f\in F$ and any choice of $r$ inputs from the set $X$ , there are at most $r+1$ possible values of the corresponding outputs $(f(x_{1}),\dots,f(x_{r}))$ , namely $(0,0,\dots,0,0)$ , $(0,0,\dots,0,1)$ , $\ldots$ , $(0,1,\dots,1,1)$ , and $(1,1,\dots,1,1)$ .

2.1 Bounds on $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$

The following theorem establishes an upper bound on $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ by exhibiting a possible learner strategy.

Theorem 3.

For non-decreasing $F$ , $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))<(r+1)\cdot\ln(|F|)$ .

Proof.

The learner’s strategy: pick the answer that corresponds to the most functions. By Remark 2, we know that there are at most $r+1$ such answers.

Each time the adversary says no, if the learner previously knew that there were $T$ possible functions left, then the learner is able to reduce the number of possible functions left by at least $\frac{T}{r+1}$ . Thus, each answer of no means that the number of remaining possibilities is multiplied by a factor of at most $\frac{r}{r+1}$ on each turn.

Then, the learner will make at most

\log_{\frac{r+1}{r}}(|F|)=\frac{\ln(|F|)}{\ln\left(1+\frac{1}{r}\right)}<\frac{\ln(|F|)}{\frac{1}{r+1}}=(r+1)\cdot\ln(|F|)

mistakes, as desired. ∎

Next we establish a lower bound on $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ by exhibiting a strategy for the adversary.

Theorem 4.

For non-decreasing $F$ , $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))\geq(1-o(1))r\ln(|F|)$ as $|F|\rightarrow\infty$ .

Proof.

In each round, the adversary will say no. When they do this, some functions fail to remain consistent with the answers given by the adversary. Let $F^{\prime}\subseteq F$ be the current set of functions that are consistent with the answers that the adversary has given so far. Furthermore, with $F^{\prime}=\{g_{1},g_{2},\dots,g_{|F^{\prime}|}\}$ , define $1\leq b_{1}<b_{2}<\dots<b_{|F^{\prime}|}\leq|X|+1$ as the minimum numbers such that $g_{i}(b_{i})=1$ (with the convention that $b_{|F^{\prime}|}=|X|+1$ if $f_{|F^{\prime}|}$ is identically 0).

In each round, the adversary will choose the inputs

x_{i}=b_{i\cdot\left\lceil\frac{|F^{\prime}|}{r+1}\right\rceil+1}

for $1\leq i\leq r$ . Then, no matter what the learner says, the adversary says no. This guarantees that the number of remaining consistent functions decreases by at most $\left\lceil\frac{|F^{\prime}|}{r+1}\right\rceil$ functions per round. Thus, the adversary can continue for at least

(1-o(1))\log_{\frac{r+1}{r}}(|F|)=(1-o(1))\cdot\frac{\ln(|F|)}{\ln(1+\frac{1}{r})}\geq(1-o(1))r\ln(|F|)

turns. Therefore, the adversary guarantees that they can say no at least $(1-o(1))r\ln(|F|)$ times, as desired. ∎

Combining the above bounds, this implies that for non-decreasing $F$ , we have the following theorem:

Theorem 5.

For non-decreasing $F$ , $\lim_{r\to\infty}\lim_{|F|\to\infty}\frac{\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))}{r\ln(|F|)}=1$ .

3 Bounds on $\operatorname{opt}_{\operatorname{amb},r}(F)$

In this section, we show that $\operatorname{opt}_{\operatorname{amb},r}(F)=(1-o(1))2^{r}\ln(|F|)$ as $|F|,r\rightarrow\infty$ for non-decreasing families $F$ . The upper bound in this section is applicable to all families of functions $F$ , but the lower bound is only applicable for non-decreasing sets of functions $F$ , as with the bounds established in Section 2.

We prove the following theorem, a general upper bound for $\operatorname{opt}_{\operatorname{amb},r}(F)$ , using a learner strategy that does not make any assumptions about the set $F$ . In particular, $F$ does not have to be non-decreasing.

Theorem 6.

For all $F$ , $\operatorname{opt}_{\operatorname{amb},r}(F)<\min(2^{r}\ln(|F|),|F|)$ .

Proof.

For each input that the adversary gives, the learner picks the answer that corresponds to the most functions which were consistent with all answers before the round started and also consistent with all earlier guesses in the current round.

Each time the adversary says no, if the learner knew that there were $T$ possible functions remaining before the round started, then they can guarantee that their answer for all $r$ inputs is consistent with at least $\frac{T}{2^{r}}$ of those functions. To see this, note that by induction, at least $\frac{T}{2^{k}}$ of the functions will be consistent with the $k$ answers they have given so far for each $1\leq k\leq r$ . Therefore, each time the adversary says no the number of remaining possible functions is multiplied by at most $\frac{2^{r}-1}{2^{r}}$ . So, the learner makes at most

\log_{\frac{2^{r}}{2^{r}-1}}(|F|)=\frac{\ln(|F|)}{\ln\left(1+\frac{1}{2^{r}-1}\right)}<\frac{\ln(|F|)}{\frac{1}{2^{r}}}=2^{r}\cdot\ln(|F|)

mistakes with this strategy.

The learner’s strategy to get $\operatorname{opt}_{\operatorname{amb},r}(F)\leq|F|-1$ : Each time the adversary says no, the learner can eliminate at least 1 function. Once the learner has eliminated $|F|-1$ functions, no more errors will be made. ∎

For the following lower bound on $\operatorname{opt}_{\operatorname{amb},r}(F)$ , we again assume that $F$ is non-decreasing.

Theorem 7.

For non-decreasing $F$ , $\operatorname{opt}_{\operatorname{amb},r}(F)\geq(1-o(1))(2^{r}-1)\ln(|F|)$ as $|F|\rightarrow\infty$ .

Proof.

The adversary will say no at the end of each round. For each round, the adversary will choose a series of input values $x_{i}$ based on the answers given by the learner. In each subround, the next input $x_{i}$ is determined as follows: suppose that $S$ is the set of all functions that are consistent with all previous adversary answers from past rounds as well as all the answers of the learner from the current round.

Since $S\subseteq F$ , we can then set $S=\{g_{1},g_{2},\dots,g_{|S|}\}$ and define $1\leq b_{1}<b_{2}<\dots<b_{|S|}\leq|X|+1$ as the minimum numbers such that $g_{i}(b_{i})=1$ (with the convention that $b_{|S|}=|X|+1$ if $f_{|S|}$ is identically 0).

The adversary then chooses $x_{i}=b_{\left\lceil\frac{|S|}{2}\right\rceil}$ for the current subround. This guarantees that at each subround, the number of functions consistent with all of the adversary’s previous answers as well as all of the learner’s answers in the current round reduces by at most $\left\lceil\frac{|S|}{2}\right\rceil$ . Thus, if $T$ functions were consistent with all of the adversary’s previous answers at the beginning of the current round, then at the end of the round, at most $\left\lceil\frac{T}{2^{r}}\right\rceil$ functions become inconsistent with the adversary’s answers. Here, we are repeatedly using the fact that $\left\lceil\frac{\left\lceil x\right\rceil}{n}\right\rceil=\left\lceil\frac{x}{n}\right\rceil$ for all positive reals $x$ and positive integers $n$ .

This means that the adversary can continue to say no for at least

\displaystyle(1-o(1))\log_{\frac{2^{r}}{2^{r}-1}}(|F|)

\displaystyle=(1-o(1))\frac{\ln(|F|)}{\ln\left(1+\frac{1}{2^{r}-1}\right)}\geq(1-o(1))(2^{r}-1)\ln(|F|)

turns, as desired. ∎

Combining the above bounds, this implies that for non-decreasing $F$ , we have the following theorem:

Theorem 8.

For non-decreasing $F$ , $\lim_{r\to\infty}\lim_{|F|\to\infty}\frac{\operatorname{opt}_{\operatorname{amb},r}(F)}{2^{r}\ln(|F|)}=1$ .

Theorem 5 and Theorem 8 imply for non-decreasing families of functions $F$ that for sufficiently large $r$ and $|F|$ that learners who are given all inputs at the beginning of each round do exponentially better in $r$ than their counterparts who receive inputs one at a time in each round.

In [1], Auer and Long proved that $\operatorname{opt}_{\operatorname{amb},r}(F)\leq 2(\ln{2r})\cdot 2^{r}\cdot\operatorname{opt}_{\operatorname{std}}(F)$ . Since $\operatorname{opt}_{\operatorname{std}}(F)\leq\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ for all families of functions $F$ , this implies the following result when combined with our Theorem 5 and Theorem 8.

Theorem 9.

The maximum possible value of $\frac{\operatorname{opt}_{\operatorname{amb},r}(F)}{\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))}$ over all families of functions $F$ with $|F|>1$ is $2^{r(1\pm o(1))}$ .

4 Comparing $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ and $\operatorname{opt}_{\operatorname{amb},r}(F)$ to $\operatorname{opt}_{\operatorname{std}}(F)$ for multi-class functions

In [12], Long compared the standard and bandit model for families of multi-class functions, and determined a bound on the maximum multiplicative gap between them. There was an error in Long’s proof of the lower bound, but Geneson fixed this error in [9]. In this section, we bound the maximum multiplicative gap between $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ and $\operatorname{opt}_{\operatorname{std}}(F)$ using similar methods, as well as the maximum multiplicative gap between $\operatorname{opt}_{\operatorname{amb},r}(F)$ and $\operatorname{opt}_{\operatorname{std}}(F)$ . For each of the gaps, the proof of the lower bound employs techniques previously used for experimental design, hashing, derandomization, and cryptography [13, 4, 15, 14]. We also adapt the proof of the upper bound in [12] to show that our lower bounds are sharp up to a factor of $r(1+o(1))$ .

In order to obtain the lower bounds, we prove the following three lemmas which generalize results from [12] and [9]. For the rest of this section, we assume that $p$ is a prime number.

Lemma 10.

Fix $n\geq 2$ , suppose that $z_{1},\dots,z_{r}\in\left\{0,\dots,p-1\right\}$ , and let $u_{1},\dots,u_{r}$ each be chosen uniformly at random from $\left\{0,\dots,p-1\right\}^{n}$ . For any $s\in\left\{1,\dots,p-1\right\}^{n}$ , we have

\Pr(s\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r)=\frac{1}{p^{r}}

Proof.

We have

	$\displaystyle\Pr(s\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r)=$
	$\displaystyle\Pr(s_{j}u_{i,j}=z_{i}-\sum_{k\neq j}s_{k}u_{i,k}\mod p\text{ for all }1\leq i\leq r)=$
	$\displaystyle\Pr(u_{i,j}=(z_{i}-\sum_{k\neq j}s_{k}u_{i,k})s_{j}^{-1}\mod p\text{ for all }1\leq i\leq r)=$
	$\displaystyle\frac{1}{p^{r}}.$

∎

Lemma 11.

Fix $n\geq 2$ , and let $u_{1},\dots,u_{r}$ each be chosen uniformly at random from $\left\{0,\dots,p-1\right\}^{n}$ . For any $s,t\in\left\{1,\dots,p-1\right\}^{n}$ that are not multiples of each other mod $p$ and for any $z_{1},\dots,z_{r}\in\left\{0,\dots,p-1\right\}$ , we have

\Pr(t\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r\text{ }|\text{ }s\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r)=\frac{1}{p^{r}}.

Proof.

By Lemma 10 and the definition of conditional probability, we have

	$\displaystyle\Pr(t\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r\text{ }\|\text{ }s\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r)=$
	$\displaystyle\frac{\Pr(t\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r\text{ }\wedge\text{ }s\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r)}{\Pr(s\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r)}=$
	$\displaystyle p^{r}\Pr(t\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r\text{ }\wedge\text{ }s\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r).$

Moreover

	$\displaystyle\Pr(t\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r\text{ }\wedge\text{ }s\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r)=$
	$\displaystyle\frac{\|\left\{(u_{1},\dots,u_{r}):\text{ }t\cdot u_{i}=z_{i}\mod p\text{ }\wedge\text{ }s\cdot u_{i}=z_{i}\mod p\text{ }\wedge\text{ }u_{i}\in\left\{0,\dots,p-1\right\}^{n}\text{ for all }1\leq i\leq r\right\}\|}{p^{nr}}.$

In order to calculate

|\left\{(u_{1},\dots,u_{r}):\text{ }t\cdot u_{i}=z_{i}\mod p\text{ }\wedge\text{ }s\cdot u_{i}=z_{i}\mod p\text{ }\wedge\text{ }u_{i}\in\left\{0,\dots,p-1\right\}^{n}\text{ for all }1\leq i\leq r\right\}|,

we must find the number of solutions $(u_{1},\dots,u_{r})$ to the system of equations $t\cdot u_{i}=z_{i}\mod p$ and $s\cdot u_{i}=z_{i}\mod p$ for all $1\leq i\leq r$ .

We form an augmented matrix $M$ with $2r$ rows and $rn+1$ columns from this system of equations. From left to right, the entries of row $i$ are $(i-1)n$ zeroes, then $s$ , then $(r-i)n$ zeroes, then $z_{i}$ for each $1\leq i\leq r$ . The entries of row $r+i$ are $(i-1)n$ zeroes, then $t$ , then $(r-i)n$ zeroes, then $z_{i}$ for each $1\leq i\leq r$ .

We row-reduce $M$ . Since $s$ and $t$ are not multiples of each other mod $p$ , $M$ has $2r$ pivot entries. Therefore the system of equations has $2r$ dependent variables and $(n-2)r$ independent variables. There are $p$ choices for each of the independent variables, and the dependent variables are determined by the values of the independent variables, so there are $p^{(n-2)r}$ solutions to the system of equations. Thus

\Pr(t\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r\text{ }|\text{ }s\cdot u_{i}=z_{i}\mod p\text{ for all }1\leq i\leq r)=\frac{p^{r}}{p^{2r}}=\frac{1}{p^{r}}.

∎

Lemma 12.

For any subset $S\subset\{1,\dots,p-1\}^{n}$ , there exists $u=(u_{1},\dots,u_{r})$ with $u_{1},\dots,u_{r}\in\{0,\dots,p-1\}^{n}$ such that for all $z=(z_{1},\dots,z_{r})\in\{0,\dots,p-1\}^{r}$ ,

\left\lvert\{x\in S:x\cdot u_{i}\equiv z_{i}\pmod{p}\text{ for all }1\leq i\leq r\}\right\rvert\leq\frac{|S|}{p^{r}}+2\sqrt{|S|}.

Proof.

Suppose that $S$ is any subset of $\{1,\dots,p-1\}^{n}$ , and let $u_{1},\dots,u_{r}$ each be chosen uniformly at random from $\{0,\dots,p-1\}^{n}$ . For each $z\in\{0,\dots,p-1\}^{r}$ , let $T_{z}$ be the set of $x\in S$ for which $x\cdot u_{i}=z_{i}$ for all $i$ . By linearity of expectation, we have $\mathbb{E}(|T_{z}|)=\frac{|S|}{p^{r}}$ for all $z$ .

Consider an arbitrary $z\in\{0,\dots,p-1\}^{r}$ . For each $s\in S$ , define the indicator random variable $X_{s,z}$ such that $X_{s,z}=1$ if $s\cdot u_{i}=z_{i}$ for all $1\leq i\leq r$ , and $X_{s,z}=0$ otherwise. If $s,t\in S$ are not multiples of each other mod $p$ , then $\mathrm{Cov}(X_{s,z},X_{t,z})=0$ by Lemmas 10 and 11. If $s$ and $t$ are multiples of each other with $s\neq t$ , then $\mathrm{Cov}(X_{s,z},X_{t,z})=\mathbb{E}(X_{s,z}X_{t,z})-\mathbb{E}(X_{s,z})\mathbb{E}(X_{t,z})$ .

If $z$ contains any nonzero $z_{i}$ , then $\mathbb{E}(X_{s,z}X_{t,z})=0$ , giving $\mathrm{Cov}(X_{s,z},X_{t,z})=-\frac{1}{p^{2r}}$ . Thus,

	$\displaystyle\mathrm{Var}(\|T_{z}\|)$	$\displaystyle=\mathrm{Var}\left(\sum_{s\in S}X_{s,z}\right)=\sum_{s\in S}\mathrm{Var}(X_{s,z})+\sum_{s\neq t}\mathrm{Cov}(X_{s,z},X_{t,z})$
		$\displaystyle\leq\sum_{s\in S}\mathrm{Var}(X_{s,z})=\|S\|\left(\frac{1}{p^{r}}-\frac{1}{p^{2r}}\right)<\frac{\|S\|}{p^{r}}.$

By Chebyshev’s inequality, $\mathbb{P}\left(|T_{z}|\geq\frac{|S|}{p^{r}}+2\sqrt{|S|}\right)\leq\frac{1}{4p^{r}}$ .

Otherwise, $\mathbb{E}(X_{s,z}X_{t,z})=\frac{1}{p^{r}}$ , giving $\mathrm{Cov}(X_{s,z},X_{t,z})=\frac{1}{p^{r}}-\frac{1}{p^{2r}}<\frac{1}{p^{r}}$ . Note that there are at most $(p-2)|S|$ ordered pairs $(s,t)$ for which $s$ and $t$ are multiples of each other $\pmod{p}$ with $s\neq t$ . Thus,

	$\displaystyle\mathrm{Var}(\|T_{z}\|)$	$\displaystyle=\mathrm{Var}\left(\sum_{s\in S}X_{s,z}\right)=\sum_{s\in S}\mathrm{Var}(X_{s,z})+\sum_{s\neq t}\mathrm{Cov}(X_{s,z},X_{t,z})$
		$\displaystyle\leq\sum_{s\in S}\mathrm{Var}(X_{s,z})+\frac{(p-2)\|S\|}{p^{r}}<\frac{\|S\|}{p^{r}}+\frac{(p-2)\|S\|}{p^{r}}<\frac{\|S\|}{p^{r-1}}.$

By Chebyshev’s inequality, $\mathbb{P}\left(|T_{z}|\geq\frac{|S|}{p^{r}}+2\sqrt{|S|}\right)\leq\frac{1}{4p^{r-1}}$ .

By the union bound,

\mathbb{P}\left(\forall z:|T_{z}|\leq\frac{|S|}{p^{r}}+2\sqrt{|S|}\right)\geq 1-\frac{p^{r}-1}{4p^{r}}-\frac{1}{4p^{r-1}}=\frac{3p^{r}-p+1}{4p^{r}}>\frac{1}{2}.

Thus, the conditions are satisfied with probability greater than $\frac{1}{2}$ when $u$ is chosen randomly, so there must always exist $u$ satisfying the conditions. ∎

Next we prove the lower bound on the maximum possible multiplicative gap between $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ and $\operatorname{opt}_{\operatorname{std}}(F)$ .

Theorem 13.

For all $M>2r$ and infinitely many $k$ , there exists a set $F$ of functions from a set $X$ to a set $Y$ with $|Y|=k$ such that $\operatorname{opt}_{\operatorname{std}}(F)=M$ and

\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))\geq(1-o(1))\left(|Y|^{r}\ln{|Y|}\right)(\operatorname{opt}_{\operatorname{std}}(F)-2r).

Proof.

Fix $n\geq 3$ and $p\geq 5$ . For all $a\in\{0,\dots,p-1\}^{n}$ , we define $f_{a}:\{0,\dots,p-1\}^{n}\rightarrow\{0,\dots,p-1\}$ so that $f_{a}(x)=a\cdot x\pmod{p}$ and define $F_{L}(p,n)=\{f_{a}:a\in\{0,\dots,p-1\}^{n}\}$ . It is known that $\operatorname{opt}_{\operatorname{std}}(F_{L}(p,n))=n$ for all primes $p$ and $n>0$ [17, 2, 3, 12].

We now determine a bound on $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F_{L}(p,n)))$ . Let $S=\{1,\dots,p-1\}^{n}$ , so $|S|=(p-1)^{n}$ . Let $R_{1}=\{f_{a}:a\in S\}\subset F_{L}(p,n)$ . In each round $t>1$ , the adversary will create a list $R_{t}$ of members of $\{f_{a}:a\in S\}$ that are consistent with its previous answers. They will always answer no and choose $(x_{1},\dots,x_{r})$ that minimizes

\max_{(y_{1},\dots,y_{r})}\left|R_{t}\cap\{f:f(x_{i})=y_{i}\text{ for all }1\leq i\leq r\}\right|.

By Lemma 12, we have

R_{t+1}\geq|R_{t}|-\frac{|R_{T}|}{p^{r}}-2\sqrt{|R_{t}|}\geq|R_{t}|-\frac{|R_{t}|}{p^{r}}-\frac{2|R_{t}|}{p^{r}\sqrt{\ln{p}}}=\left(1-\frac{1+\frac{2}{\sqrt{\ln{p}}}}{p^{r}}\right)|R_{t}|

as long as $|R_{t}|\geq p^{2r}\ln{p}$ . Thus, we have $|R_{t}|\geq\left(1-\frac{1+\frac{2}{\sqrt{\ln{p}}}}{p^{r}}\right)^{t-1}(p-1)^{n}$ . Therefore, whenever $\left(1-\frac{1+\frac{2}{\sqrt{\ln{p}}}}{p^{r}}\right)^{t-1}(p-1)^{n}\geq p^{2r}\ln{p}$ , the adversary can guarantee $t$ wrong guesses. This is true for $t=(1-o(1))np^{r}\ln{p}$ , which gives the desired result. ∎

Remark 14.

Since we have the trivial inequality $\operatorname{opt}_{\operatorname{amb},r}(F)\geq\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ because the learner has strictly more information in the scenario on the right hand side, we also have $\operatorname{opt}_{\operatorname{amb},r}(F)\geq(1-o(1))\left(|Y|^{r}\ln{|Y|}\right)(\operatorname{opt}_{\operatorname{std}}(F)-2r)$ for the families $F=F_{L}(p,n)$ .

Next we establish a similar upper bound relating $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ and $\operatorname{opt}_{\operatorname{std}}(F)$ . For this bound, we use the fact that for all sets $F$ of functions $f:X\to Y$ , we have $\operatorname{opt}_{\operatorname{std}}(\operatorname{CART}_{r}(F))=\operatorname{opt}_{\operatorname{std}}(F)$ . We also use the bound $\operatorname{opt}_{\operatorname{weak}}(F)\leq(1+o(1))(|Y|\ln|Y|)\operatorname{opt}_{\operatorname{std}}(F)$ which was proved in [12].

Theorem 15.

For any set $F$ of functions from some set $X$ to $\{0,1,\dots,k-1\}$ and for any $r\geq 1$ ,

\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))\leq(1+o(1))\left(|Y|^{r}r\ln{|Y|}\right)\operatorname{opt}_{\operatorname{std}}(F).

Proof.

Substituting $\operatorname{CART}_{r}(F)$ for $F$ (and therefore setting $Y^{r}$ in place of $Y$ ) in the upper bound from [12] and using the fact that $\operatorname{opt}_{\operatorname{std}}(\operatorname{CART}_{r}(F))=\operatorname{opt}_{\operatorname{std}}(F)$ , we get

	$\displaystyle\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$	$\displaystyle\leq(1+o(1))(\|Y\|^{r}\ln\left(\|Y\|^{r}\right))\operatorname{opt}_{\operatorname{std}}(\operatorname{CART}_{r}(F))$
		$\displaystyle=(1+o(1))(\|Y\|^{r}r\ln\|Y\|)\operatorname{opt}_{\operatorname{std}}(F).$

∎

Remark 16.

Theorem 13 demonstrates that the upper bound in Theorem 15 is sharp up to a factor of $r(1+o(1))$ .

Next we prove an upper bound for $\operatorname{opt}_{\operatorname{amb,r}}(F)$ using a method from [12]. Like the last bound, this one is sharp up to a factor of $r(1+o(1))$ .

Theorem 17.

For any set $F$ of functions from some set $X$ to $\{0,1,\dots,k-1\}$ and for any $r\geq 1$ ,

\operatorname{opt}_{\operatorname{amb,r}}(F)\leq(1+o(1))\left(|Y|^{r}r\ln{|Y|}\right)\operatorname{opt}_{\operatorname{std}}(F).

Proof.

Consider an algorithm $B$ for the $r$ -input delayed, ambiguous reinforcement model which uses a worst-case optimal algorithm $A_{s}$ for the standard model and maintains copies of $A_{s}$ which are given different inputs and answers. In each round, $B$ chooses a prediction by taking a weighted vote over the predictions of the copies.

Fix $\alpha=\frac{1}{k^{r}\ln{k}}$ . Each copy of $A_{s}$ gets a weight, where the current weight is $\alpha^{x}$ if the copy has contributed to $B$ making $x$ mistakes in the earlier weighted votes. $B$ uses these weights to make a prediction for the current round by taking a weighted vote over the predictions of each copy for the outputs of all $r$ inputs. In case of ties, the winner is chosen uniformly at random among the predictions that tied for the highest weight.

At the beginning, $B$ starts with one copy of $A_{s}$ . Whenever it gets a wrong answer for the outputs of the $r$ inputs in the current round, any copy of $A_{s}$ which predicted an answer that did not win the weighted vote is rewound as if the round did not happen, so they forget the input and their answer. The copies of $A_{s}$ that predicted the wrong answer which won the weighted vote are cloned to make $k^{r}-1$ copies, and each copy is given a different answer for the outputs of the $r$ inputs, which differs from the wrong answer which won the weighted vote.

If $W_{t}$ is the total weight of the copies of $A_{s}$ before round $t$ , we must have $W_{t}\geq\alpha^{\operatorname{opt}_{\operatorname{std}}(F)}$ since one copy of $A_{s}$ always gets the correct answers. Moreover, if $B$ makes a mistake in round $t$ , then copies of $A_{s}$ with total weight at least $\frac{W_{t}}{k^{r}}$ are cloned to make $k^{r}-1$ copies which each have weight $\alpha$ times their old weight. This implies that $W_{t+1}\leq(1-\frac{1}{k^{r}})W_{t}+\frac{1}{k^{r}}(\alpha(k^{r}-1)W_{t})<(1-\frac{1}{k^{r}})W_{t}+\alpha W_{t}$ .

Thus after $B$ has made $x$ mistakes, we have $W_{t}<(1-\frac{1}{k^{r}}+\alpha)^{x}<e^{-(\frac{1}{k^{r}}-\alpha)x}$ . Therefore $e^{-(\frac{1}{k^{r}}-\alpha)x}>\alpha^{\operatorname{opt}_{\operatorname{std}}(F)}$ , so $x<\frac{\ln(\frac{1}{\alpha})\operatorname{opt}_{\operatorname{std}}(F)}{\frac{1}{k^{r}}-\alpha}$ , which implies the desired bound. ∎

5 New Models for Permutation Functions

In this section, we define and explore new models where the family of possible functions $F$ is a set of permutations of length $n$ and where the learner tries to guess information about the relative orders of inputs.

5.1 The Order Model

We first define a new variant model called the order model.

Definition 18.

In the order model, for a set $F$ of permutations of $n$ numbers, the learner tries to guess a permutation function $f\in F.$ On each turn, the adversary chooses $r$ distinct inputs to $f$ , i.e., a set $S\subseteq[n]$ with $\absolutevalue{S}=r$ . The learner guesses the permutation of $\left\{1,\dots,r\right\}$ which is order-isomorphic to the outputs of $f$ on the given inputs.

Under weak reinforcement, the adversary informs the learner if they made a mistake, and under strong reinforcement, the adversary gives the correct answer to the learner. We denote the worst-case amount of mistakes for the learner with weak reinforcement as $\operatorname{opt}_{\operatorname{weak,<,r}}(F)$ and with strong reinforcement as $\operatorname{opt}_{\operatorname{strong,<,r}}(F).$ Note that $\operatorname{opt}_{\operatorname{strong,<,r}}(F)\leq\operatorname{opt}_{\operatorname{weak,<,r}}(F).$ If $r=2,$ then strong and weak reinforcement are identical, so equality holds.

We first find an upper bound for the order model by presenting a strategy for the learner which is analogous to Theorem 6.

Theorem 19.

For $r>1,$ $\operatorname{opt}_{\operatorname{weak,<,r}}(F)<r!\ln\absolutevalue{F}.$

Proof.

For each input that the adversary gives, the learner can pick the answer that corresponds to the most possible permutations. After each incorrect guess, at least an $\frac{1}{r!}$ fraction of the previously possible permutations get eliminated. Therefore, the number of incorrect guesses, and consequently the number of mistakes, is at most $\ln_{\frac{r!}{r!-1}}\absolutevalue{F}<r!\ln\absolutevalue{F}$ . ∎

When $F=S_{n}$ , Theorem 19 shows that $\operatorname{opt}_{\operatorname{weak,<,2}}(S_{n})<2n\ln{n}.$ We find a lower bound on $\operatorname{opt}_{\operatorname{weak,<}}(S_{n})$ which is within a factor of $2+o(1)$ of the upper bound. In order to prove this bound, we define a function called $p(n)$ and prove a lemma about it.

Definition 20.

For $n\geq 1,$ we let $v_{n}:=\left\lfloor\log_{2}n\right\rfloor$ and define $p(n):=\displaystyle\sum_{1\leq m\leq n}v_{m}$ .

Lemma 21.

For $n\geq 1,$ $p(n)=(n+1)v_{n}-2(2^{v_{n}}-1).$

Proof.

We prove this by induction. The base case $n=1$ holds.

If $n$ is not a power of two, then $v_{n-1}=v_{n},$ so $p(n)=p(n-1)+v_{n}=(nv_{n}-2(2^{v_{n}}-1))+v_{n}=(n+1)v_{n}-2(2^{v_{n}}-1)$ , as desired.

Otherwise, $n=2^{n}$ is a power of two, so $v_{n-1}=v_{n}-1$ and $p(2^{v_{n}})=p(2^{v_{n}}-1)+v_{n}=(2^{v_{n}}(v_{n}-1)-2(2^{v_{n}-1}-1))+v_{n}=(2^{v_{n}}+1)v_{n}-2(2^{v_{n}}-1)$ , as desired. ∎

Now we present a strategy resembling insertion sort for the adversary which achieves a lower bound of $p(n).$

Theorem 22.

Under the order model, the adversary can achieve $\operatorname{opt}_{\operatorname{weak,<,2}}(S_{n})\geq p(n)$ for all $n\geq 2$ .

Proof.

The adversary withholds any inquiries about $f(i)$ until the order of the smaller inputs $j<i$ is known. We use induction to show the desired bound. The base case of $n=2$ clearly holds, since $p(2)=1$ .

For $n>2,$ by the inductive hypothesis, the adversary can force the learner to make at least $p(n-1)$ mistakes without learning anything about $f(n).$ Assume without loss of generality that $f(1),\dots,f(n-1)$ are in increasing order.

The adversary then prolongs the learner from finding the position of $f(n)$ by making the learner do a binary search. Specifically, if at some point the learner’s bounds on $f(n)$ are $a<f(n)\leq b,$ the adversary asks about $(n,m)$ where $m$ is $\frac{a+b}{2}$ rounded to the nearest integer, and says no to the learner’s prediction. In this way, the adversary ensures that the number of possible values for $f(n)$ is at least $\left\lfloor\frac{b-a}{2}\right\rfloor$ after the learner’s guess. Since the number of possible values starts at $n,$ the adversary will be able to guarantee $v_{n}$ mistakes to find the value of $f(n).$ Thus, $\operatorname{opt}_{\operatorname{weak,<,2}}(S_{n})\geq\operatorname{opt}_{\operatorname{weak,<,2}}(S_{n-1})+v_{n}\geq p(n-1)+v_{n}=p(n)$ , as desired. ∎

We also get a lower bound on $\operatorname{opt}_{\operatorname{weak,<,r}}(S_{n})$ for $r>2$ with a strategy that resembles merge sort. This lower bound is within a $(1+o(1))r\ln{r}$ factor of the upper bound $\operatorname{opt}_{\operatorname{weak,<,r}}(S_{n})<(1-o(1))r!n\ln{n}$ which follows from Theorem 19.

Theorem 23.

The adversary can achieve $\operatorname{opt}_{\operatorname{weak,<,r}}(S_{n})\geq(1-o(1))(r-1)!n\log_{r}n$ as $n\to\infty$ and then $r\to\infty$ .

Proof.

We use strong induction on $n.$

First, the adversary divides $[n]$ into $\left\lfloor\frac{n}{r}\right\rfloor$ sets $S_{i}$ each of size $r$ (ignoring any remaining elements). For each $i,$ the adversary repeatedly asks for ordering of $S_{i},$ saying “NO” each time until the order is known by the learner. This takes a total of $\left\lfloor\frac{n}{r}\right\rfloor(r!-1)$ mistakes.

Then, the adversary forms sets $C_{j}$ from the relative $j^{th}$ elements of each set, and uses the induction hypothesis on $n$ ; note that knowing any of $C_{j}$ ’s orders does not eliminate possibilities for the orders of other $C_{i}$ . This gives a recursion of the form $\operatorname{opt}_{\operatorname{weak,<,r}}(S_{n})\geq(1-o(1))n(r-1)!+r\operatorname{opt}_{\operatorname{weak,<,r}}(S_{\left\lfloor\frac{n}{r}\right\rfloor})=(1-o(1))(r-1)!n\log_{r}(n)$ , as desired. ∎

5.2 The Comparison Model

We define a second variant called the comparison model.

Definition 24.

In the comparison model, for a set $F$ of permutations of $n$ numbers, the learner tries to guess a permutation function $f\in F.$ On each turn, the adversary chooses $r$ distinct pairs of inputs to $f$ , i.e., a set $S\subseteq\{(i,j):i,j\in[n],i\neq j\}$ with $\absolutevalue{S}=r$ . The learner guesses for each pair $(i,j)$ whether or not $f(i)<f(j)$ .

Under weak reinforcement, the adversary informs the learner if they made a mistake, and under strong reinforcement, the adversary gives the correct choices to the learner. We denote the worst-case amount of mistakes for the learner with weak reinforcement as $\operatorname{opt}_{\operatorname{weak,c,r}}(F)$ and with strong reinforcement as $\operatorname{opt}_{\operatorname{strong,c,r}}(F).$ Note that $\operatorname{opt}_{\operatorname{strong,c,r}}(F)\leq\operatorname{opt}_{\operatorname{weak,c,r}}(F).$ If $r=1,$ then both sides of the equation are equal to $\operatorname{opt}_{\operatorname{strong,<,2}}(F).$

We similarly get the following upper bound.

Theorem 25.

If $F\subseteq S_{n}$ , $\operatorname{opt}_{\operatorname{weak,c,r}}(F)<2^{r}\cdot\ln|F|.$

Proof.

For each input that the adversary gives, the learner can pick the answer that corresponds to the most possible permutations. After each incorrect guess, at least a $\frac{1}{2^{r}}$ fraction of the previously possible permutations get eliminated. Therefore, the number of incorrect guesses, and consequently the number of mistakes, is at most $\log_{\frac{2^{r}}{2^{r}-1}}(|F|)<2^{r}\ln|F|.$ ∎

We present a strategy imitating quicksort to obtain the following lower bound.

Theorem 26.

The adversary can achieve $\operatorname{opt}_{\operatorname{weak,c,r}}(S_{n})\geq(1-o(1))\frac{2^{r}}{r}n\log_{2}n$ as $n\to\infty$ and then $r\to\infty$ .

Proof.

The adversary first splits $[n-1]$ into $\left\lfloor\frac{n-1}{r}\right\rfloor$ sets of size $r$ with some leftover elements. For every such set $\absolutevalue{S}$ with $\absolutevalue{S}=r$ , the adversary then queries the set of pairs $\quantity{(s,n):s\in S}$ at least $2^{r}$ times, responding no each time until the answer is forced. After all queries, the learner has made at least $\left\lfloor\frac{n-1}{r}\right\rfloor(2^{r}-1)=(1-o(1))\frac{2^{r}}{r}n$ mistakes.

The learner now at most knows the set of inputs $i$ for which $\pi(i)<\pi(n)$ . The adversary can therefore force the learner to make at least $\operatorname{opt}_{\operatorname{weak,c,r}}(S_{\pi(n)-1})+\operatorname{opt}_{\operatorname{weak,c,r}}(S_{n-\pi(n)})$ more mistakes.

The total number of mistakes is thus at least $2\quantity((1-o(1))\frac{2^{r}}{r}\cdot\frac{n}{2}(\log_{2}n-1))+(1-o(1))\frac{2^{r}}{r}n=(1-o(1))\frac{2^{r}}{r}n\log_{2}n$ in total, as desired. ∎

5.3 The Selection Model

We define another variant called the selection model.

Definition 27.

In the selection model, for a set $F$ of permutations of $n$ numbers, the learner tries to guess a permutation function $f\in F.$ On each turn, the adversary chooses $r$ (not necessarily distinct) inputs to $f$ , and the learner guesses the input $x$ among them that maximizes $f(x)$ .

Under weak reinforcement, the adversary informs the learner if they made a mistake, and under strong reinforcement, the adversary gives the correct choices to the learner. We denote the worst-case amount of mistakes for the learner with weak reinforcement as $\operatorname{opt}_{\operatorname{weak,s,r}}(F)$ and with strong reinforcement as $\operatorname{opt}_{\operatorname{strong,s,r}}(F).$ Note that $\operatorname{opt}_{\operatorname{strong,s,r}}(F)\leq\operatorname{opt}_{\operatorname{weak,s,r}}(F).$ If $r=2,$ then both sides of the equation are equal to $\operatorname{opt}_{\operatorname{strong,<,2}}(F).$

Note that because the inputs the adversary chooses do not have to be distinct, the adversary can effectively ask about any set of inputs of size at most $r$ by using duplicates.

Theorem 28.

For $r>1$ ,

\operatorname{opt}_{\operatorname{weak,s,r}}(F)<r\ln|F|.

Proof.

For each input that the adversary gives, the learner can pick the input that is maximal for the most possible remaining permutations. After each incorrect guess, at least a $\frac{1}{r}$ fraction of the previously possible permutations get eliminated. Therefore, the number of incorrect guesses, and consequently the number of mistakes, is at most $\log_{\frac{r}{r-1}}(|F|)<r\ln|F|.$ ∎

We again imitate merge sort to provide a strategy for the adversary.

Theorem 29.

The adversary can achieve $\operatorname{opt}_{\operatorname{weak,s,r}}(S_{n})\geq(1-o(1))nr\log_{r}(n)/2$ as $n\to\infty$ and then $r\to\infty$ .

Proof.

We use strong induction on $n$ .

First, the adversary divides $[n]$ into $r$ sets $S_{i}$ each of size $\left\lfloor\frac{n}{r}\right\rfloor$ (ignoring any remaining elements). For each $i,$ the adversary uses the optimal strategy for $\left\lfloor\frac{n}{r}\right\rfloor$ on $S_{i}$ , forcing a total of at least $r\operatorname{opt}_{\operatorname{weak,s,r}}(S_{\left\lfloor\frac{n}{r}\right\rfloor})$ mistakes.

Then, the adversary repeatedly asks about the elements $x_{i}$ of each corresponding $S_{i}$ for $1\leq i\leq r$ for which $f(x_{i})$ is maximal, saying “NO” each time until their maximal element, say $x_{m}$ , is known. Then, the adversary deletes $x_{m}$ from $S_{m}$ and repeats this process until all sets are empty, which takes $r\left\lfloor\frac{n}{r}\right\rfloor$ iterations in total.

The number of tries it takes the learner each time to guess correctly is the number of possibilities for the maximal remaining element, which is the number of nonempty $S_{i}$ ’s remaining. Therefore, until one of the $S_{i}$ ’s is empty, which takes at least $|S_{i}|=\left\lfloor\frac{n}{r}\right\rfloor$ turns, the adversary can force the learner to make at least $(r-1)$ mistakes, and after that $(r-2)$ , and so on all the way to $0$ mistakes when all but one $S_{i}$ is empty. Therefore, the learner throughout the process makes an average of at least $(r-1)/2$ mistakes per iteration, for $r\left\lfloor\frac{n}{r}\right\rfloor\cdot(r-1)/2=(1-o(1))nr/2$ mistakes total.

Thus, $\operatorname{opt}_{\operatorname{weak,s,r}}(S_{n})\geq(1-o(1))nr/2+r\operatorname{opt}_{\operatorname{weak,s,r}}(S_{\left\lfloor\frac{n}{r}\right\rfloor})=(1-o(1))nr\log_{r}(n)/2$ , as desired. ∎

5.4 The Relative Position Model

Next we define a variant called the relative position model.

Definition 30.

In the relative position model, for a set $F$ of permutations of $n$ numbers, the learner tries to guess a permutation function $f\in F.$ On each turn, the adversary chooses a set $S$ of $r$ distinct inputs to $f$ and an element $x\notin S,$ and asks about the pair $(x,S).$ The learner guesses the relative position of $f(x)$ in the permutation of $\left\{1,\dots,r+1\right\}$ which is order-isomorphic to the outputs of $f$ on $\quantity{x}\cup S.$

Under weak reinforcement, the adversary informs the learner if they made a mistake, and under strong reinforcement, the adversary gives the correct position to the learner. We denote the worst-case amount of mistakes for the learner with weak reinforcement as $\operatorname{opt}_{\operatorname{weak,p,r}}(F)$ and with strong reinforcement as $\operatorname{opt}_{\operatorname{strong,p,r}}(F).$ Note that $\operatorname{opt}_{\operatorname{strong,p,r}}(F)\leq\operatorname{opt}_{\operatorname{weak,p,r}}(F).$ If $r=1,$ then both sides of the equation are equal to $\operatorname{opt}_{\operatorname{strong,<,2}}(F).$

We again imitate Theorem 6 to obtain an upper bound for the relative position model.

Theorem 31.

If $F\subseteq S_{n}$ , $\operatorname{opt}_{\operatorname{weak,p,r}}(F)<(r+1)\ln|F|.$

Proof.

For each input that the adversary gives, the learner can pick the answer that corresponds to the most possible permutations. After each incorrect guess, at least a $\frac{1}{r+1}$ fraction of the previously possible permutations get eliminated. Therefore, the number of incorrect guesses, and consequently the number of mistakes, is at most $\log_{\frac{r+1}{r}}(|F|)<(r+1)\ln|F|.$ ∎

As with Theorem 22, we use a strategy resembling insertion sort to obtain the following result.

Theorem 32.

Under the relative position model, $\lim_{r\to\infty}\lim_{n\to\infty}\frac{\operatorname{opt}_{\operatorname{weak,p,r}}(S_{n})}{rn\ln n}=1$ .

Proof.

The upper bound follows from Theorem 31. For the lower bound, we use strong induction on $n.$ The adversary withholds any inquiries about $f(i)$ until the order of the smaller inputs $j<i$ is known. Assume without loss of generality that $f(1),\dots,f(n)$ are in increasing order. Let $g_{i}$ be the function on $[n]$ that maps $m$ to $1$ if $m\geq i$ and $0$ otherwise, and let $G$ be the family of these functions. Note that these functions are non-decreasing.

The remaining problem for the learner is equivalent to guessing $g_{f(n+1)},$ where the adversary queries $r$ values at a time. Thus by Theorem 4, the adversary can force the learner to make at least $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(G))=(1-o(1))r\ln n$ mistakes on $g_{f(n+1)},$ as desired. ∎

In a similar fashion, we can prove a similar result for pattern-avoiding permutations.

Theorem 33.

If $S_{n,\pi}$ is the set of $\pi$ -avoiding permutations of length $n$ , then $\operatorname{opt}_{\operatorname{weak,p,r}}(S_{n,\pi})\geq(1-o(1))rn\ln(k-1)$ as $n\to\infty$ and then $r\to\infty$ , where $k=|\pi|$ denotes the length of $\pi$ .

Proof.

The adversary withholds any inquiries about $f(i)$ until the order of the smaller inputs $j<i$ is known. We use induction to show the desired bound.

Let $N$ be the set of possible values of $f(n+1)$ . Any number less than $\pi(k)$ or more than $n-k+\pi(k)$ must be in $N$ as it would not be able to form the permutation pattern $\pi$ , so $|N|\geq\pi(k)+k-1-\pi(k)=k-1$ . Let the elements of $N$ in increasing order be $s_{1},\dots,s_{|N|}$ .

Let $g_{i}$ be the function on $[|N|]$ that maps $m$ to $1$ if $m\geq s_{i}$ and $0$ otherwise, and let $G$ be the family of these functions. Note that these functions are non-decreasing.

When $\pi=I_{k}$ , the identity permutation, the size of $S_{n,\pi}$ is $(k-1)^{O(n)}$ [16]. The next result follows from combining Theorems 33 and 31.

Corollary 34.

For $S_{n,I_{k}}$ the family of $I_{k}$ -avoiding permutations of length $n$ , $\operatorname{opt}_{\operatorname{weak,p,r}}(S_{n,I_{k}})=\Theta(rn\ln k)$ as $n\to\infty$ and then $r\to\infty$ .

5.5 The Delayed Relative Position Model

In this subsection, we define delayed reinforcement for the relative position model.

Definition 35.

In the delayed relative position model, for a set $F$ of permutations of $n$ numbers, the learner tries to guess a permutation function $f\in F.$ On each turn, the adversary picks an input $x$ and proceeds to give the $r$ elements of a set $S$ one by one (with the requirement that $x\notin S$ ). After each of the adversary’s inquiries, the learner guesses either higher or lower. At the end of each round, the learner’s final guess for the relative position of $x$ in $S$ is one plus the number of times they said higher.

Under weak reinforcement, the adversary informs the learner if their final guess is incorrect, and under strong reinforcement, the adversary gives the correct position to the learner. We denote the worst-case amount of mistakes for the learner with weak reinforcement as $\operatorname{opt}_{\operatorname{wrpos},r}(F)$ and with strong reinforcement as $\operatorname{opt}_{\operatorname{srpos},r}(F).$ Note that $\operatorname{opt}_{\operatorname{srpos},r}(F)\leq\operatorname{opt}_{\operatorname{wrpos},r}(F),$ and that if $r=1,$ both sides of the equation are equal to $\operatorname{opt}_{\operatorname{strong,<,2}}(F).$

We state the analogs of the relative position model results for the delayed version.

Theorem 36.

Given a set of permutations $F\subseteq S_{n}$ and a positive integer $r$ , these analogs of the results in Section 5.4 hold:

•

Analog of Theorem 31: $\operatorname{opt}_{\operatorname{wrpos},r}(F)<2^{r}\ln|F|$ .
•

Analog of Theorem 32: $\lim_{r\to\infty}\lim_{n\to\infty}\frac{\operatorname{opt}_{\operatorname{wrpos},r}(S_{n})}{2^{r}n\ln n}=1$ .
•

Analog of Theorem 33: $\operatorname{opt}_{\operatorname{wrpos},r}(S_{n,\pi})\geq(1-o(1))2^{r}n\ln(k-1)$ as $n\to\infty$ and then $r\to\infty$ , where $k=|\pi|$ denotes the length of $\pi$ .
•

Analog of Corollary 34: $\operatorname{opt}_{\operatorname{wrpos},r}(S_{n,I_{k}})=\Theta(2^{r}n\ln k)$ as $n\to\infty$ and then $r\to\infty$ .

The proofs of these results are nearly identical to those of the previous section, with $r+1$ replaced by $2^{r}$ and cited results from Section 2 replaced with analogous results from Section 3, so we omit them here.

6 Future Work

In this final section, we outline some possible areas for future work based on the results in our paper.

In Sections 2 and 3, we mainly focused on the price of feedback for non-decreasing $F$ in the $r$ -input weak reinforcement model and the $r$ -input delayed, ambiguous reinforcement model. What bounds can be obtained for general sets of functions $F$ ? Moreover, is it possible to obtain a sharper bound on the maximum possible value of $\frac{\operatorname{opt}_{\operatorname{amb},r}(F)}{\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))}$ over all families of functions $F$ with $|F|>1$ ?

In a preliminary version of this paper ([8]), we found rough bounds for the $r$ -input weak reinforcement model and the $r$ -input delayed, ambiguous reinforcement model on non-decreasing multi-class functions. In particular we proved that $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))\leq{{r+k-1}\choose k-1}\ln(|F|)$ and $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))\geq\frac{1}{2^{k-2}}(1-o(1))r\ln(|F|)$ for any subset of functions $F$ of the non-decreasing functions from $X$ to $\{0,1,\dots,k-1\}$ . We also showed in [8] that $\operatorname{opt}_{\operatorname{amb},r}(F)<\min(k^{r}\ln(|F|),|F|)$ and $\operatorname{opt}_{\operatorname{amb},r}(F)\geq(1-o(1))2^{r-k+2}\ln(|F|)$ for all $X$ and $F$ . Since the preceding upper and lower bounds for multi-class functions have large gaps, a natural problem is to narrow these gaps.

In Theorem 23, we described an adversary strategy for the order model that gives a bound of $\operatorname{opt}_{\operatorname{weak,<,r}}(S_{n})\geq(1-o(1))(r-1)!n\log_{r}n$ , whereas Theorem 19 gives an upper bound of $\operatorname{opt}_{\operatorname{weak,<,r}}(S_{n})<r!\ln n!=(1-o(1))r!n\ln n$ . We conjecture that the latter bound is sharp, i.e., that $\operatorname{opt}_{\operatorname{weak,<,r}}(S_{n})=(1-o(1))r!n\ln n$ .

Acknowledgments

We would like to thank the MIT PRIMES-USA program and everyone involved for providing us with this research opportunity.

References

[1] P. Auer and P.M. Long “Structural Results About On-line Learning Models With and Without Queries” In Machine Learning 36, 1999, pp. 147–181
[2] P. Auer, P.M. Long, W. Maass and G.J. Woeginger “On the complexity of function learning” In Machine Learning 18, 1995, pp. 187–230
[3] A. Blum “On-Line Algorithms in Machine Learning” In Online Algorithms: The State of the Art, 1998, pp. 306–325
[4] J.L. Carter and M.N. Wegman “Universal Classes of Hash Functions (Extended Abstract)” In Proceedings of the Ninth Annual ACM Symposium on Theory of Computing, 1977, pp. 106–112
[5] K. Crammer and C. Gentile “Multiclass classification with bandit feedback using adaptive regularization” In Machine Learning 90, 2012, pp. 347–383
[6] V. Dani, T. Hayes and S. Kakade “Stochastic Linear Optimization under Bandit Feedback” In 21st Annual Conference on Learning Theory, 2008, pp. 355–366
[7] A. Daniely and T. Helbertal “The price of bandit information in multiclass online classification”, 2013 arXiv:1310.8378
[8] R. Feng, A. Lee and E. Slettnes “Results on various models of mistake-bounded online learning”, 2021 URL: https://math.mit.edu/research/highschool/primes/papers
[9] J. Geneson “A note on the price of bandit feedback for mistake-bounded online learning” In Theoretical Computer Science 874, 2021, pp. 42–45
[10] E. Hazan and S. Kale “NEWTRON: An efficient bandit algorithm for online multiclass prediction” In 25th Annual Conference on Neural Information Processing Systems, 2011, pp. 891–899
[11] N. Littlestone “Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm” In Machine Learning 2, 1988, pp. 285–318
[12] P.M. Long “New bounds on the price of bandit feedback for mistake-bounded online multiclass learning” In Theoretical Computer Science 808, 2020, pp. 159–163
[13] M. Luby and A. Wigderson “Pairwise Independence and Derandomization” In Found. Trends Theor. Comput. Sci. 1, 2006, pp. 237–301
[14] C.R. Rao “Factorial Experiments Derivable from Combinatorial Arrangements of Arrays” In Supplement to the Journal of the Royal Statistical Society 9, 1947, pp. 128–139
[15] C.R. Rao “Hypercubes of strength d leading to confounded designs in factorial experiments” In Bulletin of the Calcutta Mathematical Society 38, 1946, pp. 67–68
[16] A. Regev “Asymptotic values for degrees associated with strips of young diagrams” In Advances in Mathematics 41, 1981, pp. 115–136
[17] H. Shvaytser “Linear manifolds are learnable from positive examples” Unpublished manuscript, 1988

Sharp bounds on the price of bandit feedback for several models of mistake-bounded online learning

Abstract

1 Introduction

2 Bounds on optweak⁡(CARTr⁡(F))\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))

Definition 1.

Remark 2.

2.1 Bounds on optweak⁡(CARTr⁡(F))\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))

Theorem 3.

Proof.

Theorem 4.

Proof.

Theorem 5.

3 Bounds on optamb,r⁡(F)\operatorname{opt}_{\operatorname{amb},r}(F)

Theorem 6.

Proof.

Theorem 7.

Proof.

Theorem 8.

Theorem 9.

4 Comparing optweak⁡(CARTr⁡(F))\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F)) and optamb,r⁡(F)\operatorname{opt}_{\operatorname{amb},r}(F) to optstd⁡(F)\operatorname{opt}_{\operatorname{std}}(F) for multi-class functions

Lemma 10.

Proof.

Lemma 11.

Proof.

Lemma 12.

Proof.

Theorem 13.

Proof.

Remark 14.

Theorem 15.

Proof.

Remark 16.

Theorem 17.

Proof.

5 New Models for Permutation Functions

5.1 The Order Model

Definition 18.

Theorem 19.

Proof.

Definition 20.

Lemma 21.

Proof.

Theorem 22.

Proof.

Theorem 23.

Proof.

5.2 The Comparison Model

Definition 24.

Theorem 25.

Proof.

Theorem 26.

Proof.

5.3 The Selection Model

Definition 27.

Theorem 28.

Proof.

Theorem 29.

Proof.

5.4 The Relative Position Model

Definition 30.

Theorem 31.

Proof.

Theorem 32.

Proof.

Theorem 33.

Proof.

Corollary 34.

5.5 The Delayed Relative Position Model

Definition 35.

Theorem 36.

6 Future Work

Acknowledgments

References

2 Bounds on $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$

2.1 Bounds on $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$

3 Bounds on $\operatorname{opt}_{\operatorname{amb},r}(F)$

4 Comparing $\operatorname{opt}_{\operatorname{weak}}(\operatorname{CART}_{r}(F))$ and $\operatorname{opt}_{\operatorname{amb},r}(F)$ to $\operatorname{opt}_{\operatorname{std}}(F)$ for multi-class functions