Multiple sequences Prophet Inequality Under Observation Constraints.

In many applications, multiple data sequences are monitored sequentially with an aim to decide the best instant to terminate the observation procedure, and use the collected information to maximize an objective. Financial applications include the design of posted pricing mechanisms for auctions [1, 2, 3] and contention resolution schemes [4]. Recent engineering applications focus on the optimization of computer hardware performance, such as computational sprinting [5, 6], which provides a significant performance boost to microchips. These applications, among many others, have motivated a large body of work in optimal stopping theory, with the dynamic programming [7, Ch. 24] being the primary resolution method for a broad class of them.

In systems with multiple data sequences, it is not always possible to observe the current sample of each sequence due to resource limitations or other observation constraints [8, 9]. For example, when following a large number of auctions in parallel, it is difficult to analyze all offers at all times. In computational sprinting hardware mechanisms, a software predicts the performance boost of a microchip when we allow short-term overheating, but limited computational resources render it impossible to run the software for all microchips, at each instant. In both cases, only a smaller number of sequences can be processed at any given instant, but without restrictions on which they are. This type of observation constraints leads to a problem at the intersection of optimal stopping and multi-armed bandit theory [10].

In our problem, we assume that the data sequences comprise i.i.d. random variables that are independent of each other, and whose distributions are allowed to differ. Our constraint is that we can only observe a fixed-size subset of the sequences, at each instant. Our goal is to determine which sequences to observe at each instant, and when to stop sampling at each sequence, in order to maximize our reward, which is the sum of the expected values of the observations at the selected stopping times. This is a combined optimal stopping and sampling problem, which could be treated by dynamic programming with an exponential complexity on the length of the sequences. In order to reduce the complexity, we introduce a decoupling approach, based on the Prophet Inequality [3, 11, 12, 13, 14], that reduces the complexity to polynomial, and guarantees at least $0.745$ of the optimal expected reward of the joint problem.

The Prophet Inequality compares the expected value of the picked element at the optimal stopping time, for each sequence, with the expected value of its’ maximum element, by providing a tight lower bound on the ratio of the respective expected values. The lower bound is proven to be equal to $0.745$, for an i.i.d. sequence, in [3, 15], independent of the distribution and of the length of the sequence. This benchmark has motivated the design of stopping rules for our problem, which always satisfy the Prophet Inequality, although they might be sub-optimal in some cases. In [3, Corollary 4.7], an algorithm is provided for the computation of the thresholds of such a stopping rule, which is simpler compared to that of the single-sequence dynamic programming (Single-DP), although both methods have linear complexity on the length of the sequences. We refer to the algorithm in [3], as Prophet Inequality thresholding method (PI-thresholding).

The basic requirement of the decoupling approach, is the calculation of the optimal number of samples for each sequence, based on which we can apply Single-DP, or PI-thresholding on each sequence separately. This results in a constrained optimization problem that depends on the distributions of all sequences. The computation might be intensive for large sequences, thus, under some smoothness assumptions on the optimization objective, we develop an approximation technique for the number of samples, whose error converges to zero as the length of the sequences goes to infinity.

The paper is organized as follows. In Section II, we present the problem formulation, while in Section III we prove our first result, pertaining to the approximation ratio of the decoupling approach. Then, in Section IV, we describe the approximation technique for the computation of the optimal number of samples for each sequence. In our last Section V, we present computational examples, which support the efficiency of the decoupling approach, for both the Single-DP and the PI-thresholding method.

Let $(\mathbb{R}_{+},\mathcal{S})$ be an arbitrary measurable space and $(\Omega,\mathcal{F},\mathsf{P})$ be a probability space which hosts $M$ independent sequences of $n$ i.i.d. $\mathbb{R}_{+}$-valued random variables,

where $[n]:=\{1,\ldots,n\}$, $[M]:=\{1,\ldots,M\}$. We aim to find the stopping time $\tau_{i}\in[n]$, for each sequence $i\in[M]$, that attains

where $\mathcal{T}$ the class of all stopping times that take values in $[n]$. Formally, a stopping time is a random variable $\tau_{i}$, $i\in[M]$, for which the event $\{\tau_{i}=m\}$, $m\in[n]$, is fully determined by the observations up to time $m$. Since each sequence $i\in[M]$ is associated to a stopping time $\tau_{i}$, we make subsequent use of the vector of stopping times of all sequences, i.e., $T:=(\tau_{1},\ldots,\tau_{M})$.

The observations from each sequence $i\in[M]$ are made sequentially and our decision to stop at time $\tau_{i}$ and pick $X_{i}(\tau_{i})$ as our “best” choice for the optimization problem (1) is irrevocable, i.e., we cannot revisit samples we rejected, nor we can examine samples that follow after we stop. The Prophet inequality [3, 15] provides a tight lower bound of $0.745$ on the ratio of (1) over $\mathsf{E}[\max_{m\in[n]}X_{i}(m)]$, for each $i\in[M]$. By the term “tight”, we mean that for any $n$, there exists a distribution that satisfies the inequality by equality, [15].

The main aspect that distinguishes our problem from the relevant bibliography, is the introduction of observation constraints to our model, i.e., it is not possible to observe the current element of each sequence at each instant $m\in[n]$, but only from a subset of them of size $K<M$, independent of which they actually are. Hence, we denote by $R(m)$ the subset of sequences that are observed at time $m$, i.e.,

where

We say that $R$ is a sampling rule if, for every $m\in[n-1]$, the $R(m+1)$ is $\mathcal{F}^{R}(m)$-measurable, where $\mathcal{F}^{R}(m)$ is the $\sigma$-algebra generated by the observed elements up to time $m$ according to rule $R$, i.e.,

The policy $(R,T)$ belongs to the class $\mathcal{C}(K)$ if the number of sequences observed at each sampling instant is equal to $K$, i.e.,

Our goal is to find a policy $(R,T)\in\mathcal{C}(K)$ that optimizes the objective

under the assumption that for each $i\in[M]$, the i.i.d. sequence $X_{i}$ has a finite first moment.

The optimization problem (3) can be solved via dynamic programming, with a computational complexity of $O\left({M\choose K}^{n}\right)$. In order to reduce the complexity, we describe a decoupling approach that produces a $0.745$-constant approximation for (3), with complexity polynomial in $n$.

For any sampling rule $R$, we denote by

the total number of elements we have observed from sequence $i$ up to time $n$, and by $\tau^{R}_{i}$ the optimal stopping time of sequence $i$, associated with the sampling rule $R$. The decoupling approach consists of the following steps:

1. (i)

   Since for each $i\in[M],$ the sequence $X_{i}$ is i.i.d., it suffices to determine the number of observations $N_{i}(n)$ for sequence $i$, without pinpointing the exact times at which we observe. The values $\{N_{i}(n)\,:\,i\in[M]\}$ shall satisfy a particular optimization criterion, independent of $R$.

2. (ii)

   We design a sampling rule $R^{d}$ which guarantees $N_{i}(n)$ observations for each $i\in[M]$, and respects the sampling constraint (2), which implies

   	$\displaystyle N_{i}^{R^{d}}(n)$	$\displaystyle=N_{i}(n),\quad\forall\;i\in[M],$		(4)
   	$\displaystyle\sum_{i=1}^{M}N_{i}^{R^{d}}(n)$	$\displaystyle=Kn.$

3. (iii)

   Given the $N_{i}^{R^{d}}(n)$ observations for each $i\in[M]$, in order to determine the decoupled optimal stopping times $\tau_{i}^{R^{d}}$, $i\in[M]$, we can use either the single-sequence dynamic programming [7, Chapter 24], or the Prophet Inequality thresholding method in [3, Corollary 4.7]. The latter, although computationally simpler, it may provide a sub-optimal reward.

In order to formulate the optimization criterion that the $\{N_{i}(n):i\in[M]\}$ must satisfy, we note that since the elements of each sequence are i.i.d., all subsequences of $N_{i}(n)$ elements have the same statistical behavior, and thus, by Prophet inequality [3, 15], for $R^{d}$, and for each $i\in[M]$,

where $D:=0.745$. Thus, the $\{N_{i}(n):i\in[M]\}$ are chosen as the maximizers of

where

because by (5), the criterion (6) guarantees that

We focus on the computation of the maximizers $n_{1}^{*},\ldots,n_{M}^{*}$ of (6). For large $n$, finding the exact values is computationally demanding, and thus we suggest an approximation technique along with error guarantees.

For each $i\in[M]$, we restrict our attention to absolutely continuous random variables, whose probability density and cumulative distribution functions are denoted by $f_{i}$ and $F_{i}$, respectively. The density of the maximum of $n_{i}$ observations from sequence $i$ is denoted by $g_{i}$, i.e.,

The maximization problem (6) turns into Problem $\mathbf{P_{1}}$:

subject to

and $n\geq n_{i}\geq 0$, for all $i\in[M]$.

A solution approach to $\mathbf{P_{1}}$ is the exhaustive search, of computational complexity $O\left(n^{M}\right)$. Thus, in the following subsection, we introduce a computationally simpler approximation method, whose computational complexity is $O\left(2^{M}\right)$, and hence independent of $n$, with an approximation error converging to $0$ as $n\to\infty$.

We compare the expected reward of the joint dynamic programming problem (3) versus that of the decoupled problem. For the decoupled problem, we precompute the optimal number of observations for each sequence, and then on each individual sequence, we run either (i) a single-sequence dynamic programming (Single-DP), or (ii) a Prophet Inequality thresholding (PI-thresholding) method [3, Corollary 4.7].

We consider $M=3$ sequences that follow three different uniform distributions $U[0,3]$, $U[0.5,2.5]$, $U[1,2]$, which have the same mean but different variances, and of lengths $n=5,\ldots,10$. We consider two cases $K=1$ and $K=2$, and we plot the ratio of the expected reward of the decoupled problem, for both Single-DP and PI-thresholding, over that of the joint problem.

For $K=1$ the decoupling approach offers a very good approximation for the problem (3), with a ratio above $0.92$ for the Single-DP case, and above $0.91$ for the PI-thresholding, for all $n$. We also note that the ratio of the PI-thresholding method is no more than $1\%$ smaller than that of the Single-DP method which is always optimal for a single sequence.

For $K=2$ the ratio is smaller compared to the former case, but it is still a good approximation, with a ratio above $0.88$ for the Single-DP, and above $0.87$ for the PI-thresholding method, for all $n$. In this case the gap between Single-DP and PI-thresholding is larger, but no more than $10\%$.

The work was supported in part by NSF awards 2008125 and 1956384, through Coordinated Science Laboratory, University of Illinois at Urbana-Champaign.