How to sample and when to stop sampling: The generalized Wald problem and minimax policies

Acquiring information is expensive. Experimenters need to carefully choose how many units of each treatment to sample and when to stop sampling. This paper seeks to develop techniques for incorporating the cost of information into experimental design. Specifically, we focus our analysis of costly experimentation within the context of comparative trials where the aim is to determine the best of two treatments.

In the computer science literature, such experiments are referred to as A/B tests. Technology companies like Amazon, Google and Microsoft routinely run hundreds of A/B tests a week to evaluate product changes, such as a tweak to a website layout or an update to a search algorithm. However, experimentation is expensive, especially if the changes being tested are very small and require evaluation on large amounts of data; e.g., Deng et al. (2013) state that even hundreds millions of users were considered insufficient at Google to detect the treatment effects they were interested in. Clinical or randomized trials are another example of A/B tests. Even here, reducing experimentation costs is a key goal. For instance, this has been a major objective for the FDA since 2004 when it introduced the ‘Critical Path Initiative’ for streamlining drug development; this in turn led the FDA to promote sequential designs in clinical trials (see, e.g., US Food and Drug Admin., 2018, for the current guidance, which was influenced by the need to reduce experimentation costs). For this reason, many of the recent clinical trials, such as the ones used to test the effectiveness of Covid vaccines (e.g., Zaks, 2020), now use multi-stage designs where the experiment can be terminated early if a particularly positive or negative effect is seen in early stages.

In practice, the cost of experimentation directly or indirectly enters the researchers’ experimental design when they choose an implicit or explicit stopping time (note that we use stopping time interchangeably with the number of observations in the experiment). For instance, in testing the efficacy of vaccines, experimenters stop after a pre-determined number of infections. In other cases, a power analysis may be used to determine sample size before the start of the experiment. But if the aim is to maximize welfare (or profits), neither of these procedures is optimal.¹¹1See, e.g., Manski and Tetenov (2016) for a critique on the common use of power analysis for determining the sample size in randomized control trials.

In this paper, we develop optimal experimentation designs that maximize social welfare (or profits) while also taking into account the cost of information. In particular, we study optimal sampling and stopping rules in sequential experiments where sampling is costly and the decision maker (DM) aims to determine the best of two treatments by: (1) adaptively allocating units to one of these treatments, and (2) stopping the experiment when the expected welfare, inclusive of sampling costs, is maximized. We term this the generalized Wald problem, and use minimax regret (Manski, 2021), a natural choice criterion under ambiguity aversion, to determine the optimal decision rule.²²2We do not consider the minimax risk criterion as it leads to a trivial decision: the DM should never experiment and always apply the status quo treatment.

We first derive the optimal decision rule in continuous time, under the diffusion regime (Wager and Xu, 2021; Fan and Glynn, 2021). Then, we show that analogues of this decision rule are also asymptotically optimal under parametric and non-parametric distributions of outcomes. The asymptotics, which appear to be novel, involve taking the marginal cost of experimentation to $0$ at a specific rate. Section 4 delves into the rationale behind these ‘small cost asymptotics’, and argues that they are practically quite relevant. It is important to clarify here that ‘small costs’ need not literally imply the monetary costs of experimentation are close to $0$. Rather, it denotes that these costs are small compared to the benefit of choosing the best treatment for full-scale implementation.

The optimal decision rule has a number of interesting, and perhaps, surprising properties. First, the optimal sampling rule is history independent and also independent of sampling costs. In fact, it is just the Neyman allocation, which is well known in the RCT literature as the (fixed) sampling strategy that minimizes estimation variance; our results state that one cannot better this even when allowing for adaptive strategies. Second, it is optimal to stop when the difference in average outcomes between the treatments, multiplied by the number of observations collected up to that point, exceeds a specific threshold. The threshold depends on sampling costs and the standard deviation of the treatment outcomes. Finally, at the conclusion of the experiment, the DM chooses the treatment with the highest average outcomes. The decision rule therefore has a simple form that makes it attractive for applications.

Our results also apply to the best arm identification problem with two arms.³³3The results for best arm identification were previously circulated in an unpublished note by the author, accessible from ArXiV at https://arxiv.org/abs/2204.05527. The current paper subsumes these results. Best arm identification shares the same aim of determining the best treatment but the number of observations is now exogenously specified, even as the sampling strategy is allowed to be adaptive. Despite this difference, we find Neyman allocation to be the minimax-regret optimal sampling rule in this context as well. However, by not not stopping adaptively, we lose on experimentation costs. Compared to best arm identification, we show that the use of an optimal stopping time allows us to attain the same regret, exclusive of sampling costs, with $40\%$ fewer observations on average (under the least favorable prior); this is independent of model parameters such as sampling costs and outcome variances.

For the most part, this paper focuses on constant sampling costs (i.e., constant per observation). This has been a standard assumption since the classic work of Wald (1947), see also Arrow et al. (1949) and Fudenberg et al. (2018), among others. In fact, many online marketplaces for running experiments, e.g., Amazon Mechanical Turk, charge a fixed cost per query/observation. Note also that the costs may be indirect: for online platforms like Google or Microsoft that routinely run thousands of A/B tests, these could correspond to how much experimentation hurts user experience. Still, one may wonder whether and how our results change under other cost functions and modeling choices, e.g., when data is collected in batches, or, when we measure regret in terms of nonlinear or quantile welfare. We asses this in Section 6. Almost all our results still go through under these variations. We also identify a broader class of cost functions, nesting the constant case, in which the form of the optimal decision stays the same.

Following Fudenberg et al. (2018) and Liang et al. (2022), we start by describing the problem under a stylized setting where time is continuous and information arrives gradually in the form of Gaussian increments. In statistics and econometrics, this framework is also known as diffusion asymptotics (Adusumilli, 2021; Wager and Xu, 2021; Fan and Glynn, 2021). The benefit of the continuous time analysis is that it enables us to provide a sharp characterization of the minimax optimal decision rule; this is otherwise obscured by the discrete nature of the observations in a standard analysis. Section 4 describes how these asymptotics naturally arise under a limit of experiments perspective when we employ $n^{-1/2}$ scaling for the treatment effect.

The setup is as follows. There are two treatments $0,1$ corresponding to unknown mean rewards $\bm{\mu}:=(\mu_{1},\mu_{0})$ and known variances $\sigma_{1}^{2},\sigma_{0}^{2}$. The aim of the decision maker (DM) is to determine which treatment to implement on the population. To guide her choice, the DM is allowed to conduct a sequential experiment, while paying a flow cost $c$ as long as the experiment is in progress. At each moment in time, the DM chooses which treatment to sample according to the sampling rule $\pi_{a}(t)\equiv\pi(A=a|\mathcal{F}_{t}),a\in\{0,1\}$, which specifies the probability of selecting treatment $a$ given some filtration $\mathcal{F}_{t}$. The DM then observes signals, $x_{1}(t),x_{0}(t)$ from each of the treatments, as well as the fraction of times, $q_{1}(t),q_{0}(t)$ each treatment was sampled so far:

Here, $W_{1}(t),W_{0}(t)$ are independent one-dimensional Weiner processes. The experiment ends in accordance with an $\mathcal{F}_{t}$-adapted stopping time, $\tau$. At the conclusion of the experiment, the DM chooses an $\mathcal{F}_{\tau}$ measurable implementation rule, $\delta\in\{0,1\}$, specifying which treatment to implement on the population. The DM’s decision thus consists of the triple $\bm{d}:=(\pi,\tau,\delta)$.

Denote $s(t)=(x_{1}(t),x_{0}(t),q_{1}(t),q_{0}(t))$ and take $\mathcal{F}_{t}\equiv\sigma\{s(u);u\leq t\}$ to be the filtration generated by the state variables $s(\cdot)$ until time $t$.⁴⁴4As in Liang et al. (2022), we restrict attention to sampling rules $\pi_{a}$ for which a weak solution to the functional SDEs (2.1), (2.2) exists. This is true if either $\pi_{a}:\left\{s(z)\right\}_{z\leq t}\to[0,1]$ is continuous, see Karatzas and Shreve (2012, Section 5.4), or, if it is any deterministic function of $t$. Let $\mathbb{E}_{\bm{d}|\bm{\mu}}[\cdot]$ denote the expectation under a decision rule $\bm{d}$, given some value of $\bm{\mu}$. We evaluate decision rules under the minimax regret criterion, where the maximum regret is defined as

We refer to $V(\bm{d},\bm{\mu})$ as the frequentist regret, i.e., the expected regret of $\bm{d}$ given $\bm{\mu}$. Recall that regret is the difference in utilities, $\mu_{0}+(\mu_{1}-\mu_{0})\delta-c\tau$, generated by the oracle decision rule $\{\tau=0,\delta=\mathbb{I}\{\mu_{1}>\mu_{0}\}\}$, and a given decision rule $\bm{d}$.

Following Wald (1945), we characterize minimax regret as the value of a zero-sum game played between nature and the DM. Nature’s action consists of choosing a prior, $p_{0}\in\mathcal{P}$, over $\bm{\mu}$, while the DM chooses the decision rule $\bm{d}$. The minimax regret can then be written as

The equilibrium action of nature is termed the least-favorable prior, and that of the DM, the minimax decision rule.

The following is the main result of this section: Denote $\gamma_{0}^{*}\approx 0.536357$, $\Delta_{0}^{*}\approx 2.19613$, $\eta:=\left(\frac{2c}{\sigma_{1}+\sigma_{0}}\right)^{1/3}$, $\gamma^{*}=\gamma_{0}^{*}/\eta$ and $\Delta^{*}=\eta\Delta_{0}^{*}$.

Theorem 1 makes no claim as to the uniqueness of the Nash equilibrium.⁵⁵5In fact, this would depend on the topology defined over $\mathcal{D}$ and $\mathcal{P}$. Even if multiple equilibria were to exist, however, the value of the game $V^{*}=\inf_{\bm{d}\in\mathcal{D}}\sup_{p_{0}\in\mathcal{P}}V(\bm{d},p_{0})$ would be unique, and $\bm{d}^{*}$ would still be minimax-regret optimal.

The optimal strategies under best-arm identification can be derived in the same manner as Theorem 1, but the proof is simpler as it does not involve a stopping rule. Let $\Phi(\cdot)$ denote the CDF of the standard normal distribution.

We now turn to the analysis of parametric models in discrete time. As before, the DM is tasked with selecting a treatment for implementation on the population. To this end, the DM experiments sequentially in periods $j=1,2,\dots$ after paying an ‘effective sampling cost’ $C$ per period. Let $1/n$ denote the time difference between successive time periods. To analyze asymptotic behavior in this setting, we introduce small cost asymptotics, wherein $C=c/n^{3/2}$ for some $c\in(0,\infty)$, and $n\to\infty$.⁶⁶6The rationale behind the $n^{3/2}$ normalization is the same as that in time series models with linear drift terms. The author is grateful to Tim Vogelsang for pointing this out.

Are small cost asymptotics realistic? We contend they are, as $C$ is not the actual cost of experimentation, but rather characterizes the tradeoff between these costs and the benefit accruing from full-scale implementation following the experiment. Indeed, one way to motivate our asymptotic regime is to imagine that there are $n^{3/2}$ population units in the implementation phase (so that the benefit of implementing treatment $a$ on the population is $n^{3/2}\mu_{a}$), $c$ is the cost of sampling an additional unit of observation, and time, $t$, is measured in units of $n$. This formalizes the intuition that, in practice, the cost of sampling is relatively small compared to the population size; this is particularly true for online platforms (Deng et al., 2013) and clinical trials. The scaling also suggests that if the population size is $n^{3/2}$, we should aim to experiment on a sample size of the order $n$ to achieve optimal welfare.

In each period, the DM assigns a treatment to a single unit of observation according to some sampling rule $\pi_{j}(\cdot)$. The treatment assignment is a random draw $A_{j}\sim\textrm{Bernoulli}(\pi_{j})$. This results in an outcome $Y^{(a)}\sim P_{\theta}^{(a)}$, with $P_{\theta}^{(a)}$ denoting the population distribution of outcomes under treatment $a$. In this section, we assume that this distribution is known up to some unknown $\theta^{(a)}\in\mathbb{R}^{d}$. It is without loss of generality to assume $P_{\theta^{(1)}}^{(1)},P_{\theta^{(0)}}^{(0)}$ are mutually independent (conditional on $\theta^{(1)},\theta^{(0)}$) as we only ever observe the outcomes from one treatment anyway. After observing the outcome, the DM can decide either to stop sampling, or call up the next unit. At the end of the experiment, the DM prescribes a treatment to apply on the population.

We use the ‘stack-of-rewards-representation’ for the outcomes from each arm (Lattimore and Szepesvári, 2020, Section 4.6). Specifically, $Y_{i}^{(a)}$ denotes the outcome for $i$-th data point corresponding to treatment $a$. Also, ${\bf y}_{nq}:=\{Y_{i}^{(a)}\}_{i=1}^{\left\lfloor nq\right\rfloor}$ denotes the sequence of outcomes after $\left\lfloor nq\right\rfloor$ observations from treatment $a$. We can imagine that prior to the experiment, nature draws an infinite stack of outcomes, ${\bf y}^{(a)}:=\{Y_{i}^{(a)}\}_{i=1}^{\infty}$, corresponding to each treatment $a$, and at each period $j$, if $A_{j}=a$, the DM observes the outcome at the top of the stack (this outcome is then removed from the stack corresponding to that treatment).

Recall that $t$ is the number of periods elapsed divided by $n$. Let $q_{a}(t):=n^{-1}\sum_{j=1}^{\left\lfloor nt\right\rfloor}\mathbb{I}(A_{j}=a)$, and take $\mathcal{F}_{t}$ to be the $\sigma$-algebra generated by

the set of all actions and rewards until period $nt$. The sequence of $\sigma$-algebras, $\{\mathcal{F}_{t}\}_{t\in\mathcal{T}_{n}}$, where $\mathcal{T}_{n}:=\{1/n,2/n,\dots\}$, constitutes a filtration. We require $\pi_{nt}(\cdot)$ to be $\mathcal{F}_{t-1/n}$ measurable, the stopping time, $\tau$, to be $\mathcal{F}_{t-1/n}$ measurable, and the implementation rule, $\delta$, to be $\mathcal{F}_{\tau}$ measurable. The set of all decision rules $\bm{d}\equiv(\{\pi_{nt}\}_{t\in\mathcal{T}_{n}},\tau,\delta)$ satisfying these requirements is denoted by $\mathcal{D}_{n}$. As unbounded stopping times pose technical challenges, we generally work with $\mathcal{D}_{n,T}\equiv\left\{\bm{d}\in\mathcal{D}_{n}:\tau\leq T\ \textrm{a.s}\right\}$, the set of all decision rules with stopping times bounded by some arbitrarily large, but finite, $T$.

The mean outcomes under a parameter $\theta$ are denoted by $\mu_{a}(\theta):=\mathbb{E}_{P_{\theta}^{(a)}}[Y_{i}^{(a)}]$. Following Hirano and Porter (2009), for each $a\in\{0,1\}$, we consider local perturbations of the form $\{\theta_{0}^{(a)}+h_{a}/\sqrt{n};h_{a}\in\mathbb{R}^{d}\}$, with $h_{a}$ unknown, around a reference parameter $\theta_{0}^{(a)}$. As in that paper, $\theta_{0}^{(a)}$ is chosen such that $\mu_{1}(\theta_{0}^{(1)})=\mu_{0}(\theta_{0}^{(0)})=0$ for each $a\in\{0,1\}$; the last equality, which sets the quantities to $0$, is not necessary and is simply a convenient re-centering. This choice of $\theta_{0}^{(a)}$ defines the hardest instance of the generalized Wald problem, with

for each $h\in\mathbb{R}^{d}$, where $\dot{\mu}_{a}:=\nabla_{\theta}\mu_{a}(\theta_{0}^{(a)})$. When $\mu_{1}(\theta_{0}^{(1)})\neq\mu_{0}(\theta_{0}^{(0)})$, determining the best treatment is trivial under large $n$, and many decision rules, including the one we propose here (in Section 4.3), would achieve zero asymptotic regret.

Let $P_{h}^{(a)}:=P_{\theta_{0}^{(a)}+h/\sqrt{n}}^{(a)}$ and take $\mathbb{E}_{h}^{(a)}[\cdot]$ to be its corresponding expectation. We assume $P_{\theta}^{(a)}$ is differentiable in quadratic mean around $\theta_{0}^{(a)}$ with score functions $\psi_{a}(Y_{i})$ and information matrices $I_{a}:=\mathbb{E}_{0}^{(a)}[\psi_{a}\psi_{a}^{\intercal}]$. To reduce some notational overhead, we set $\theta_{0}^{(1)}=\theta_{0}^{(0)}=\theta_{0}$, and also suppose that $\mu_{n,a}(h)=-\mu_{n,a}(-h)$ for all $h$. In fact, the latter is always true asymptotically. Both simplifications can be easily dispensed with (at the expense of some additional notation). We emphasize that our results do not fundamentally require $\theta_{0}^{(1)},\theta_{0}^{(0)}$ to be the same or even have the same dimension.