Regression Oracles and Exploration Strategies for
Short-Horizon Multi-Armed Bandits ^††thanks: Work partially funded by the NSF (Grant Number: IIS-1816470)

In both simulation environments, the task of the agent is to maximize the daily steps of the virtual players in an exergame or health study. The only thing that can be manipulated by the agent is the selection among three variants of the intervention that the player will see each day. The task is formulated as an MAB problem in which the agent will sequentially select from $k=3$ arms $A_{1,...,3}$ one choice for each player to maximize the reward $\rho_{t}$ (i.e., daily steps) observed each day $t$ over the course of the study’s horizon $h$.

To generate a realistic reward from the virtual players in our simulators, we used data from a Mechanical Turk experiment conducted in 2016 by Furberg et al. [18] that collected participants’ daily steps measured by a Fitbit. After removing outliers (values less than 100, $n=20$), we graph the data ($n=1665$) as a histogram in Figure 1. After confirming the data to be not normally distributed, we fit a Gamma distribution based on literature regarding human walking patterns and other intermittent behaviors [19, 20]. The probability density function for $\Gamma(k=2.8,\theta=3100)$ is included in Figure 1.

In our first simulator, which we call the stationary step simulator, we generate daily steps for each virtual player by drawing randomly from this Gamma distribution. However, we note that the nature of human walking behavior is not fully replicated by this approach, even if we are able to generate data that matches this distribution in aggregate. Therefore, we pursue a higher degree of fidelity in our second simulator, discussed below.

Many human activities are habitual, bound to routine, and exhibit cyclical patterns [21]. For example, if presuming a societal convention of planning routine activities within a 7-day week, it’s reasonable to expect that an individual’s walking patterns on the first day of that week would correlate with their walking patterns on the first day of other weeks. Thus, in our second simulator, which we call the pattern step simulator, we aim to exhibit this correlative and cyclical nature of daily step behavior while maintaining adherence to the overall expected distribution.

Specifically, to estimate the degree to which prior days’ steps might predict the current day’s steps, we constructed a regression model from the sequential step data using the seven days prior to a given step count as the features of the regression:

An Ordinary Least Squares (OLS) regression revealed the coefficients $\beta$ for these features, with which we employed backward elimination to construct a minimal model, sequentially removing the features that showed the least statistical significance until only those with $p<0.05$ remained. All features survived this process with $p<0.001$ except $S_{t-5}$, the number of steps five days prior to the current day. The resulting coefficients are presented in Table I.

To prime the steps for the first seven days, we simply pull from a Gamma distribution with $k=2.8$ and $\theta=3100$ as with the stationary step simulator. After that (i.e., when $t>7$), the step count is generated as follows:

To verify the accuracy of the simulator, results of an experiment of 500k daily steps is presented in Figure 2, where a histogram of the resulting probability density overlays a histogram of the original Mechanical Turk data. Additionally, we performed the same regression procedure with backward elimination on this data to verify that it resulted in a model with precisely the same significant features and similar coefficients. The results can be seen in the third column of Table I, verifying that the trends in our generated data match those observed in the real-world data.

We thus have a step simulator that both 1) reflects the observed real human step distribution and 2) maintains relative correlations observed in the real human daily step data. Let us now formulate the MAB problem used for evaluating our approach.

The step value $S_{t}$ generated for a simulated user on a given day $t$ is influenced by the arm selected by the MAB strategy. Specifically, each arm is defined by an adjustment range $[l,h]$. When the MAB strategy selects an arm, an adjustment value $r$ is uniformly sampled from the interval $[l,h]$, and the observed reward $\rho_{t}$ is calculated as follows:

The intuition is that these adjustment values would represent the degree to which a selected intervention could influence the number of steps the player would walk. Specifically, positive adjustment values indicate that the player would walk more steps as a consequence of the intervention, and negative values indicate the opposite. In this scenario, we devised a bandit that would select from three potential interventions, and the adjustment ranges for each of these three arms (A, B, and C) are shown on the third column of Table II.

A typical MAB strategy, such as $\epsilon$-greedy, will remember the rewards received for each pull of each arm and estimate the expected reward of a given arm by calculating the mean of those previously observed rewards. We call this calculation procedure the oracle used by the strategy to predict the expected reward of a given arm at the current time step. The policy for such a strategy only needs to compare the expected values for the rewards of all arms to determine which it should pull to receive the greatest reward. Other strategies like UCB1 additionally incorporate an estimation of the confidence interval to choose which arm to pull at each time step.

However, we note that these types of oracles make two assumptions inherent to the standard MAB problem formulation: (1) they assume reward distributions are stationary, and (2) they assume reward distributions of different arms are unrelated. Both of these assumptions are violated in our motivating domain (and in any domain involving repeatedly interacting with the same human subject). Standard approaches for non-stationary rewards (see Section II) such as weighted averages with decaying weight for past observations would not be useful in our short horizon setting; they would not be sample-efficient enough, nor would they leverage the type of patterns observed in human walking behavior.

To address this, we propose replacing the standard oracle used in MAB strategies with a regression oracle that makes a prediction based on not only the past observed rewards for a given arm, but on all past observed rewards. We implement this regression oracle by collating the previously observed rewards into a linear regression model, one that is capable of potentially capturing some of the temporal patterns we anticipate to be present in the generated step data. We expect this model to provide more accurate predictions of the expected values of arm rewards than simple means of past observations.

Specifically, at a given time step $t$ our regression model takes an input vector $\hat{x}_{t,a}=(\rho_{t-1},...,\rho_{t-m},O_{a})$, containing the observed rewards for the past $m$ iterations ($m=7$ in our experiments), and $O_{a}$, which indicates the arm $a$ for which we desire to generate a prediction (values of $O_{a}$ for each of the 3 arms in our experiments are shown in Table III). The model is trained to predict $\rho_{t,a}$, the expected reward we would obtain when pulling arm $a$ at time $t$.

Each time an arm is pulled and a reward observed, an OLS regression is performed on all the past observations to determine an updated set of regression coefficients $\beta$. The next time ($t$) an arm must be selected, a predicted reward value $\hat{\rho}_{t,a}$ is calculated for each arm $a$ as follows:

Notice that this is a strict generalization over calculating the mean; if there are no temporal correlations that can be exploited, the $\beta$ parameters corresponding to the reward for the past $m$ time steps will converge to either: (a) 0, and the coefficient for the $O_{a}$ parameter will converge to the mean reward for arm $a$ divided by $O_{a}$, or (b) a uniform distribution, and the coefficient of the $O_{a}$ term will be used to distinguish the reward of each arm. However, if temporal patterns or correlation among arms do exist, this oracle will be able to exploit them. More elaborate regression models could be devised, but given that we are focusing on the short horizon scenario and we cannot expect the model to be trained with more than a few dozen data points, simple linear regression is justified.

Also, we note that this formalization is very reminiscent of a contextual bandit, where we could consider the rewards observed over the past $m$ steps as the context vector. Moreover, we are also aware that because the observed rewards in the previous time steps depend on the arm being pulled, we are violating one of the basic assumptions of the MAB problem formulation–namely, that the bandit should be stateless, and previous arm selections should not affect future rewards. Our problem therefore resembles more a reinforcement learning setting than a bandit setting. However, we choose to approach it with MAB strategies due to the focus on short horizon; introducing the notion of state would imply more parameters to estimate in the model and would thereby decrease sample efficiency.

Most bandit strategies, such as $\epsilon$-greedy, rely on stochastic exploration of the arms. However, in short horizon settings, stochastic exploration might not be appropriate, and alternative strategies that perform heavier exploration at the beginning have been proposed. In order to determine the behavior of different exploration patterns, we compared a range of MAB strategies in both step simulators:

1. 1.

   UCB [7]: selects the arm with the highest potential reward based on confidence intervals around the average of past rewards.

2. 2.

   $\epsilon$-greedy: selects the “best” performing arm except when (with probability $\epsilon$) it explores by choosing a random arm.

3. 3.

   $\epsilon$-decreasing: similar to $\epsilon$-greedy, except with exploration probability $1/t^{\epsilon}$.

Additionally, for each of the strategies we explored the idea of forced exploration periods in which the MAB strategy is not permitted to engage its policy until each arm has been explored a specified number of times. During the forced exploration period, all arms are pulled an equal number of times, but the order in which they are pulled remains random. It is worth noting that we did not include the $\epsilon$-first strategy, as we view $\epsilon$-first to be simply $\epsilon$-greedy with forced exploration, a modification that could be similarly applied to any MAB strategy. Moreover, although this is only strictly necessary for UCB strategies, in our experiments all strategies (UCB, $\epsilon$-greedy, and $\epsilon$-decreasing) still undergo a forced exploration period of one pull per arm (in random order) to establish an estimate for the mean reward for each arm. Thus, in our experiments, when we label a strategy with forced exploration, we refer to a forced exploration period of four pulls per arm before engaging the strategy’s policy, whereas no forced exploration refers to forced exploration of just one initial pull per arm.

The UCB1 algorithm leverages the Chernoff-Hoeffding bounds [7] to provide an estimate of the upper confidence bound for a reward distribution derived from the observed rewards and a variance factor based on the number of observations. For reward distributions not confined to $[0,1]$, often an additional exploration factor $C$ is also included to empower the variance factor to influence the oracle’s valuation to a degree appropriate for the scale of rewards:

UCB1 is a popular strategy but may hold disadvantages in some applications. For instance, not every scenario can normalize rewards to $[0,1]$, such as when rewards have no upper limit. In these cases, the $C$ factor must be specially tuned to the scenario in order for the strategy to work effectively. However, some applications do not provide a means for pre-sampling and tuning this factor, in which case UCB1 may underperform.

In our scenario, rewards are unbounded, and thus to address this problem, we designed an alternative strategy UCBT that considers traditional statistical confidence bounds using sample variance. Because the variance factor calculation includes this consideration for the scale of rewards, the additional exploration factor $C$ is no longer required. UCBT does presume normality in the underlying reward distribution and therefore constructs a confidence interval using Student’s $T$ distribution. The critical value $t^{*}$ used in our experiments was drawn from a $T$-distribution lookup table with a 1-sided critical region and 99% confidence. We used a factor from a standard normal distribution ($t^{*}=2.326$) when degrees of freedom exceeded 200.

Specifically, on pull $t$, UCBT selects the arm $a$ for which the following yields the greatest value:

The next section presents experimental results to evaluate each of the three ideas considered in this paper to address the short horizon bandit problem in our target domain.

The reward distribution in the stationary step simulator is fixed, and the steps generated one day do not hold any correlations to steps from previous days. Average rewards across all steps from $t=1$ to $t=70$ for each MAB strategy (without forced exploration) can be found in the left columns of Table IV, and average reward over time is shown in the upper graph in Figure 3.

The first thing we observe is that even in the stationary step simulator where there are no temporal correlations among steps, the strategies with the regression oracle still outperformed the other four strategies. In strategies that do not use the regression oracle, the expected rewards of each arm are calculated independently from each other. However, in the case of the regression oracle variants, rewards observed for all previous iterations (including those for other arms) are used for estimating expected reward. Since in our scenario the rewards of different arms are correlated (i.e., they all correspond to the number of daily steps for the same player), this enables the regression oracle to estimate the expected reward of each arm earlier and more accurately.

As discussed above, the pattern step simulator attempts to structure a temporal correlation among daily steps that reflects the relationships observed in real human walking behavior. Average rewards across all steps without forced exploration in this simulator are shown in the left columns of Table V, and average reward over time is shown in the upper graph of Figure 4.

In the experiment without forced exploration, the regression oracle strategies struggle in the early steps as the regression models begin to populate with observations. However, these two strategies very quickly overtake the other four strategies once enough data has been collected. Interestingly, notice that this happens as early as step 10, indicating that the regression oracle requires very little data to start making a difference. We also observe that the more clustered rewards of the pattern step simulator allow the strategies to converge in performance to a greater degree than the higher variance rewards of the stationary step simulator.

Most important, these experiments strongly support our expectations regarding the regression oracle’s advantage when working with data containing temporal patterns.

These same experiments were repeated with a forced exploration period of four pulls per arm, the results of which are presented in the right-hand side of Tables IV and V for the stationary and pattern step simulators respectively. The lower graphs in Figures 3 and 4 show the average rewards observed by the MAB strategies over time.

Results show that the relative order in performance among the six MAB strategies does not change (with the exception of $\epsilon$-decreasing regression slightly outperforming $\epsilon$-greedy regression). The use of forced exploration endures a cost of lower rewards during the forced steps with the expectation of higher rewards afterward to reclaim that cost. Looking at the Last 7 Days columns in Tables IV and V, comparisons can be made for any of the MAB strategies regarding performance with and without forced exploration. As it can be seen, forced exploration strategies generally obtain a higher reward in the last time steps than strategies without forced exploration. However, because of the lower reward early on, the average reward across the whole experiment is generally lower or even. Therefore, we conclude that if the target horizon in a given application domain is very short (even shorter than our horizon of 70), forced exploration may not provide advantage in our domain, but it may still be advantageous in scenarios with (short) horizons longer than 70.

In the results presented in the previous sections, it may appear that our new strategy UCBT is outperformed by UCB1. However, those results correspond to UCB1 using $C=1600$, which was the parameter value that performed the best in our pre-experiments. However, if $C$ is not correctly tuned, UCB1 can significantly underperform. Figure 5 displays the results of UCBT versus UCB1 with different values for $C$. We show results both for a well tuned value ($C=1600$) as well as a poorly tuned value ($C=10000$). UCBT is scenario-agnostic and does not require any parameters or preparatory tuning. As the results demonstrate, UCB1 with an appropriate $C$ parameter excels in the early pulls and throughout the experiment, while the UCB1 strategy with the inappropriate $C$ parameter struggles. The UCBT strategy has a slower start due to the two-period forced exploration required to construct variance metrics for each arm; however, UCBT quickly catches up to match the performance of the optimal UCB1 in a relatively short horizon test. Thus, we believe UCBT is an promising strategy for real scenarios where it is not possible to tune the $C$ parameter beforehand. An example might be when we deploy our MAB strategies with real human players and find that some exhibit step distributions significantly different from those used to generate the simulators in this paper, where each would require different values for $C$.