Opportunistic Advertisement Scheduling in Live Social Media: A Multiple Stopping Time POMDP Approach

In this section, we develop a model of the live channel. The three main components of the live channel are i) Viewer engagement: How to model the viewer engagement of a live channel? ii) Dynamics of the channel viewers: How does the number of viewers vary with respect to the engagement? iii) Reward of the channel owner: What is the (monetary) reward obtained by the channel owner through advertising? Below, we develop models to address each of these questions.

1. 1)

   Dynamics of viewer engagement: Similar to [24], viewer engagement, in the context of live channels, can be defined as the following process: (i) The viewer decides to watch the live channel. (ii) The viewer is “engaged” with the content of the live channel. (iii) The viewer will watch the live content without switching to other channels. (iv) The viewer is more attentive when watching the live content. The potential benefits of (iii) and (iv) above is that it increases the odds of the viewers being exposed and persuaded by advertisements.

   Viewer engagement, as defined above, is an abstract concept which captures viewer attitude, behaviour and attentiveness. Archak et. al. [8, 9] developed a Markov chain model for online behaviour of users and the effects of advertising. The main finding is that the user behaviour in online social media can be approximated by a first-order Markov chain. Following Archak et. al. [8, 9], we model the viewer engagement at time $t$, denoted by $X_{t}$, as an $S$-state Markov chain with state-space $\mathcal{S}\equiv\left\{1,2,\cdots,{S}\right\}$. The dynamics of the viewer engagement of the channel, modelled as a Markov chain, can be characterized by the transition matrix $P$ and initial probability vector $\pi_{0}$ as follows:

   $$P(i,j)=\mathbb{P}(X_{t+1}=j|X_{t}=i),\quad\pi_{0}(i)=\mathbb{P}(X_{0}=i)$$

   (1)

   The Markov chain model for the viewer engagement of the channel is validated using simulations in Sec. V.

2. 2)

   Dynamics of channel viewers: The number of viewers at time $t$ depend on the viewer engagement of the live channel. As viewers are more engaged with the content, they are less likely to switch the channel. Hence, a higher viewer engagement state has higher number of viewers compared to a lower engagement state. Therefore, we model the dynamics of channel viewers as follows: The number of viewers at time $t$, denoted by $Y_{t}$, belongs to the countably infinite set $\mathcal{Y}$ of non-negative integers. Denote, the conditional probability of $j$ viewers ($Y_{t}=j$) in viewer engagement state $i$ ($X_{t}=i$) by $B(i,j)$. Note that the conditional probability $B(i,j)$ is assumed to be time homogeneous⁸⁸8The conditional probability $B(i,j)$ does not depend on the time index, $t$. . The number of viewers $Y_{t}$ is modeled as a Poisson random variable with state dependent mean $g_{i},i\in\mathcal{S}$, based on evidence in [25] and [11], as follows:

   $$B(i,j)=\mathbb{P}\left(Y_{t}=j|X_{t}={i}\right)=\frac{g_{i}^{j}\exp{(-g_{i})}}{j!},~\forall i\in\mathcal{S},j\in\mathcal{Y}.$$

   (2)

   The states with higher state dependent mean correspond to states with higher viewer engagement.

   The channel owner does not observe the true viewer engagement of the channel, $X_{t}$. However, at each time instant $t$, the channel owner receives a noisy observation of the viewer engagement of the channel by the number of viewers, $Y_{t}$. Hence, the channel owner needs to estimate the viewer engagement using the history of noisy observations and schedule ads.

3. 3)

   Reward of channel owner: The channel owner agrees to show $L$ ads during the live session, which are decided prior to the beginning of the session. For example, the ads during the Super Bowl 50 in YouTube Live had to be pre-booked in advance.⁹⁹9http://www.campaignlive.co.uk/article/youtube-launches-real-time-ads-major-live-events-starting-super-bowl-50/1380260 Hence, at each time instant $t$, the channel owner chooses an action $u_{t}$ as follows: The channel owner can continue with the live session (denoted by Continue) or can pause the live session to insert an ad (denoted by Stop). Hence, $u_{t}\in\mathcal{A}=\{\text{Stop(1)},\text{Continue(2)}\}$¹⁰¹⁰10The Stop and Continue actions will be denoted by $1$ and $2$, respectively in the remainder of the paper. denote the actions available to the channel owner at time $t$. This problem of scheduling the $L$ ads opportunistically, so as to obtain maximum revenue, corresponds to a multiple stopping problem with $L$ stops.

   Choosing to stop at time $t$ (and schedule an ad), with $l$ stops remaining ($l$ ads remaining), the channel owner will accrue a reward $r_{l}(X_{t},a=1)$, where $X_{t}$ is the state of the Markov chain at time $t$. The reward obtained by the channel owner depends on two factors: (i) the number of viewers (ii) the completion rate and click rate of the ad . To capture these, we model the reward as follows:

   $$r_{l}(X_{t}=i,a=1)=\alpha_{i}g_{i}.$$

   (3)

   In (3), $g_{i}$ captures the average number of viewers in any viewer engagement state. The term $\alpha_{i}\in\left[0,1\right]$ captures the completion and click rate of the ads at any viewer engagement state. Similarly, if the channel owner chooses to continue, he will accrue $r_{l}(X_{t},a=2)$. When an ad is not shown, the reward obtained by the channel owner is usually zero, hence, $r_{l}(X_{t},a=2)=0$.

   Hence, to maximize revenue, the channel owner needs to opportunistically schedule ads at time slots when the viewer engagement is high, corresponding to a higher number of viewers and higher click rate.

The value iteration algorithm is a successive approximation approach for solving Bellman’s equation (8).

The procedure is as follows: For iterations $k=0,1,\dots$, the value function $V_{k}(\pi,l)$ and the policy $\mu_{k}(\pi,l)$ is obtained as follows

where

and

with $V_{0}(\pi,l)$ initialized arbitrarily. It can be easily shown that the above procedure converges [14].

In order to prove the structural results of the stationary ad scheduling policy, defined in Sec. II, we define the stopping and the continuance regions of the policy as below. Let $W_{k}(\pi,l)$ be defined as

The stopping and continuance region (at each iteration $k$) is defined as follows:

Since the value iteration converges, the optimal stationary policy $\mu^{*}(\pi,l)$ is defined as

Correspondingly, the stationary stopping and continuance sets are defined by

The value function, $V_{k}(\pi,l)$ in (10), can be rewritten, using (15), as follows:

where $\mathcal{I}_{C_{k}^{l}}$ and $\mathcal{I}_{S_{k}^{l}}$ are indicator functions on the continuance and stopping regions respectively, for each iteration $k$.

Assume $S_{k}^{l-1}\subset S_{k}^{l}$ (see Proposition 3) and substituting (18) in the definition of $W_{k}(\pi,l)$ in (14),

The value iteration algorithm (10) to (13) does not translate into a practical solution methodology since the belief state belongs to an uncountable set. Hence, there is a strong motivation to characterize the structure of the optimal policy. In the following section, we use the definition of $W_{k}(\pi,l)$ in (19) to prove structural results on the stopping region defined in (15).

In this section, we provide structural results on the optimal policy of the multiple stopping problem corresponding to maximizing the ad revenue.

Consider a threshold hyperplane, on the simplex $\Pi$, of the form (21) where $\theta_{l}\in{\rm I\hskip-1.9919ptR}^{S-1}$ denotes the coefficient vector. The linear threshold scheduling policy, denoted by $\mu_{\theta}(l,\pi)$ is defined as

Here, $\theta=\left(\theta_{1},\theta_{2},\dots,\theta_{L}\right)\in{\rm I\hskip-1.9919ptR}^{L\times(S-1)}$ is the concatenation of the $\theta_{l}$ vectors.

To capture the essence of Theorem 1A, we require that the policy be decreasing on lines, i.e. for $\pi_{1}\geq_{\mathcal{L}_{1}}\pi_{2},\;\mu_{\theta}(\pi_{1},l)\leq\mu_{\theta}(\pi_{2},l)$. Theorem 2 gives necessary and sufficient conditions on the coefficient vector $\theta_{l}$ such that the above condition holds.

The proof of Theorem 2 is similar to the proof of Theorem 12.4.1 in [27] and hence is omitted in this paper.

The stopping sets are connected since we parametrize the threshold curve using a linear hyperplane. Finally, the linear threshold approximation curve needs to satisfy the sub-setting property in Theorem 15C. Theorem 22 provides sufficient conditions such that the parametrized linear threshold curve satisfy the sub-setting property and the proof is provided in the Appendix B-E.

Therefore, under the conditions of Theorem 2 and Theorem 22 the linear threshold policy in (21) satisfy all the conditions in Theorem 1 and hence qualify as the best linear threshold policy.

The parameter $\theta$ can be re-parametrized as follows:

It can be easily checked the parametrization in (23) satisfies the conditions in Theorem 2 and Theorem 22.

Theorem 2 and Theorem 22 characterize the parameters of the linear threshold policy. In Sec. IV-B we provide an algorithm to compute the best linear threshold policy satisfying Theorem 2 and Theorem 22.

In this section, we provide an algorithm to compute the best linear thresholds satisfying the conditions in Theorem 2 and Theorem 22. Recall, the optimal policy minimizes the average discounted reward in eqs. 6 and 7. The optimal linear thresholds can be obtained by a policy gradient algorithm [14]. Algorithm 1 is a policy gradient algorithm to compute the best linear threshold policy. In this algorithm, we approximate $J_{\mu}$ in eqs. 6 and 7 by the finite time approximation

Here, we have made explicit the dependence of the parameter vector, $\theta$, on the discounted reward and with an abuse of notation, have suppressed the dependence on the policy $\mu$. It can be shown that $J_{N}(\theta)$ is an asymptotically biased estimate of $J_{\mu}$ and can be obtained by simulation for large $N$.

The policy gradient algorithm in Algorithm 1 requires the gradient $\nabla_{\phi}J_{N}(\cdot)$ at each iteration. Computation of the gradient is quite difficult due to the non-linear dependence of the parameter $\phi$ on the cost function. Hence, in this paper, we estimate the gradient through a stochastic approximation algorithm.

There are several stochastic approximation algorithms available in the literature: infinitesimal perturbation analysis[30], weak derivatives [31] and the SPSA algorithm [32]. In this paper, we use the SPSA algorithm and because of the constraints in Theorem 22 we use a variant of SPSA that can handle linear inequality constraints [33].

Following [32] and [33], the gradient estimate is obtained by picking a random direction $\omega_{n}$, at each iteration $n$. The estimate of the gradient is then given by

The equations for the parameter update are as follows [33]:

The parameters $a_{n}$ and $c_{n}$ and $r_{n}$ are chosen as in [33] as follows:

The stochastic approximation in Algorithm 1 converges to a local minimum. Hence, it is necessary to try several initial conditions and pick the best threshold.

In this section, we do a simulation study based on synthetic data for the optimal multiple stopping problem. The objective is to illustrate the analysis in Theorem 1. In addition, we obtain the policy by solving the dynamic programming equations (8) and show through simulations that it outperforms conventional method of scheduling ads periodically.

In this “toy” example, the viewer engagement of the live channel can be categorized into $3$ states: “Popular”, “Interesting” and “Boring”, denoted by $1$, $2$ and $3$, respectively. The transition between the various states follow a Markov chain with transition matrix given in (29). The viewer engagement state is observed through a state dependent Poisson process with mean given in (30). The reward structure for scheduling the ads is as in (31).

Fig. 2 shows a sample trajectory of the observation and the state sequence. For $L=5$, i.e. for a total of $5$ ads to be scheduled, we obtained the policy by solving the dynamic programming equations in (8). The resulting policy, in terms of stopping and continuance set as defined in (14), is shown in Fig. 3. We also show the corresponding points where the policy was chosen in Fig. 2. Only the stopping set for $l=5$ and $l=1$, i.e. $S^{5}$ and $S^{1}$ respectively are shown in Figure 3. As can be seen from Figure 3, $S^{1}\subset S^{5}$, verifying Theorem 1.C. The stopping regions $S^{1}$ and $S^{5}$ are connected and the optimal policy is threshold on any line $\mathcal{L}(e_{1},\pi)$, verifying Theorem 1.B and Theorem 1.A respectively.

We now compare the policy obtained by solving the dynamic programming equation in (8) with the conventional technique of scheduling ads periodically within the live session. We also include a random scheduling policy, where the ads are scheduled randomly within the live session, for benchmarking purposes. Fig. 4 shows the comparison between the various schemes. The results in Fig. 4 was obtained by $10^{4}$ independent Monte Carlo (M.C.) simulations. It can be seen from Fig. 4 that the policy obtained from (8) significantly outperforms (close to $4$ times) periodic scheduling. This is not surprising since the policy obtained by solving the dynamic programming in (8) opportunistically schedules ads when the viewer engagement of the channel is high.

In this section, we illustrate Theorem 1 for change detection problems. Change detection problems are special case of multiple stopping problems, when $L=1$. We apply Theorem 1 for detecting changes in ground truth using an online search dataset. Online search is linked to advertising in television and online social media [34]. In addition, detecting changes in ground truth using online search data have been used for detection of outbreak of illness, political election, or major sporting events [35]. Hence, detection of changes in ground truth is important for optimizing advertising strategy. The dataset that we use is the Tech Buzz dataset from Yahoo!. We first describe the Tech Buzz dataset in Sec. V-B1 and then show through simulations that the policy obtained by solving the dynamic programming equation in (8) can be used for detecting changes in ground truth using data from online search.

In this section, we illustrate the policy from Algorithm 1 on real data from YouTube and Twitch. In comparison to Sec. V-A and Sec. V-B, real data from YouTube and Twitch has a wide range of viewer engagement states and hence requires more states in the Markov chain model. As the number of states increases, solving the dynamic programming equation in (8) becomes impractical. Hence, we resort to best linear threshold policy through Algorithm 1. We first describe the dataset in Sec. V-C1 and then show that the policy obtained from Algorithm 1 outperforms conventional periodic scheduling.