BCRLSP: An Offline Reinforcement Learning Framework
for Sequential Targeted Promotion

In recent years, we have witnessed the rapid development of RL techniques and widespread applications of RL. In digital marketing, RL is expected to revitalize the industry and modernize various operations. For example, prior research has applied RL to solve digital marketing problems related to search (Wei et al., 2017; Xia et al., 2017; Xiao et al., 2019b; Hu et al., 2018), recommendation (Zhao et al., 2018b; Cai et al., 2018; Theocharous et al., 2015; Zhao et al., 2018c), online advertising (Yang and Lu, 2016; Zhao et al., 2018a; Jin et al., 2018; Cai et al., 2017; Wu et al., 2018; Schwartz et al., 2017), and pricing (Misra et al., 2019).

However, most prior works in this stream of literature only focus on a single objective. For example, budget is usually not involved in the applications of search or recommendation. Our paper differentiates from them by solving a multi-task problem: return maximization under constraints. Although some RL applications in online advertising also consider the budget constraint, for instance, Cai et al. (2017) and Wu et al. (2018) use RL to solve the budget-constrained real-time-bidding problem,they do not allow for offline training to make sure the costs of the proposed policy are controlled.

The most relevant work to our study is Xiao et al. (2019a), which uses a model-based MDP with a global constraint in sequential targeted promotion. Our work differentiates from Xiao et al. (2019a) in two major aspects. First, the state space is in high dimension, estimating the transition dynamics is very challenging, so we choose model-free algorithms over the model-based ones. Second, the CMDP setting controls the average budget over the whole trajectory of each user, while budget constraint in BCRLSP is set as the average cost per time step, which is more flexible and easier for the company to adjust.

In standard reinforcement learning (RL), a learning agent seeks to optimize the long-term return, which is based on a real-valued reward signal (Sutton et al. (1998)). However, in our setting, we want to ensure reasonable performance and at the same time respect budget constraints during deployment. RL algorithms with constraints are mostly formulated as a constrained Markov Decision Process (CMDP) introduced by Altman (1999). In CMDPs, the agent’s goal is to maximize the long-run rewards while satisfying some linear constraints on the long-run costs. Altman (1999) give an LP-based solution under the assumption that the CMDP process has known dynamics and finite states.

Based on that, some model-free approaches are also proposed to solve RL with constraints in high-dimensional settings. Achiam et al. (2017) use constrained policy optimization (CPO), focus on online safe exploration, and solve CMDP for continuous actions during the learning process. Le et al. (2019) describe a batch offline algorithm with PAC-style guarantees for CMDPs. Later, Miryoosefi et al. (2019) generalize this algorithm that can work with arbitrary convex constraints.

Although CMDP is the mainstream framework in RL with constraints, the long-run cost constraint in CMDP cannot satisfy the requirements of most business applications. CMDP models consider cost as part of the return and maximize the accumulated return. Mostly, companies have a budget constraint for each period of time, a week or a month. It’s difficult to define the value of the constraints for the long-run costs in this circumstance. In contrast, our BCRLSP method can meet the dynamic budget requirement set by the company by applying constrained optimization after solving the RL model. Our constraint is set per time period, which is more flexible and easier for the company to adjust. In this case, we are able to have exact control over the budget in near real-time.

In this research, we aim to learn a policy from offline data such that when deployed online, it maximizes the cumulative user retention rate while still keeping costs under constraints on the Taobao Special Offer Edition. We address this problem with a forward-looking sequential targeting method based on the model-free deep reinforcement learning algorithm. A learning agent chooses a value of cash bonus under a policy, and the value will be allocated to the user once claimed. We define the act of choosing the value of cash bonuses as an action. We formulate the sequential targeted promotion problem as a Markov Decision Process (MDP).

Interactions between the agent and the environment (users collecting the cash bonuses) are recorded in logged offline data $\mathcal{D}$, which is a set of trajectories $\tau_{i}$ as follows: Let $T$ be the maximum length of time step as far as we concern.¹¹1 We consider a cycle of user behavior as a trajectory and restart a trajectory when a cycle ends, so $T=4$ in our context. A trajectory is a sequence of user states, agent actions, and rewards, $\tau_{i}=\{(s_{t}^{i},a_{t}^{i},r_{t}^{i})\}_{t=1}^{T}$ for time step t and customer $i$.²²2For simplicity, in the rest of the paper, we denote a trajectory as $\tau$ without the subscript $i$. State $s_{t}^{i}$ is the user characteristics, action $a_{t}^{i}$ is the value of cash bonuses, and reward $r_{t}^{i}$ is a 0-1 value indicating whether or not the user logs in the next day. Note that $\mathcal{D}$ is collected from the behavior policy $\pi_{b}$ that interacted with the real users in the past and that action $a_{t}^{i}\sim\pi_{b}(\cdot\mid s_{t}^{i})$. For each action $a_{t}^{i}$, it has a corresponding cost $c_{t}^{i}$.

The first part of our model is reinforcement learning, which is solved by the Batch-Constrained Q-learning (BCQ) algorithm in the discrete action space (Fujimoto et al., 2019a). We choose to use BCQ in an effort to restrict the action space and force the agent towards behaving close to the behavior policy. We do this for economic concerns as this is a cost-sensitive problem. We train the default BCQ model from (Fujimoto et al., 2019a) using the offline data to estimate the expected return of all state-action pairs.

The second part of our model is a linear programming model followed by the RL module. It is natural to use the LP method for incentive allocation problems in a business setting to solve the large-scale constrained optimization problem efficiently. Under the given budget constraints, to maximize the return, i.e., the long-term retention of users, we use Q-values of actions learned from BCQ as the input of the LP model.

Q-values are a result of the unconstrained RL algorithm. Since Q-values represent the expected return $R_{t}$ starting from state s, following policy $\pi$, and taking action $a$, we are essentially maximizing the total return across all customers while satisfying the budget constraint in the LP model.

For a group of $N$ customers, each customer, $i$, is assigned to one and only one level of a cash bonus $j$ (i.e., action) among $M$ levels. The objective is to maximize the Q values of all customers under the constraint of the average cost. The problem formulation is as below:

where $x_{ij}$ denotes whether customer $i$ receives action $j$, $q_{ij}$ is the Q value corresponding to the state of customer $i$ and action $j$, $c_{j}$ is the cost incurred from action $j$, and $\bar{c}$ is the average cost externally set by the company.

It is worth noting that our budget constraint is the average cost per customer per day, so the total budget is not fixed but rather depends on the number of users who log in on a specific day. This setting fits our goal to increase the number of active users. Suppose we set an upper limit to the total budget, then as the number of users increases, the average cash bonus each user gets will drop, and it will be harder and harder to keep new customers active.

By introducing a Lagrange multiplier $\lambda$, we have the dual problem:

Note that the formulation above approximates the 0-1 integer program in Equation 4.3, which is acceptable in exchange for the efficiency of solving this problem. We further transform the dual problem above into a problem only with respect to $\lambda$:

After solving for $\lambda$, the $x_{ij}$ in the original problem is given as:

The $\lambda$ parameter reflects how sensitive users are to cash bonuses. If $\lambda$ is large, it means that users’ retention rates do not change much as cash bonuses increase. Thus, the optimal strategy is to assign small cash bonuses to users, which will not affect the retention rates a lot but significantly decrease the costs. On the contrary, a small $\lambda$ indicates that uses are very sensitive to cash bonuses, and we need to send large cash bonuses to make sure they continue to use the app.

The key to solving the LP problem is to calculate $\lambda$ such that (1) it accurately reflects the distribution of user sensitivity and (2) it is updated in a short time. To achieve the first goal, it is necessary that the distribution of user sensitivity is stable over time. Since we employ the $\lambda$ values solved using historical data, we use the logged data in the past 24 hours up to the current time point to calculate $\lambda$, rather than the logged data in a shorter time period. The distribution of user sensitivity on the previous day will be similar to that on the current day, enabling us to acquire more accurate $\lambda$ values and have better control of costs.

As for the second goal, we update the logged data and recalculate the corresponding $\lambda$ every 10 minutes. Theoretically, we can update $\lambda$ more frequently or even in real-time, and using a shorter time interval also gives more accurate results. But increasing the frequency increases computational time, and we find that a time interval of 10 minutes achieves the best balance between accuracy and efficiency.

We compare the proposed approach with several alternatives to demonstrate the advantages and disadvantages of different approaches. The baseline models we use are briefly introduced as follows.

We investigate the performance of BCRLSP on a large-scale real-world dataset. We collaborate with Alibaba Inc. and collect data from the Taobao Special Offer Edition App. This real-world data is used to train and evaluate our framework in an offline manner. The dataset comprises 1.4 million records for a continuous one-month period in December 2020.We split the interactions into the training, validation, and test sets dis-jointly according to different time ranges. The first 2 weeks are used for training, and the remaining 2 weeks are used for offline testing.

For both BCQ and Rainbow algorithms, we use a 2-layer fully connected neural network with 128-dimension hidden states in the user behavioral model. We use a threshold of $\xi=0.3$ to eliminate actions when selecting the optimal action using BCQ.³³3We tried different thresholds of $\xi=0.3,0.4,0.5,0.6$ and the results are similar. We use the exploration rate $\epsilon=0.01$ to assign a random action in the exploration subsample.⁴⁴4We tried different exploration rates of $\epsilon$ between 0.01 and 0.1 and $\epsilon=0.01$ shows the best results. The discount factor is set to $\gamma=1$ as the company values the user retention in the whole trajectory and does not discount the rewards across different time periods.⁵⁵5Each trajectory has a finite length with the maximum length = 4, so $\gamma$ can be set to 1. Training on 6 CPU cores, 30 GB of memory, and 2 GPU Tesla V100 took roughly 5 hours for the RL models. The hyper-parameters in the baseline models are set as follows: We use a grid search to find the best hyper-parameters in each supervised learning model in the holdout validation dataset. For GBDT: learning rate = 0.1, max depth =3, number of estimator = 500; For Xgboost: learning rate = 0.05, max depth = 5, number of estimator = 500.

We launch an offline experiment to calculate the evaluation results based on the approach proposed by Fonteneau et al. (2013). Specifically, we use all the states in the test dataset and consider them as users interacting with the agent. Then, given a state in the dataset $s_{t}$, the agent generates an action $\hat{a}_{t}$ according to its policy. Then we find the records in the dataset where the actual action given by the behavior policy matches the estimated action by the agent: $a_{t}=\hat{a}_{t}$. Denote these matching records as $\mathcal{M}=\{\tau_{1},\dots,\tau_{m}\}\subset\mathcal{D}$, where $\tau_{i}=\{(s_{t}^{i},a_{t}^{i},r_{t}^{i})\}_{t=1}^{T}$.

The results of offline experiments are shown in Table 1. Our proposed BCRLSP model and all baseline models are able to maintain the same average cost because the LP model is set under a fixed budget of 0.87 Yuan per customer per day. Under the same budget setting, BCRLSP model outperforms all baseline models and achieves the highest retention rate. Compared with the second-best model (GBDT), the retention rate of BCRLSP model increases from $36.15\%$ to $37.22\%$, an improvement of $2.96\%$. We train the models using the same data and hyper-parameters for 10 times, and the retention rates change within the range of $0.1\%$.

f We also evaluate the BCRLSP framework for a week in the online environment, i.e., real-world cash bonuses on the check-in page in the Taobao Special Offer Edition App. The comparison baseline models are the expert policy, Rainbow, GBDT, and logistics regression. To assess the effectiveness of the proposed framework, the platform assigns $92\%$ of the online traffic to the expert policy and $2\%$ of the online traffic to each of the BCRLSP, Rainbow, GBDT, and logistic regression models, which is big enough considering the overall amount of user interactions. Table 2 lists the performance of the two main online metrics.

Compared with the expert policy, $0.53\%$ growth on retention rate exhibits that our proposed framework increases the retention rate of the platform when put into production. Note that during the online experiment, the firm decided to increase the budget to 1.60 Yuan, hence we can see an increase in both the average cost and the retention rate in the online experiment results compared with offline ones.