Trajectory-wise Iterative Reinforcement Learning Framework for Auto-bidding

We introduce Parameter Space Noise (PSN) (Plappert et al., 2018; Fortunato et al., 2018) for exploration in the iterative offline RL framework, and explain its effectiveness from a trajectory view. PSN refers to injecting noise in an RL policy’s parameter space in order to induce exploratory behaviours. It has brought performance gain on a wide range of control tasks when applied to online deep RL algorithms (Badia et al., 2020; Hessel et al., 2018). For a parameterized policy (e.g. a neural network) $\pi(s;\theta)$, where $\theta$ is the parameter vector, applying additive Gaussian noise to $\theta$ gives $\hat{\theta}=\theta+\epsilon$, where $\epsilon\sim\mathcal{N}(0,\sigma^{2}I)$. Then the exploration policy based on PSN is $\pi_{e}^{PSN}(s_{t})=\pi(s_{t};\hat{\theta})$. Importantly, the perturbed parameter vector $\hat{\theta}$ is only sampled at the beginning of each episode and remains fixed afterwards. This is substantially different from ASN where independent noise is added at every time step.

We now present our key observation on datasets collected by PSN by experiments in a simulated bidding environment (details are provided in Appendix D). We first construct two exploration policies with ASN and PSN based on one policy $\pi$, and denote them by $\pi_{e}^{ASN}$ and $\pi_{e}^{PSN}$ respectively. Subsequently, two datasets $D^{ASN}$ and $D^{PSN}$ of equal size are collected with $\pi_{e}^{ASN}$ and $\pi_{e}^{PSN}$. ³³3To ensure a fair comparison, we control the noise strength of both exploration policies to guarantee that the average return of $D^{ASN}$ and $D^{PSN}$ are equal. As shown in Figure 3, the return distribution of trajectories in $D^{PSN}$ is more dispersed than that of $D^{ASN}$. Specifically, $D^{PSN}$ contains high-return trajectories (e.g. trajectories with return higher than 750) which are almost absent in $D^{ASN}$.

The high return variance observed in PSN can be attributed to two main factors: decoupling between the exploration policy and the base policy (i.e. the original noiseless policy $\pi$), and the consistency of exploration behaviours. The following example provides further explanation in the context of auto-bidding, by highlighting the incapability of ASN on generating high-return trajectories. Assume that the current policy $\pi$ is suboptimal in that it produces relatively low bids across most states. In this case, positive perturbations $\epsilon_{t}$ are desirable when applying ASN. However, when $\pi_{e}^{ASN}(s_{t})$ is lifted, $\pi(s_{t+1})$ would become low since the base policy $\pi$ aims to maintain the smoothness of budget consumption, thereby offsetting the effect of high bid in step $t$. The counterbalancing actions of the base policy substantially prevent ASN from effectively exploring unknown areas. Another issue with ASN is that, since the perturbations $\epsilon_{t}$ of different time steps are drawn from independent probability distributions, consistently realizing positive $\epsilon_{t}$ throughout an entire episode is barely possible.

PSN enables us to sample a variety of policy parameters, which could be distributed to a great number of advertising campaigns in a DSP to explore different bidding behaviours. These campaigns run in parallel and collect a large dataset covering a wide range of behaviours, thus achieving large-scale trajectory-wise exploration.

While PSN-based exploration policies could effectively generate valuable high-return trajectories, their collected datasets still consist primarily of inferior trajectories, as depicted in Figure 3. Therefore, the trained policy’s performance is still limited by the inherent conservatism of offline RL. To alleviate this problem and fully exploit the dataset, we propose Robust Trajectory Weighting for trajectory-wise exploitation.

We consider a dataset containing multiple trajectories $D=\{\tau_{i}\}_{i=1}^{N}$, where $\tau_{i}=\{(s_{i,t},a_{i,t},s_{i,t+1},r_{i,t})\}_{t=0}^{T}$. Inspired by (Yue et al., 2022; Hong et al., 2023; Yue et al., 2023), instead of uniform sampling, we assign large sample probabilities to well-performing trajectories during training. A straightforward way to realize this is to assign weight $w_{i}$ for trajectory $\tau_{i}$ based on its return $R_{i}\coloneqq R(\tau_{i})$, as high return typically indicates good behaviours. Nonetheless, two main issues arise when applying return-based trajectory weighting in the auto-bidding task. Firstly, due to the instability of auction environments and user feedbacks, the reward function $r(s_{t},a_{t})$ is highly stochastic, thus a high return might not necessarily be achieved by a good policy, but rather a ”lucky” trial that happens to obtain high rewards in most time steps. Secondly, trajectories within the dataset may come from different advertising campaigns, and the intrinsic characteristics of a campaign, such as its budget level and item category, also affect the trajectory return. Thus, directly comparing trajectories from different campaigns by their returns is unfair.

We address the first issue by learning a reward model $\bar{r}$ on the dataset for predicting the expected reward of a state-action pair:

The reward model could be implemented by a neural network with the above loss function. Then the original stochastic rewards $r_{i,t}$ are replaced with their expectations $\bar{r}_{i,t}=\bar{r}(s_{i,t},a_{i,t})$ to calculate $\bar{R}_{i}=\sum_{t=0}^{T}\gamma^{t}\bar{r}_{i,t}$, producing more robust quality indicators.

To deal with the second issue mentioned above, we regularize the trajectory returns by subtracting the value of the initial state of the trajectory, estimated as $\hat{V}=\operatorname*{arg\,min}_{V}\sum_{i=1}^{N}(V(s_{i,0})-\bar{R}_{i})^{2}$. The initial state typically contains information (e.g. the total budget) of the advertising campaign, therefore $\hat{V}(s_{i,0})$ provides estimation of the expected return of the campaign behind trajectory $i$. Our final indicator of trajectory quality is $\hat{A}_{i}$:

The sample probability $w_{i,t}$ of transition tuple $(s_{i,t},a_{i,t},s_{i,t+1},r_{i,t})$ is computed according to $\hat{A}_{i}$ as follows:

where $\alpha\in\mathbb{R}^{+}$ is a temperature parameter, and the weights should be normalized to ensure $\sum_{i=1}^{N}\sum_{t=0}^{T}w_{i,t}=1$.

After computing weights for dataset $D$, we could run any model-free offline RL algorithm (e.g. CQL, IQL) using the reweighted data sampling strategy. We provide a theoretical justification of Robust Trajectory Weighting in Appendix A, showing how it alleviates the problem brought by conservative algorithms.

Although TEE boosts the effectiveness of iterative offline RL, the safety of data-collecting policies, which is of great importance when training in real-world advertising systems, have not been considered. In this section, we propose a novel algorithm named SEAS to guarantee the safety of online exploration.

On the problem of safe exploration in auto-bidding, Mou et. al. (Mou et al., 2022) designed a method based on safety zone, to restrict the exploratory actions around a safe policy. Specifically, $\pi_{e}(s_{t})\leftarrow clip(\pi_{e}(s_{t}),\pi_{s}(s_{t})-\xi,\pi_{s}(s_{t})+\xi)$, where $\pi_{s}$ is a known safe policy. Though this method is provably safe under some assumptions on the MDP, its safety heavily relies on the radius $\xi$ which is intractable to determine. Besides, this approach lacks awareness of the quality of original exploration actions, and poses constraint on both bad and good actions, thus hurting the quality of collected datasets. SEAS mitigates these problems through an adaptive design. The aim of SEAS is to prevent the low-performing trajectories caused by the original exploration policy (e.g. $\pi_{e}^{PSN}$) to emerge, while preserving the high-quality ones to the fullest extent.

The procedure of SEAS interacting with the environment for one episode is shown in Algorithm 1. In each step, SEAS selects between an exploratory action $a_{e}$ and a safe action $a_{s}$ according to the condition in line 6. Note that multiple safe policies are provided to the algorithm, and in each step the safe policy with maximum Q value is chosen for constructing the condition. Utilizing multiple safe policies instead of one loosens the restriction in line 6, thus better preserving exploratory actions of $\pi_{e}$. The algorithm is ”adaptive” in the sense that each action $a_{t}$ depends on the rewards accumulated up to time $t$.

The following theorem shows that SEAS theoretically guarantees safety for any exploration policy $\pi_{e}$. Proof of Theorem 1 is provided in Appendix B.

The advantages of SEAS are summarized as follows: (i) SEAS needs only one hyperparameter $\epsilon$, which is in the definition of safety and is straightforward to set. (ii) The safety of SEAS is provable without additional assumptions on the underlying MDP. (iii) Experiments demonstrate that when functioning together with TEE, SEAS exhibits minimal performance sacrifice compared to baseline methods.

TEE and SEAS together form an iterative RL framework for auto-bidding. Appendix C provides a detailed description of the overall procedure.

A simulated advertising system is constructed for all the offline experiments in this work. Details of the setup and hyperparameters are provided in Appendix D.

We test the effectiveness of TEE and SEAS by combining them with three different offline RL algorithms. We also implement three baselines: iterative versions of IQL with and without exploration, as well as SORL (Mou et al., 2022). An expert policy is trained with TD3 (Fujimoto et al., 2018), an online RL algorithm, to serve as a performance upper bound. Methods for comparison are listed below.

• •

  TEE+SEAS+IQL (Kostrikov et al., 2021) / CQL (Kumar et al., 2020) / TD3BC (Fujimoto and Gu, 2021). Iterative offline RL with the proposed TEE and SEAS, using IQL / CQL / TD3BC as the offline RL algorithm.

• •

  IterIQL+ASN. Iterative offline RL using IQL as the offline RL algorithm. IQL is selected as a representative because it is the state-of-the-art model-free offline RL algorithm. Exploration policy $\pi_{e}^{k}$ is constructed by adding ASN on $\pi^{k}$.

• •

  IterIQL. Iterative offline RL using IQL as the offline RL algorithm. No exploration noise is added, therefore $\pi_{e}^{k}=\pi^{k}$.

• •

  SORL (Mou et al., 2022). SORL follows the iterative offline RL framework. The authors proposed V-CQL for offline policy training and designed an SER policy for safe and efficient exploration.

Evaluation Metrics. We evaluate the performance of a policy in terms of expected return. Besides, we check the safety of exploration policy by comparing average return in the collected dataset with the safety threshold $(1-\epsilon)J_{s}$.

Figure 4 presents the overall performance through iterations. Our proposed framework, combined with any of the three offline RL algorithms, substantially outperforms the baseline methods and achieves near-expert performance in approximately 5 iterations. Both Iterative IQL+ASN and SORL get stuck in suboptimal policies after limited performance improvements, suffering from the performance bottleneck we discussed in section 4. Additionally, Iterative IQL without exploration achieves barely any performance improvement, which demonstrates the necessity of exploration in iterative offline RL.

We present the performance of exploration policies during the training process in Figure 5. Combined with any offline RL algorithm, our framework consistently ensures the performance of exploration policies to be above the safety threshold $(1-\epsilon)J_{s}$, which validates the safety guarantee ability of SEAS. We also show the results when SEAS is omitted, in which the safety constraint is violated in early stages.

We conduct real-world experiments on Alibaba display advertising platform. In each iteration, we utilize 20000 advertising campaigns to collect data for an epsiode, and employ IQL for offline training. Each episode contains 48 time steps, which means that the bidding parameter is adjusted every 30 minutes. After each policy update, we conduct a 10-day online A/B test using 1500 campaigns.

Evaluation Metrics. The return of the trained policy acts as our main metric of performance, and is referred to as BuyCnt in the online experiments. Additionally, we introduce several other metrics that are commonly used in the auto-bidding field.

• •

  BuyCut. The total value of ad impressions won by the advertiser. Our objective $J$ in the RL formulation.

• •

  ROI. The ratio between the total revenue and the consumed budget of the advertiser.

• •

  CPA. Cost per aquisition, defined as the average cost for each successfully converted impression. A smaller CPA indicates a better performance of an auto-bidding policy.

• •

  ConBdg. The total consumed budget of the advertiser.

• •

  GMV. Gross Merchandise Volume, the total amount of sales over the campaign duration.

Table 1 shows the performance of our method in each iteration, compared with a static baseline policy trained by CQL(Kumar et al., 2020) on a pre-collected dataset. We can see that the BuyCnt of our policy improves steadily through iterations, and our method consistently outperforms the baseline across all metrics.

We conduct ablation studies for a deeper analysis of the how different components work with each other in our method. Specifically, we aim to answer the following questions: (1) Do trajectory-wise exploration and trajectory-wise exploitation operate in close conjunction instead of being two independent components? (2) Does the reward model effectively reduce the influence of stochastic rewards in Robust Trajectory Weighting? (3) Does SEAS achieve the theoretical safety bound in practice? (4) While achieving the same safety bound, does SEAS sacrifices less performance than other baseline safety-guaranteeing methods?

To answer these questions, we conduct extensive experiments in the simulated environment described in Section 6.1.

To answer Question 1. We develop variants of TEE to delve deeper into how trajectory-wise exploration and trajectory-wise exploitation work together. We focus on one data-collection process followed by one offline training process. We omit SEAS in this experiment, to focus solely on TEE. An IQL policy is used as base policy $\pi$. Details of the variants are presented as follows.

• •

  TEE is implemented as described.

• •

  w/o T-explore removes trajectory-wise exploration (i.e. PSN). Instead, traditional ASN is used for $\pi_{e}$. To ensure a fair comparison, we control the strength of ASN to guarantee that the average return in $D$ are equal to that of PSN.

• •

  w/o T-exploit removes trajectory-wise exploitation (i.e. Robust Trajectory Weighting). After collecting $D$ by PSN, we train a new policy with uniform sampling.

• •

  w/o TEE removes TEE. ASN is used for exploration, and uniform sampling for training.

Table 2 presents the performance of policies trained under different settings, and their performance gain over the base policy. We observe that TEE achieves significant performance improvement over the base policy under various budgets, while the absence of either component hurts its effectiveness. Interestingly, eliminating Trajectory-wise Exploitation leads to a substantial performance degradation. The reason behind this is that datasets collected by PSN contains a large fraction of undesirable actions, which are imitated by conservative algorithms. The above observation indicates that Trajectory-wise Exploration and Trajectory-wise Exploitation are inherently interconnected components, and their combination contributes substantially to the enhancement of the algorithm’s performance.

To answer Question 2. We test the effectiveness of reward model in environments with different degrees of stochasticity. We first construct simulated environments with various stochasticity by controlling the variance of impression numbers per time step. In each environment, we collect an exploratory dataset with the same policy, and use two different data sampling strategies to train a policy on the dataset: a) Our proposed robust trajectory weighting. b) Trajectory weighting with the raw rewards $r_{i,t}$ instead of the reward model’s prediction $\bar{r}_{i,t}$. We compare the performance of trained policies, to examine how reward model benefits the trajectory weighting method.

The results of the experiments are shown in Figure 6(a), where $R_{robust}$ denotes the return of policy trained with Robust Trajectory Weighting, $R_{raw}$ represents the return of policy trained with raw rewards, and $R_{base}$ the return of the data-collecting policy. Figure 6(b) shows the probability distributions of the impression number, where $U_{[a,b]}$ denotes a uniform distribution over $[a,b]$. The red line in Figure 6(a) indicates that the reward model is increasingly useful as the stochasticity of the environment intensifies. Moreover, the blue bars in the figure exhibit values lower than 1 in high stochasticity instances, which reflects that raw stochastic rewards could be misleading signals for trajectory weighting.

To answer Question 3. We fully evaluate the safety-ensuring ability of SEAS by setting different values of $\epsilon$ and observe the rate of performance decrease $1-J(\pi_{e}^{k})/J_{s}$, where $\pi_{e}^{k}$ is the policy produced by SEAS. We expect the safety constraint $1-J(\pi_{e}^{k})/J_{s}\leq\epsilon$ to be satisfied. In this experiment, we take a USCB (He et al., 2021) policy as the safe policy, and obtain its $Q$ function through fitting a dataset collected by itself.

From Table 3, we observe that SEAS consistently satisfies the safety constraint over a wide range of input $\epsilon$. Interestingly, for small $\epsilon$ values, the exploration policy generated by SEAS even outperforms the base policy.

To answer Question 4. We compare SEAS with two baseline safety-ensuring methods in terms of performance sacrificing, while ensuring safety to the same degree ($\epsilon=0.05$). One USCB (He et al., 2021) policy is leveraged as the safe policy. In this experiment, we start from an IQL policy, preserve the design of TEE, and substitute SEAS with baselines presented below. Strength of PSN is $\sigma=0.05$.

• •

  Small noise. We limit the strength of PSN at a low level, by setting $\sigma=0.01$.

• •

  Fixed range. As proposed in (Mou et al., 2022), the safe exploratory action $a_{e}$ is given by $a_{e}\leftarrow clip(\pi_{e}(s_{t}),\pi_{s}(s_{t})-\xi,\pi_{s}(s_{t})+\xi)$. The safe policy $\pi_{s}$ is the same as that in SEAS, and $\xi$ is set to $0.1$.

The results are presented in Table 4. We also show the result of not imposing any safety constraint, which is a performance upper bound. SEAS significantly outperforms the two baselines in terms of the average return of the trained policy. This observation suggests that the adaptive design of SEAS allows for minimal performance sacrifice.