Treatment Effect Estimation Amidst Dynamic Network Interference in Online Gaming Experiments

A/B testing is heavily used to drive product decisions in the dynamic landscape of technology and digital platforms. Central to the effectiveness of these tests is the Stable Unit Treatment Value Assumption (SUTVA) (Rubin, 1980), which posits that the outcome of any unit (e.g., a user) in an experiment is unaffected by the treatment assignment of other units. This assumption is critical for unbiased estimation of treatment effects. However, in many real-world applications of A/B testing, particularly within online gaming, the SUTVA assumption is commonly violated.

Online games, such as the multiplayer online battle arena (MOBA) game, frequently set up strategies to match players into teams. Connections and interactions between players are temporarily formed for the span of a single game session. Players’ experiences are inherently affected by teammates and adversaries. For instance, consider an online experiment to assess the impact of game difficulty – a scenario where players in the treatment group are assigned a less challenging version of the game. The assignment to treatment and control groups is delineated before the game starts. However, once the game session unfolds, the game difficulty level may be re-defined based on all the players’ treatment assignments. In such a team-oriented environment, the exposure of a single player to a lower difficulty setting can ripple through the entire team, altering the actual treatment receipt of all involved and potentially affecting the game’s outcome. And since the data are collected afterwards, interference among players becomes an inevitable issue. Furthermore, the network structure within online gaming is inherently dynamic and short-lived, continually evolving as game sessions conclude and new ones unfold. Each game session reshapes the network connections as players disperse and regroup. During the experiment period, players may enter multiple games and receive different numbers of in-game treatments. This post-hoc network interference, stemming from the interactions between players and complicated by the number of treatments players received, may lead to biased estimations of causal effects.

Recently, various experimental designs have been proposed to address the challenges posed by network interference, particularly in terms of mitigating spillover effects and enhancing the accuracy of treatment effect estimation. Notably, cluster-based randomized experiments (e.g., Aronow and Middleton, 2013; Eckles et al., 2016; Ugander et al., 2013; Walker and Muchnik, 2014), have been at the forefront of these developments. These experiments group subjects into clusters based on their network connections, and then randomize the treatment at the cluster level rather than the individual level. Multi-level experiments (Hudgens and Halloran, 2008) introduce sophisticated frameworks where interventions are administered at multiple hierarchical levels within the network. Additionally, mixed experiments, as explored by (Karrer et al., 2020), simultaneously employ both unit-level and cluster-level randomization to capture a broader range of network effects.

Despite the innovative nature of these online experimental designs, there can be methodological, ethical, and cost-related challenges in applying these designs in the online gaming environment. For instance, it is not realistic to completely isolate players into distinct clusters without affecting their typical gaming experience. Players often interact in a dynamic, interconnected online space, making it difficult to create clear-cut, isolated clusters for experimentation. Additionally, cluster-based or multi-level experiments are typically inefficient to implement both computationally and financially.

More recent studies have focused on causal effect estimations under interference. One line of research particularly emphasized the concept of partial interference, which assumed that network interference is restricted within the non-overlapped groups (e.g., Sobel, 2006; Tchetgen and VanderWeele, 2012; Liu and Hudgens, 2014; Basse and Feller, 2017). Another line of work studied the causal estimation in a more generalized setting, with arbitrary interference or introducing specific network graphs as extra information into the models (e.g., Bowers et al., 2013; Manski, 2013; Goldsmith-Pinkham and Imbens, 2013; Forastiere et al., 2016). Some other papers discussed causal effect estimations under potentially mis-specified or unknown network interference (e.g., Halloran and Hudgens, 2016; Choi, 2016; Forastiere et al., 2016). Complementing these studies, (Karwa and Airoldi, 2018) presented an innovative approach for causal estimations by developing a semi-parametric form of potential outcome. The potential outcome function includes the specification of the exposure mapping with network neighborhood information. (Sävje et al., 2017) investigated the large sample properties of generalized treatment effect estimation under the unknown interference structure.

While existing literature offers valuable insights into causal inference in the presence of network interference, treatment effect estimation in dynamic environments like online gaming remains less explored and comes with its set of unique challenges. Specifically, within a short period, players repeatedly start new gaming sessions with potentially different team members. In contrast to applications like traditional social networks with a static and long-term network structure (e.g., Xu et al., 2015; Gui et al., 2015), the mechanisms of interference are not well-defined or consistent over time, thus all players can be subject to treatment spillover through a random process that experiment designers cannot control. Nevertheless, we will adopt similar framework as some of the previous work and define a more general exposure mapping in such settings.

Causal analysis within the literature on online gaming has been sparse. (Meng et al., 2019) proposed a general causal analysis framework ‘exCause’ for the real-time game sessions. (He et al., 2021) employed causal inference techniques, specifically focusing on the estimation of heterogeneous treatment effects, to assess the impact of software updates and patches in games. (Dong et al., 2020) innovated the novel Attention Neural Networks to refine the estimation of causal effects using mobile gaming data. However, it is notable that the network interference problem for causal inference, specifically in the online gaming area, remains unexplored.

In this study, we introduce an innovative framework that combines causal inference under network interference with the context of online gaming. Our main contributions can be summarized as follows:

• •

  We formalize the problem of treatment effect estimation in online gaming with post-hoc network interference where the networks are dynamic and ephemeral.

• •

  We develop an estimator for treatment effect with interference, specifically when the completely randomized experimental design is required and the network structure is not explicitly known.

• •

  We evaluate the proposed estimator through a series of simulation studies with various interference settings, and demonstrate the accuracy and robustness of our approach, especially in comparison to more naive methodologies.

• •

  We validate the effectiveness of the proposed framework through its deployment on real-world data from a mobile gaming online experiment.

The rest of the paper is organized as follows. In Section 2, we introduce the experimentation process in online mobile games, and discuss the limitations of the naive estimator for the Average Treatment Effect (ATE). In Section 3, we define the specific causal estimand of interest and describe our proposed framework for estimation in detail. In Section 4, we conduct a series of simulation studies under various interference scenarios and compare the performance of different estimators. Section 5 includes the application and validation of our proposed estimator using data from a real-world mobile gaming experiment, thereby demonstrating its practical efficacy.

Mobile gaming has seen unprecedented growth in recent years, becoming a significant part of the digital entertainment industry. Specifically, MOBA games have not only attained immense popularity but also cultivated a vast, global player community, largely due to their captivating gameplay mechanics and dynamic player interaction. Take a 5v5 game as an example, 5 players work together as a team to achieve objectives and defeat the opposing team. In this study, we ignore the influence of the opposite team and treat the games as the ‘Player versus Environment’ (PVE).

In this study, we focus on the causal analysis of the ‘treated game’ feature, an intervention designed to enhance player engagement and promote a more active gaming community. Specifically, when the ‘treated game’ is activated in a game session, it modulates the game’s experience level for the players. In an ideal scenario without interference among players, those in the treatment group would consistently participate in the ‘treated game’, while those in the control group would never engage in the ‘treated game’.

Our experimental design incorporates a substantial segment of the online traffic, utilizing 40% of it to gauge the effects of our treatment. This allocation is divided evenly, with 20% of users randomly assigned to the treatment group and the remaining 20% serving as a control group for a comparative analysis. This process can be seen as the initial stage of the randomized treatment assignment mechanism.

In the second stage, the players engage in games with an unknown team-matching process. The actual treatment status, or in other words, the activation status of the ‘treated game’, for each player in a given game is determined based on a specific criterion: all team members will receive the treatment if at least one member was initially designated to receive the treatment. Under this criterion, players who were initially assigned to the treatment group will consistently receive the treatment throughout the experiment. In contrast, those initially designated as ‘control’ may experience a shift in experience due to the influence of their teammates from the treatment group. During the experimentation period, each player may participate in multiple gaming sessions. An illustration of this process is shown in Figure 1.

Under this setting, players can be further categorized into three distinct groups based on their original treatment assignment and their engagement with the ‘treated game’ feature:

• •

  Treatment Group: Players with an initial treatment assignment who consistently participate in the ‘treated game’;

• •

  Control-Mixed Group: Players originally assigned to the control group who have played both the standard (control) game and the ‘treated game’;

• •

  Control-Control Group: Players initially assigned to the control group, who exclusively play the control game without any exposure to the ‘treated game’.

In the rest of the paper, we adopt the following notation. For each player $i\in\{1,...,N\}$, let $Z_{i}\in\{0,1\}$ be the initial treatment assignment. We let $M_{i}\in\mathbb{N}$ denote the number of treated games played by player $i$ and $Y_{i}\in\mathbb{R}^{+}$ is the outcome of interest. As for the three treatment receipt groups, we use $T\subset\{1,...,N\}$ to denote the set of players in the treatment group; $C_{1}\subset\{1,...,N\}$ the player in the control-mixed group and $C_{0}\subset\{1,...,N\}$ as the player in the control-control group. We let $X_{i}\in\mathcal{X}$ denote any pre-experiment covariates we have for each player.

In this section, we move beyond the naive estimators and formally define the causal estimand of interest in our setting, together with assumptions that allow us to define a valid estimator of the average treatment effect.

In this section, we evaluate the proposed framework with simulation studies reflecting different scenarios of network interference. The source codes are provided in this link¹¹1https://github.com/YuZoeyZhu/-KDD-2024-Network-Interference-Online-Gaming/tree/main . We assume a 5v5 game setting, i.e., $5$ players participate in each game. Let the total number of players $N=1000$. Under the completely randomized treatment assignment, we generate the treatment assignment $Z_{i}\sim Bernoulli(0.5)$ for each player $i$. For each player, we also generate a single pre-experiment covariate $X_{i}\sim Beta(0.5,0.5)$. The player-matching process is simulated as follows. For the $j_{th}$ round of the game,

• •

  Sample the number of treatment players $n_{T(j)}\in\{0,1,...,5\}$ based on the pre-set probability. The number of control players $n_{C(j)}=5-n_{T(j)}$;

• •

  Sample $n_{T(j)}$ players from the treatment group with probability $p(i\mid Z_{i}=1)\propto\frac{0.8}{|T|}+0.2(\frac{X_{i}}{\sum X_{i}})^{2}$;

• •

  Sample $n_{C(j)}$ players from the control group with probability

  $\displaystyle p(i\mid Z_{i}=0)$

  $\displaystyle\propto\begin{cases}X_{i}\bm{1}\{X_{i}<0.2\}&\text{if }n_{T(j)}=0,\\ \frac{0.8}{|C_{0}|+|C_{1}|}+0.2\left(\frac{X_{i}}{\sum X_{i}}\right)^{2}&\text{ o.w.}\end{cases}$

We generate $N_{g}$ rounds of game. Let $M_{i}$ denote the number of ‘treated games’ player $i$ participated. We assume the potential outcome follows the following distribution with mean depending on both the feature $X$ and number of treatment games $M$:

This synthetic data generating process mimics the distribution of real data where the heterogeneity of $Y$ increase for larger $X$. For example, if $X$ measures the level of activity of a player, active players tend to have larger variance in $Y$ than less active players.

The simulation setting allows us to control the amount of interference by varying the probability of sampling players in the treatment group. More specifically, we consider three different settings with a progressively increasing level of interference in the control-mixed group from Case I to Case III. The number of games $N_{g}$ and the distribution of exposure $p(M)$ are summarized in Table 1. Figure 2 illustrates the distribution of sample size proportions across different values of $M$ for all three groups of players: treated, control-mixed, and control-control group. The amount of interference, increasing from I to III, is the sample size proportion of control-mixed group under each $M$ relative to the treatment group, which can be compared via the overlapping area under the blue line (the control-mixed group) and red line (the treatment group). In addition, in Case I, the sample size proportions are relatively small in both the treatment and control-mixed groups at lower $M$ levels, which poses more challenges in estimating $\tau(m)$ when $m$ is small. In Case II, the probability of $M$ in the control-mixed group is skewed to the right with the mode at 3. The sample size proportions are slightly smaller in both small and large $M$s, but is overall more balanced comparing with Case I. In Case III, the probability of $M$ in the control-mixed group and treatment group are both skewed to the right with similar modes. In contrast to Case I, the sample size proportions are small in both groups at higher $M$s. In all three cases, around $10\%$ of samples are in the control-control group.

In the simulation, we assume $\mu(x)$ is known, which is a reasonable assumption when there are enough pre-experiment data without treatment. Thus we focus on the estimation of the counterfactual under non-zero exposures. We truncate the $M$ at 10, treating $M$ as categorical with values in $\{0,1,2,...,9,10+\}$ and apply the XGBoost classification model (Chen and Guestrin, 2016) to estimate the propensity scores $\hat{e}_{m}(x)$. We also evaluate a modified version of the proposed estimator by using only the players in the treatment group to estimate $\hat{\mathbb{E}}\left[Y_{i}(m,1)\right]$ in equation 6, which we refer to the ‘proposed estimator without control-mixed’.

Let $\bar{Y}^{(m)}$ be the average outcome for players in the treatment group who are exposed to $m$ treated games, $\bar{Y}^{\mbox{c}}$ and $\bar{Y}^{\mbox{cc}}$ be the average outcome for players in the control group and control-control group respectively. We compute our estimator to two naive estimators ignoring interference:

• •

  Naive: This is the simplest DiM estimator ignoring interference and confounding and only computes the contrast between players in the treatment group who are exposed to $m$ treated games with the whole control group

  (8)

  $\displaystyle\hat{\tau}^{\mbox{naive, 1}}$

  $\displaystyle=\bar{Y}^{(m)}-\bar{Y}^{\mbox{c}}$

• •

  Naive - w/o Control-Mixed: Alternatively, we may remove all players subject to interference and consider the comparison to the control-control group only,

  (9)

  $\displaystyle\hat{\tau}^{\mbox{naive, 2}}$

  $\displaystyle=\bar{Y}^{(m)}-\bar{Y}^{\mbox{cc}}$

To evaluate the performance of each estimator, we generate 100 datasets in each case and compare the mean and 95% uncertainty interval of the estimated effects with the truth for $\hat{\tau}(m)$. As shown in Figure 3, in all three cases, the first naive estimator tends to overestimate the effects in all levels of $M$, whereas the second naive estimator ignores the control-mixed group’s consistently overestimated effect. This is as expected as in the former case, $\mathbb{E}\left[Y(0,0)\right]$ is over-estimated as some of the players in the control group are exposed to treatment through interference; whereas in the latter case, $\mathbb{E}\left[Y(0,0)\right]$ is under-estimated as the players in the control-control group are more likely to be less active players. Both versions of the proposed estimator show improvement over the naive estimators in general. We also observe that the proposed estimator without using the control-mixed group exhibits larger uncertainty in general, especially for levels of $M$ with a small sample size. By incorporating both the treatment and control-mixed group, the proposed estimator is able to achieve the smallest bias with low variation.

This simulation study offers clear insights into the performance of our estimator across various probabilities of treatment player matching. In real cases, these matching probabilities can be strategically adjusted. For instance, selecting an appropriate matching probability can help maintain a balance between enhancing the user engagement of new features and improving the accuracy of the estimator. It can be further explored to manage and control network interference, thereby aligning the experimentation with practical business considerations.

This simulation study provides a straightforward insight of the performance of the estimator under different treatment player matching probability. In the real case, the matching probability can be adjusted on purpose with business concerns. And we discuss it as one of the strategies to control the interference.

In this case study, we consider a online experimentation for the ‘treated game’ feature of a MOBA game from Tencent. We obtained the experiment dataset with a total of 58,565 players for a two-week experimentation period. And we collected a pre-experiment dataset from the same set of players, covering the two weeks immediately preceding the start of the experiment. The control-control group takes around 7.8% of the whole sample.

In this experiment, we focus on monitoring a specific target metric (TM) for each player. This business metric is a measurement of a player’s engagement in the mobile game. An increase in TM generally indicates a higher level of player involvement and satisfaction with the game, which can lead to increased loyalty, longer-term player retention, and potentially greater revenue through in-game purchases. We expect to see a significant increase in TM with the new ‘treated game’ feature.

This specific TM, being inherently positive, displays a distribution with decreasing tread and a long right tail. Figure 4 illustrates the distributions of TM across the three groups, comparing the experimental period and the pre-experimental phase data. In both datasets, the TM for the control-control group is relatively smaller, aligning with the inference that this group consists of less active players. Additionally, the pre-experiment TM generally surpasses the experiment TM, which could be attributed to various factors, such as in-game events, public holidays, etc.

The pre-experiment features, which serve as potential confounders, encompass a variety of descriptors that capture different characteristics of the players. These include metrics that measure players’ gaming abilities, their levels within the game, and their in-game purchases.

As for the pre-processing of the data, we treat the observations with TM $\geq 60$ as outliers and remove them from the data. We also truncate $M$ with a threshold of 21.

In conclusion, our comprehensive study provides a novel framework for analyzing causal effects in the context of online gaming, where traditional A/B testing faces the challenges of post-hoc network interference. By focusing on the ‘treated game’ feature of the mobile MOBA game, we have demonstrated the potential of our proposed framework to accurately estimate treatment effects amidst the complex dynamics of player interactions.

It is worth noticing that certain limitations could impact the generalizability and applicability of our findings. First, our model relies on assumptions that may not hold across all gaming environments or user demographics. Moreover, our model presumes that the post-hoc network interference is uniformly distributed across all players, an assumption that might oversimplify the complex and often unique interactions between players. Besides, potential selection biases may not be fully accounted for due to the unknown team-matching strategy. For future work, we can further refine and validate our model with the underlying assumptions. For instance, exploring models that account for heterogeneity in player behavior or that explicitly model the unique network structures inherent in different gaming communities.

The potential applications of our work extend beyond the realm of online gaming. The principles and methodologies we have outlined could be adapted to other digital platforms and social networks where user engagement and interaction are pivotal to the product’s success. Specifically, we can generalize our work to the exploration of dynamic network interference scenarios, such as those encountered in real-time strategy or ride-sharing platforms like Uber with continuously evolving user interactions.