Causal Estimation of Position Bias in Recommender Systems Using Marketplace Instruments

We have a group of users (viewers of feed or PYMK), who interact with or respond to a set of items (ads or candidates for PYMK) under certain context, where the response event can be defined as a click, a conversion, or other deep funnel feedback from the user. We seek to study the effect of an item’s position changing on the user’s propensity to interact with it, all else held equal.

Let $I$ and $J$ denote items and requests and let $i$ and $j$ be a specific item and request, respectively. For each request, multiple items are shown to the user based on the underlying serving system with various relevance models. In request $j$, let $e_{ijk}=\mathbbm{1}(\text{response}_{ijk})$ be the counterfactual response indicator given that item $i$ is shown in position $k$ in request $j$. In this work, we only consider binary responses. We define the item-level and system-level position effect between any positions $k_{1}>k_{2}$ as

These position effects could be interpreted as the expected change in users’ response once an item is moved from position $k_{2}$ to $k_{1}$. In this work, our goal is to estimate these position effects in two use cases we face at LinkedIn. Although the contexts are specific, the methodology proposed to identify and estimate these effects are general and can be applied to other use cases. For example, one could also focus on the request-level position effects defined as $\tau_{j}(k_{1},k_{2})=\mathbbm{E}_{I}[e_{ijk_{1}}\mid J=j]-\mathbbm{E}_{I}[e_{ijk_{2}}\mid J=j].$

To estimate $\tau_{i}$ and $\tau$, we rely on observations at the request level. Specifically, each observation corresponds to a (item, request, position) tuple. Table 1 shows some example observations in this format.

The key challenge in estimating the position effects with such observational data is that the responses are confounded by factors such as the relevance. We address confounding using an instrumental variables approach.

We leverage instrumental variables (Angrist et al.,, 1996) provided by past controlled experiments in ranking algorithms to estimate the position effect.

We first focus on a specific item $i$ and consider the item-level position effect $\tau_{i}(k_{1},k_{2})$. Suppose that there was a past request-side experiment that affected the items’ position, and only affected users’ response through its impact on the positions. Consider all requests in which item $i$ was shown either in position $k_{1}$ or $k_{2}$ during the experiment. Let $W_{j}$ be the position of the item. Let $Z_{j}$ be the treatment request $j$ received in the selected experiment and let $Y_{j}$ be the corresponding binary response (click, connection request etc.). We consider the following IV model

where $\bm{X}_{j}$ is the request-level covariates. $\pi_{2}$ is the position effect of interest. A few comments are in order. Equation 2 imposes a linear restriction on the position effect. That is, it computes an average position effect, such that, e.g., an increase in three positions has the same effect as three times the increase in a single position. Further, this assumption rules out position effects that vary based on what the starting position, $k_{2}$ is. To relate this assumption with the notation in Equation 1, it imposes that $\tau_{i}(k_{1},k_{2})=c_{i}\times(k_{2}-k_{1})$, for some constant $c_{i}$. When the position effects have such a linear relationship, the indirect least squares (ILS) or two-stage least squares (2SLS) estimator of model 2 with $W_{j}$ treated as a continuous variable provide an unbiased estimate of $c_{i}$: $\mathbb{E}[\hat{\pi}_{2}]=c_{i}$.

The instruments, $Z_{j}$ we pick are relevant because they are part of routine model tuning at LinkedIn, which operate precisely through the channel of reranking items. Finally, we assume the exclusion restriction: that the instruments only affect outcomes, $Y_{j}$ via their effect on positions, $W_{j}$. We find this assumption plausible by picking our instruments carefully, leveraging the two-sided nature of the marketplaces we study. In the ads marketplace, we have users and advertisers; in PYMK, we have two kinds of users: viewers (those who send connection invites) and candidates (those who receive connection invites). We leverage institutional knowledge in the construction on the rankings, which are determined by an ensemble AI model that includes a model that predicts user-side relevance (whether the user will click on the ad or send a connection invite) and advertiser-side relevance (how much do sponsored advertisers bid for a particular user in ads or what the predicted connection acceptance probability is in PYMK). As we seek to estimate a item-side position effect, i.e. the effect of changing only the position at which an item appears on the interaction probability of users, we must tackle confounding from user-side relevance of items. We do this by picking instruments that tune the model concerning the opposite side. Such model tuning occurs to improve relevance for sponsored advertisers in ads and the connection acceptance probability in PYMK. At the same time, they do not directly affect user-side relevance. Thus, these instruments induce exogenous variation in positions such that they do not affect outcomes, $Y_{j}$ except through their effect on rankings, thereby satisfying the exclusion restriction.

To estimate $\tau(k_{1},k_{2})$, it is intuitive to consider an estimator based on the estimated item-level position effects. For example, let $i=1,2,\ldots,N$ be a random sample of items. One can estimate $\tau(k_{1},k_{2})$ with

where $\tau_{i}(k_{1},k_{2})$ is the ILS or 2SLS estimator based on model (2). Although this estimator is unbiased for $\tau(k_{1},k_{2})$, its variance can be hard to compute due to dependencies among the item-level estimators in certain applications. To see this, note that in Table 1, responses in request 1 appear in the estimate of $\tau_{1}(1,2)$ and $\tau_{2}(1,2)$. Since responses in the same request are likely to be related, the naive variance estimator of $\hat{\tau}$ can be biased. In an extreme case where there can be at most one positive response in each request (as in our ads application), this relation cannot be ignored. To deal with this structured dependency, we randomly sample one row (item) in each request and only use responses in the sampled rows to estimate $\tau$. This sampling step ensures the independence of observations in the inference for $\tau(k_{1},k_{2})$.

In the following sections, we provide details on how to apply this IV model in our two applications. We will go over what outcomes we estimate the position bias over, what our instrumental variables design is, which data we use, and what our empirical estimates are. In Section 3, we study the ads application and in Section 4, we look at the PYMK application.

Experiments that affect ad position often do so exactly by changing the models that predict ad relevance. Such experiments, however, rule out using predicted relevance as a feature in our model. This is a problem, because we know that an ad’s true relevance should be correlated with both position and click probability; hence our best estimates of relevance should be included as a feature in the model. Naturally, if those estimates are different under treatment and control, we cannot include them as features.

This motivates our instrument selection, a prior experiment that affected ad positions not by changing the predicted CTR (and hence the estimated ad relevance), but by experimenting with the ad bidding procedure itself, on the user level. We leverage an experiment that compared the existing bidding procedure, to a new reinforcement learning-based, automatic bidding method, to help advertisers determine bid values. Observe that this experiment has no impact on predicted relevance. This experiment naturally has a strong impact on ad position, but still allows us to use consistent ad relevance estimates as features in the IV model.

Observe that an instrumental variables model works on the relevance of the instrument, i.e. on the nonzero effect of the instrument on the treatment, in this case the feed position. If we consider the full set of ad campaigns (a campaign is the ads from a specific advertiser) together, this regression is not meaningful. For a given user and query, several ads will appear; and moreover, if ad A is in position 1, ad B cannot be in position 1, because it is already occupied. This introduces problematic correlation. Consider the toy example in which we have only two ads, A and B. Then the position of A exactly determines B and our regression coefficients are not identifiable. While in our case we likely could find regression coefficients, they are not trustworthy due to this collinearity. Moreover, if we were attempting to perform a stage 1 regression, the coefficient is essentially meaningless when we consider the full set of ads in one regression.

We therefore assign the level of estimation for this regression to be each ad campaign. By reducing the data to those requests featuring a specific ad campaign, we only sample one observation per request, since the same ad campaign does not typically appear multiple times in a single request. Thus, by only sampling at most a single observation per request, we remove problematic correlation across observations in the same request. For this analysis, we considered the 100 largest ad campaigns, which amounts to 44,349,019 ad views across the top 30 feed positions on one day. There are two treatment variants, randomized at the user level.

We consider three model specifications to estimate the position effects (Table 2). Specification 1 is the simplest IV specification, which seeks the effect of feed position on 1(click) using the treatment as our instrument. Specification 2 is Specification 1 with the addition of a control variable, predicted CTR (PCTR), attempting to capture ad relevance. Recall that the ad relevance model remains stable throughout the duration of the experimentation we use as instruments. Specification 3 is an ordinary least squares (OLS) non-causal baseline, which does not include the instrument (Experiment = treatment). We prefer Specification 2; this allows us to leverage the past randomness and estimate a causal effect, while also controlling for any residual effects of predicted ad relevance in the model.

The most robust IV results will correspond to a strong stage 1 regression, meaning a low p-value and a relatively large coefficient; otherwise, we are unlikely to be able to detect a causal effect. Moreover, it is reasonable to assume that the bidding experiment will affect different ad campaigns differently, and indeed we have a range of stage 1 regression information.

Figure 1 shows the coefficient on the bidding experiment’s effect on feed position, for the 30 ads with the most feed views, along with 95% confidence intervals. A negative coefficient here indicates that the bidding experiment used as the instrument on average moved ads up in the feed (decreased their feed position). Results are color-coded according to their distance from zero; campaigns with a significantly negative coefficient in red, positive coefficient in blue, and not different from zero in black. While the plurality of ad campaigns (47%) have a negative impact, we see many without a significant impact (43%), and a few with a positive impact (10%).

For Figure 2, we select the ads with the most statistically significant stage 1 coefficients among the 50 ads with the most feed appearances. This gives us an estimate of position bias for campaigns with the most robust available IV estimation.

The position effect here is our coefficient on feed position. These results show a position effect across several model specifications, including a model accounting for predicted click-through rates (our preferred model specification). We see these results for ad campaigns in which our instrument, namely the bidding experiment, had a strong impact on the feed position in which the ad appeared. Hence interpretation should be constrained as follows: in ad campaigns that were strongly affected by the bidding experiment, there is a robust and statistically significant effect of ad position on click-through rate. These ad campaigns might differ systematically from the campaigns without as large an impact from the bidding experiment, and that difference could apply to position effects as well.

For more details, in Table 3 we display the full regression information for ad campaign A from Figure 2. Across the model specifications, moving an ad to a higher feed position decreases click probability; higher PCTR is consistently correlated with greater click probability. In model (2), the coefficient estimating the impact of feed position drops in magnitude. This is likely because we include PCTR in this model, and the feed position coefficient in model (1) accounts for lower feed position being correlated with higher PCTR.

The coefficients on feed position are consistently negative, which lines up with intuition that moving an ad up higher in the feed (that is, decreasing feed position) will raise click probability. For our preferred specification, the coefficient for ad campaign A on feed position is -0.0007 (with a standard deviation of 0.0003). That is, if we move ad A from position $k$ to position $k-1$, the probability of a click will increase on average (over users) by $+0.07\%$. Compared to the constant estimate of $0.58\%$, this amounts to an increase of $12\%$. Hermle and Martini, (2022) also consider position bias for ads, comparing specifically CTR in the top two ad positions with a regression discontinuity model where the running variable is the difference in relevance score predictions for a pair of adjacently positioned ads. They find that the effect of going from the lower to the higher position on CTR is an increase in probability of $50.81\%$. This discrepancy is intuitive; the regression discontinuity model gives an estimate for a pair of ads where the relevance scores are exactly equal, and hence CTR is highly likely to be affected by position. The instrumental variables model includes relevance as a predictor, but gives an estimate across ads with a range of predicted relevance scores, and therefore should be expected to give an estimate with lower magnitude.

The PYMK offline ranker is an ensemble model consisting of four constituent AI models: 1) pInvite: a model predicting the probability that the viewer sends the candidate an invite, 2) pAccept: a model predicting the probability that the candidate accepts an invite from the viewer, 3) destSessionUtility: a model predicting the interaction between the two once connected, and 4) destRetentionUtility: a model measuring broader engagement of the candidate on linkedin.com after connecting with the viewer. Candidates are ranked based on a multi-objective optimization function (MOO), which is a function of the predicted values of each one of these four models.

In particular, the pInvite model is trained using historical PYMK data, i.e., candidates who have been impressed in viewers’ PYMK and their corresponding outcomes (whether the viewer sent an invite) become labels for training. Ignoring position effect during model training can potentially yield biased estimates if two candidates’ labels are treated equally by the model but one candidate was positioned lower than the other.

Our proposal for causal identification is to use A/B testing in the constituent AI models of the PYMK MOO score. Specifically, we look at experimentation in the pAccept model and the weights of each of the individual AI models in the overall MOO score.

Since the experiments are randomized on viewers, any systematic difference we see in the positions candidates take between the PYMK grids of viewers in the treatment group and those in the control group is random and therefore unrelated to viewer characteristics.

Thus, we may use the PYMK AI experiments as instruments to exogenously move the position of candidates, which we may then use to identify the position effect on the probability of sending PYMK connection invites.

Our dataset is at the edge-level, meaning that each observation corresponds to one (viewer, candidate, query) tuple. It consists of 38,553,973 observations, 3,823,649 unique viewers, 24,596,318 unique candidates, and 22 PYMK related reasons. There are two treatment variants, randomized at the viewer level. We also observe pInviteScore: a PYMK AI model that indicates the likelihood that the viewer will send an invite to the candidate, as well as pInviteOutcome: the final outcome of whether the viewer sent a connection invite to the candidate or not. Finally, we observe the rank each candidate was displayed to the viewer in the latter’s PYMK query, as well as the session depth, i.e. the number of recommendations the viewer was served in a particular session.

The independent variable we seek to model is pInviteOutcome. We use the instrumental variables (IV) framework for our causal results here. To identify multiple endogenous variables, we require multiple instruments. Given we have a single binary treatment, we interact it with relatedReason to give us some two dozen instruments.

Given this setup, we run the following edge-level regression specifications:

Specification 4 is the simplest IV specification that seeks the effect of onlineMooRank on pInviteOutcome. Specification 5 is Specification 4 with the addition of the control variable, pInviteScore. Since our treatment variants did not explicitly seek to randomize PYMK positions between treatment and control, there may be systematic match quality differences between the two variants. For this reason, pInviteScore acts as a way to control for match quality.

Our preferred specification is 6, which is Specification 5 with an additional endogenous variable, sessDepth. Again, the treatment did not just randomize PYMK position. If session depths vary systematically between treatment and control, that may indicate a difference in match quality, and this difference should not be driving our IV results. As sessDepth responds to treatment, we include it as an endogenous variable in this specification. Finally, as a baseline, we include ordinary least squares (OLS) estimates that do not involve any instruments.

The coefficient on onlineMooRank is our position effect of interest. We find consistently non-zero position effects across our regressions, with our IV point estimates spanning the range between -0.063 and -0.038. In words, this means that when a candidate is bumped up one rank on the PYMK page, then on average they are likely to see between 3.8% and 6.3% higher probability of receiving a connection invite.