DPR: An Algorithm Mitigate Bias Accumulation in Recommendation feedback loops

Recommendation systems make recommendations and collect user feedback, and then train the next model based on collected feedback, a process that induces a feedback loop (Figure 1). The recommendation system forces the exposure mechanism to change the probability of an item being exposed to the user. For example, a personalized recommendation system aims to recommend items that the user prefers as much as possible. Such items will have a higher probability of being exposed to users than others. As a result, we can observe more interactions of such items in the collected feedback, and the next model trained on such feedback might infer that the user likes it and continue to recommend it. However, the recommendation system cannot know all user preferences perfectly which leads to biased feedback data. The new model optimizes its parameter $\hat{\Theta}$ can be seen as a maximum likelihood estimator under the feedback data:

where $P\left(\bm{S}\mid\bm{Y}\right)$ indicates the user preference we can learn from the feedback data, and $P_{\Theta}\left(\bm{S}\mid\bm{Y}\right)$ is the inference of user preference by the new model. It is obvious from Equation 1 that models try to fit the feedback data, which means that models will inherit the biased user preferences in the collected feedback data and make sub-optimal recommendations. From this, we make the assumption that the original recommendation system does affect the exposure mechanism, which is reflected in the feedback data and will be inherited by the new model in the feedback loops.

To validate our assumption, we choose two datasets for analysis: Coat¹¹1https://www.cs.cornell.edu/ schnabts/mnar/ and KuaiRec²²2https://chongminggao.github.io/KuaiRec/ (Gao et al., 2022). Both have full-observed and partial-observed data, which better reflect the difference in exposure mechanisms under and out of the feedback loops. Coat obtains fully observed data from questionnaires under randomized control and thus does not suffer from feedback loops, while KuaiRec does the opposite and obtains data from an online recommendation system without randomized control where impact of the recommendation system and feedback loops cannot be ignored. We choose the positive interaction counts of items in feedback data as the dominant indicator of the exposure mechanism, treating the full observed data as Missing At Random (MAR) data and the partial-observed data as Missing Not At Random (MNAR) data for Coat, and both the full observed data and the partial-observed data in Kuairec are Missing Not At Random (MNAR) data. The MNAR and MAR data are representative of the biased and unbiased exposure mechanisms, respectively. We process both datasets into implicit feedback without loss of generality, the results are shown in Figure 2.

The results show clear differences between the full observed data and the partial-observed data in Coat and Kuairec. The difference between the full observed data and the partial-observed data in Figure 2 indicates that the exposure mechanism does have an impact on the user-item interaction, thus most of the datasets in real scenario are long-tailed distributions. Otherwise, MAR data is not affected by feedback loops, and items in MAR data have nearly the same ratio, meaning that items have the same probability of being exposed to users, as shown in Figure 2 (a). The MNAR data is affected by both feedback loops and exposure mechanisms, so the items have different exposure probabilities and show significantly different trends with respect to the MAR data. However, the KuaiRec dataset, which suffers more from feedback loops, shows a high similarity between the full observed data and the partial-observed data, with a significantly higher ratio of head-to-tail items in partial-observed data compared to Coat. The results between the two types of datasets justify the above assumption, the exposure mechanism suffers from feedback loops.

The exposure mechanism plays an indispensable role in the user-item interaction process by determining the exposure probability of the item to the user, which we can describe as $\bm{O}(i)$. Interaction occurs when the user is interested in the item and observes it:

$P(\bm{S}_{u,i}=1)$ indicates that the user rates the item, which means that an interaction has occurred. If the exposure mechanism is not affected by recommendation systems, such as the system makes random recommendations, which means feedback data obey MAR assumption, and items have the same opportunity to be exposed to users, leading to:

However, most recommendation systems make personal recommendations to improve the quality of recommendations and the user experience, which inevitably affects the exposure mechanism. To measure the change in the exposure mechanism, we can derive the exposure probability from the feedback data because we theoretically have access to all the data. That is:

Since our problem is focused on the exposure mechanism under the feedback loops, we further transform Equation 4 into a more general formulation of the pattern. The current exposure mechanism at T = $t$ is mostly depend on the relevance score predicted by the current model and the past exposure mechanism at T = $t-1$.

Based on Equation 5, we can calculate the difference of the exposure mechanisms after one feedback loop.

From Equation 6, we can learn that items with high popularity can have low $\Delta\bm{O}^{t}(i)$, which means that the exposure probability will be higher compared to unpopular items. This can better explain why mainstream models suffer from the ”rich-get-richer” problem. For ease of calculation, we extract the relationship between variation in exposure mechanism and static data.

If the data is collected from a recommendation system that makes random recommendations or through other methods such as surveys, we can regard it as free from the feedback loops, which means $T=0$. Then we can have an iterative computation between $\bm{O}^{t}(i)$ and $\bm{O}^{t-1}(i)$:

By Equation 8, we can calculate the $\bm{O}^{t}(i)$ at $T=t$ with no known exact exposure mechanism in the current timestamp, which can be formulated as:

where $\alpha$ is the frequency weight of the exposure mechanism that suffers from feedback loops. However, direct computation of relevance score between user-item pairs is time-consuming and the accuracy of the computed results is difficult to guarantee, so we need to find a replacement. From Equation 2, we can obtain an inequality between the relevance score and the observed ratings:

Then the lower bound on $\bm{O}^{t}(i)$ can be directly computed from the observed ratings without external information:

The calculation method of the lower bound is understandable, existing methods such as UBPR and Rel-MF typically exploit the interaction counts of items as the main source of propensity scores and achieve better performance, but they focus on the current timestamp and ignore the variability of exposure mechanisms. Otherwise, Eq. 11 tells us that it is fallacious to simply use the interaction counts of the items, and therefore it is suboptimal to compute the propensity score based on the interaction counts.

This finding makes us aware that the exposure mechanism suffers from feedback loops, which will harm the new model. So the key to unbiased recommendation is to free the model from the loops. We conduct an experiment on Coat to prove that the change in the exposure mechanism does improve the quality of the recommendation. The performance of the backbone model MF in different exposure mechanisms is presented in Figure 3. We mix various percentages of MAR data into the training set (MNAR data) to represent different exposure mechanisms. Increasing the percentage of mixing implies a closer fit to the original exposure mechanism.

As shown in Figure 3, the performance of the model does improve at the beginning, which proves that the change in the exposure mechanism does improve the performance of the model. However, the performance of the model deteriorates as the exposure mechanism closes to the original exposure mechanism up to a point since the change in the exposure mechanism is not always negative. For example, the fashion of clothes changes over time, and the corresponding styles of clothes that are popular at a certain time are bound to be popular with the public, therefore the exposure mechanism that gives preferential attention to such clothes will improve the performance of the recommendation system and user satisfaction, which is clearly different from the original exposure mechanism. Moreover, the average popularity of the final recommended items keeps decreasing as the similarity between the exposure mechanism and the original exposure mechanism increases, which indicates that the original exposure mechanism benefits the recommendation of cold items.

In summary, the effects of unknown exposure mechanisms on model performance under feedback loops are confounded, and the key to solving the problem is to maintain its beneficial effects while eliminating its negative ones.

Pairwise ranking-based methods, which learn a user’s preference for two items by maximizing the predicted score of a positive item, are always biased. Since the choice of positive and negative items depends on the observed interactions instead of the relevance of user-item. Previous studies (Saito, 2020; Saito et al., 2020a; Schnabel et al., 2016) have defined unbiasedness in terms of expectation, making the expectation of the loss equal to the ideal loss, which suffers from practical limitations, such as the high variance of the reweighted loss. Therefore, we propose a more universal definition of unbiasedness similar to (Wan et al., 2022):

Most of the paired ranking methods derive from Bayesian Personalized Ranking (BPR) (Rendle et al., 2009):

where $\mathcal{D}_{pair}=\left\{\left(u,i,j\right)\mid\mathbf{S}_{u,i}=1,\mathbf{S}_{u,j}=0\right\}$ is the set of all possible triplets for pairwise ranking. Then with Equation 13 we prove that BPR is biased.

We begin by presenting the construction of DPR:

where ${\gamma_{i}}$ is the exposure probability of item i, which can be used as a proxy for the probability of exposure according to Equation 11; $\alpha$ is a hyper-parameter that reflects the length of the feedback loops suffered by the data, a higher value of $\alpha$ means longer loops. When $\alpha=0$, DPR is equivalent to BPR because there are no feedback loops on the data, and the presence of the exposure mechanism does not contribute to bias. The value of the $\alpha$ depends on the training data.

Unbiased pairwise methods commonly use inverse propensity scores of items as weights, such as UBPR and PDA, and lower accuracy in computing the propensity scores might lead to higher variance and sub-optimal recommendations, especially for the tail items with low exposure probability. To alleviate this problem, existing methods (Lee et al., 2021; Saito, 2020) apply non-negative estimators that clip large negative values for practical variance control but introduce additional bias.

Unlike existing methods, DPR uses the lower bound of the exposure mechanism as the weight, and once $\alpha$ is determined, the accuracy of the lower bound is guaranteed. Equation 16 shows that $\gamma_{i}$ is bounded at $[1,4]$ and $\frac{1}{\gamma_{i}}$ is therefore bounded at $[0.25,1]$. Due to the large number of items, the final value will tilt towards the left side of the range, so that the variance of DPR is acceptable even for the least popular items. In summary, the variance of DPR is much lower than the existing re-weighting methods without any additional operations.

Similar to BPR, DPR also encourages the score of positive pairs to be higher than negative ones. DPR can also be reformulated as:

The pairwise ranking method usually assumes that unobserved interactions are all negative, which is unrealistic because an item may not be exposed to the user, rather than the user does not prefer it, thus introducing false negative samples into the training process. The existence of false negative samples hurts the performance of the model because it prevents the model from fully tapping into user preferences. Previous studies (Ding et al., 2020a; Wang et al., 2021) have shown that both false negative samples and hard negative samples have large scores, but false negative samples have comparatively lower prediction variance. This means that false negative samples maintain a high prediction score in the training process. On the basis of these findings, we introduce a plugin named Universal Anti-False Negative (UFN) to alleviate the false negative problem:

where $\hat{s}_{u,j}$ is the predicted score of negative items, and $\beta$ is the hyper-parameter to control the strength, with a higher $\beta$ implying a stronger effort to alleviate false negative samples. $f(\hat{s}_{u,j})$ is a mapping function like $sigmod(\cdot)$, for the convenience of the proof, we use $tanh(\cdot)$. The DPR with UFN can be written as:

IPS and DPR both focus on exposure mechanisms to achieve unbiased recommendations, but with different views of the exposure mechanism and thus provide different solutions. Here we briefly explain the difference between DPR and IPS in modeling the exposure mechanism and the superiority of our novel design. UBPR is the application of IPS in pairwise ranking:

The problems of IPS can be summarized as follows:

• •

  IPS believes that the exposure mechanism is constant, but the fact is that it changes over loops.

• •

  IPS substitutes propensity score $\theta_{u,i}$ for the exposure mechanism, which is an oversimplification of the exposure mechanism. In addition, whether the re-weight loss is equal to the ideal expectation as the condition for unbiased or not will suffer from high variance if the propensity score is incorrect.

Our DPR alleviates the following problems:

• •

  We conduct an in-depth analysis of the exposure mechanism to find its key factors over time, which are used in the DPR to counteract the effects of the exposure mechanism.

• •

  DPR considers changes in the exposure mechanism over time and uses a parameter $\alpha$ to improve its generalization across different datasets.

• •

  Compared to using the ideal expectation to achieve unbias in IPS, DPR focuses on discovering the true performance of users and theoretical proof of its validity.

• •

  IPS-based methods need to compute the propensity score of items and thus suffer from high variance problems. DPR uses a lower bound on the exposure probability and thus has a stable performance compared to IPS-based methods.

We conduct experiments with all baselines and the proposed method DPR on simulated datasets, and the results are shown in Figure 5. As can be seen from Figure 5 (a), as the feedback loops progress, there are almost no new user-item interactions in baselines after 30 loops. This means that users are surrounded by popular items and have no opportunity to come into contact with unpopular items, a phenomenon often referred to as filter bubbles. The user-item interaction growth rate of baselines is significantly below the theoretical value, and the baseline with better performance on debiasing may even be lower than the BPR, largely due to they seek to eliminate the bias in the current timestamp and ignore the impact of the feedback loops. DPR achieves a significant rate increase compared to other baselines, the existence of a gap between the growth curve of DPR and the theoretical value is justified by the fact that items cannot be preferred by all users and thus we cannot achieve a perfect curve like the theoretical value. The performance of DPR on the growth rate of user-item interactions validates that DPR can help the model break feedback loops and delay the arrival of filter bubbles.

DPR helps the model escape the feedback loop and perform better than the baselines in terms of debiasing. As can be seen from Figure 5 (b) and (a), DPR has a significantly high value in the percentage of tail items, while the baseline reaches a lower value. As the model iterates, the baselines fail to capture the variability of exposure mechanisms, such that items with high popularity have high exposure probabilities and deteriorate as the iteration progresses, thus achieving lower performance in terms of tail items. Debiasing loss functions such as UBPR, MFDU and EBPR perform better than BPR, demonstrating that debiasing methods can alleviate the bias accumulation in feedback loops, but the performance is not measurable in the long term as they formulate the problem in a single time step.

In Table 4 and 5, we present the overall performance of applying baseline methods and our method over six datasets on two recommendation backbones. The baseline models include five pairwise ranking methods and two pointwise methods. For all datasets, the ranking performance of our DPR method outperforms the alternative baseline methods.

• •

  The performance of the IPS-based methods is not stable and even worse than BPR on some datasets. In a major part, because this type of approach requires relatively accurate inference prior, Ref-MF, for example, is an IPS-based method, and a better estimate of the property score might make it closer to the theoretical maximum. EPR adds additional user-item information to enhance explainability, but introducing additional information does not fully exploit the model and thus performs poorly on three common datasets. The IPS-based methods UPR and Rel-MF show worse performance in KuaiRec, as the IPS-based methods cannot well mitigate the effects of exposure mechanisms in lengthy feedback loops.

• •

  The baseline model fails to achieve the best performance in both debiasing and ranking performance. Baselines forcing the model to include de-biasing as an additional requirement in the optimization options come at the cost of model performance. Instead of adding a forcing requirement in the training process, DPR increases the adaptability of the model and shows the expected performance of the model as much as possible. Thus DPR has shown promising results on different backbones. Although the de-biasing effect of DPR on the Kuairec dataset is poor compared to RelMF, the accuracy of DPR is significantly higher than that of RelMF. The goal of DPR is to achieve the best possible debiasing performance while maintaining accuracy.

• •

  The proposed plug-in UFN further improves the performance of DPR. UFN reduces spurious information from false negative samples, which helps the model to avoid falling into sub-optimal traps. Further, DPR equipped with UFN can achieve better performance on most datasets, it is possible that unpopular items have a higher probability of being false negative samples, and UFN can help DPR to be more sensitive to unpopular items, thus achieving lower ARP scores.

In summary, DPR achieves better performance than other baseline methods in mitigating the negative effects of the feedback loops and unknown exposure mechanisms without sacrificing recommendation performance in most cases.

We conduct experiments on two datasets to show the effectiveness of the proposed plugin for model performance improvement. As shown in Table 6, deploying UFN on BPR alone can make a certain improvement in the model performance and decreases the weight of popular items in the final recommended items. This proves that UFN can mitigate the problem of false negative samples and improve the performance of the model. Moreover, the DPR also shows better performance than the baseline methods without UFN. This demonstrates that the two modules proposed in this paper are useful for model performance improvement and that the effects can be superimposed. UFN can further improve the performance of the loss function on both BPR and DPR, indicating that it can be widely used in the pairwise loss function to mitigate the interference caused by false negative samples.

We compare the performance of DPR with different levels of feedback loop parameter $\alpha$ on different datasets. As shown in Figure 6, DPR performs best with $\alpha=2$ on Coat, which is a dataset with short feedback loops. In addition, DPR performs best with $\alpha=4$ on Movielens 100K, which is a dataset that suffers from more feedback loops. An increase in the starting $\alpha$ improves the performance of the model on Yahoo! R3, but the model is not as sensitive to an increase in $\alpha$ as it is on Movielens 100K. We speculate that this may be due to the higher sparsity of the data. The different levels of optimal $\alpha$ in the three datasets demonstrate the ability of $\alpha$ to mitigate the different lengths of feedback loops suffered by datasets.

Moreover, the performance of DPR deteriorates when the level of alpha continues to increase. The results match our previous analysis of exposure mechanisms, where having the same exposure for all items is not necessarily beneficial for personalized recommendations. Compared to a fully fair exposure mechanism, a personalized user-oriented exposure mechanism is the ideal exposure mechanism. This means that full debiasing does not achieve the best recommendation performance.

To demonstrate the generalizability of DPR, we conduct comparison experiments on multiple types of backbone. Here, MF is representative of DNN-based models, LightGCN is representative of graph-based models, DIN(Zhou et al., 2018) is representative of self-attention-based models, and Wide&Deep(Cheng et al., 2016) is representative of MLP-based models. As shown in Table 7, DPR demonstrated superior performance with an average improvement of 5.79 % over BPR on four different backbones. The final results using DPR on MF can compete with and even outperform complex algorithms such as LightGCN using BPR, and complex models such as DIN can be further improved by using DPR, thus demonstrating that DPR can further develop the potential of the model itself and achieve better performance.

The outstanding performance of DPR on different classes of baselines highlights that DPR is an effective algorithm that can be widely used in mainstream recommendation models today. We believe that DPR has the potential to replace BPR as a commonly used loss function because of its ability to achieve high accuracy and powerful debiasing effects.

We deploy UFN on multiple baseline methods to demonstrate its universality. Figure 5 shows that the baseline methods all show a certain improvement after the deployment of UFN. Among them, pairwise-based methods have a more obvious improvement, while the improvement of the pointwise method is smaller. Probably because they have different objectives, pointwise methods place more importance on the accuracy of prediction scores than pairwise methods, and the deployment of UFN may affect the accuracy of model prediction scores. For pairwise-based methods, UFN can help to widen the score gap between positive and negative items, leading to better performance.

We compare the performance of DPR and baseline methods with different levels of parameter $\beta$. As shown in Figure 7, Initially, the model performs better with increasing $\beta$ since a larger beta means that the model focuses more on false negative samples, which improves the model performance. However, the high attention to false negative samples means that the effect of some strongly negative samples is equally compromised so that the model becomes worse when $\beta$ grows beyond a certain point and continues to increase. To our knowledge, items as false negative examples are more likely to be cold items, gaining low exposure due to their low initial popularity and deteriorating as the feedback loop proceeds, transforming into false negative examples because they have no opportunity to interact with users. Thus, the average popularity of the final recommendation keeps decreasing as the beta increases.

As previously introduced, DPR can help the model mitigate accumulated bias in feedback loops and model the true preferences of users. We investigate whether DPR has such advantages by visualizing the learned item embeddings using t-SNE (van der Maaten and Hinton, 2008).

Figure 8 shows the learned item embeddings of BPR and DPR on MF, where the items with different popularity are painted in different colors. We observe that the embeddings of BPR are significantly separated according to item popularity, with popular items distributed at both ends, meaning that popular items receive more attention from the model. Items with higher popularity are considered high-quality items by the model and thus receive higher exposure. At this point, the dominant factor in the recommendation of the model is not the properties of the item itself, but other external factors such as popularity. The accumulated bias in the feedback data is inherited by the trained model and will be amplified via feedback loops.

On the other hand, there is no significant difference between popular and unpopular items in the embeddings of DPR, which means that explicit properties in the dataset, such as item popularity, do not play an influential role in the modeling process, but the true identity of items does. DPR can help reduce the influence of irrelevant factors in the training process and establish more reliable user and item embedding, so as to achieve unbiased recommendations. Visualization of the learned item embeddings illustrates the advantages of DPR and makes reasonable interpretations.