StableDR: Stabilized Doubly Robust Learning for Recommendation on Data Missing Not at Random

In RS, due to the fact that users are more inclined to evaluate the preferred items, the collected ratings are always missing not at random (MNAR). We formulate the data MNAR problem using the widely adopted potential outcome framework (Neyman, 1990; Imbens & Rubin, 2015). Let $\mathcal{U}=\{1,2,...,U\}$, $\mathcal{I}=\{1,2,...,I\}$ and $\mathcal{D}=\mathcal{U}\times\mathcal{I}$ be the index sets of users, items, all user-item pairs. For each $(u,i)\in\mathcal{D}$, we have a treatment $o_{u,i}\in\{0,1\}$, a feature vector $x_{u,i}$, and an observed rating $r_{u,i}$, where $o_{u,i}=1$ if user $u$ rated the item $i$ in the logging data, $o_{u,i}=0$ if the rating is missing. Let $r_{u,i}(1)$ is defined as the be the rating that would be observed if item $i$ had been rated by user $u$, which is observable only for $\mathcal{O}=\{(u,i)\mid(u,i)\in\mathcal{D},o_{u,i}=1\}$. Many tasks in RS can be formulated by predicting the potential outcome $r_{u,i}(1)$ using feature $x_{u,i}$ for each $(u,i)$.

Let $\hat{r}_{u,i}(1)=f(x_{u,i};\phi)$ be a prediction model with parameters $\phi$. If all the potential outcomes $\{r_{u,i}(1):(u,i)\in\mathcal{D}\}$ were observed, the ideal loss function for solving parameters $\phi$ is given as

where $e_{u,i}$ is the prediction error, such as the squared loss $e_{u,i}=(\hat{r}_{u,i}(1)-r_{u,i}(1))^{2}$. $\mathcal{L}_{ideal}(\phi)$ can be regarded as a benchmark of unbiased loss function, even though it is infeasible due to the missingness of $\{r_{u,i}(1):o_{u,i}=0\}$. As such, a variety of methods are developed through approximating $\mathcal{L}_{ideal}(\phi)$ to address the selection bias, in which the propensity-based estimators show the relatively superior performance (Schnabel et al., 2016; Wang et al., 2019), and the IPS and DR estimators are

where $\hat{p}_{u,i}$ is an estimate of propensity score $p_{u,i}:=\mathbb{P}(o_{u,i}=1|x_{u,i})$, $\hat{e}_{u,i}$ is an estimate of $e_{u,i}$.

Debiased learning in recommendation. The data collected in RS suffers from various biases (Chen et al., 2020; Wu et al., 2022b), which are entangled with the true preferences of users and pose a great challenge to unbiased learning. There is increasing interest in coping with different biases in recent years (Zhang et al., 2021; Ai et al., 2018; Liu et al., 2016; 2021). Schnabel et al. (2016) proposed using inverse propensity score (IPS) and self-normalized IPS (SNIPS) methods to address the selection bias on data missing not at random, Saito (2019) and Saito et al. (2020) extended them to implicit feedback data. Marlin et al. (2007) and Steck (2013) derived an error imputation-based (EIB) unbiased learning method. These three approaches adopt two-phase learning (Wang et al., 2021), which first learns a propensity/imputation model and then applies it to construct an unbiased estimator of the ideal loss to train the recommendation model. A doubly robust joint learning (DR-JL) method (Wang et al., 2019) was proposed by combining the IPS and EIB approaches. Subsequently, strands of enhanced joint learning methods were developed, including MRDR (Guo et al., 2021), Multi-task DR (Zhang et al., 2020), DR-MSE (Dai et al., 2022), BRD-DR (Ding et al., 2022), TDR Li et al. (2023b), uniform data-aware methods (Bonner & Vasile, 2018; Liu et al., 2020; Chen et al., 2021; Wang et al., 2021; Li et al., 2023c) that aimed to seek better recommendation strategies by leveraging a small uniform dataset, and multiple robust method (Li et al., 2023a) that specifies multiple propensity and imputation models and achieves unbiased learning if any of the propensity models, imputation models, or even a linear combination of these models can accurately estimate the true propensities or prediction errors. Chen et al. (2020) reviewed various biases in RS and discussed the recent progress on debiasing tasks. Wu et al. (2022b) established the connections between the biases in causal inference and the biases, thereby presenting the formal causal definitions for RS.

Stabilized causal effect estimation. The proposed method builds on the stabilized average causal effect estimation approaches in causal inference. Molenberghs et al. (2015) summarized the limitations of doubly robust methods, including unstable to small propensities (Kang & Schafer, 2007; Wu et al., 2022a), unboundedness (van der Laan & Rose, 2011), and large variance (Tan, 2007). These issues inspired a series of stabilized causal effect estimation methods in statistics (Kang & Schafer, 2007; Bang & Robins, 2005; van der Laan & Rose, 2011; Molenberghs et al., 2015). Unlike previous works that focused only on achieving learning with unbiasedness in RS, this paper provides a new perspective to develop doubly robust estimators with much more stable statistical properties.

Even though DR estimator has double robustness property, its performance could be significantly improved if the following three stabilization aspects can be enhanced.

More stable to small propensities. As shown in Schnabel et al. (2016), Wang et al. (2019) and Guo et al. (2021), if there exist some extremely small estimated propensity scores, the IPS/DR estimator and its bias, variance, and tail bound are unbounded, deteriorating the prediction accuracy. What’s more, such problems are widespread in practice, given the fact that there are many long-tailed users and items in RS, resulting in the presence of extreme propensities.

More stable through weakening extrapolation. DR relies more on extrapolation because the imputation model in DR is learned from the exposed events $\mathcal{O}$ and extrapolated to the unexposed events. If the distributional disparity of $e_{u,i}$ on $o_{u,i}=0$ and $o_{u,i}=1$ is large, the imputed errors are likely to be inaccurate on the unexposed events and incur bias of DR. Therefore, it is beneficial to reduce bias if we can develop an enhanced DR method with weaker reliance on extrapolation.

More stable training process of updating a prediction model. In general, alternating training between models results in better performance. From Figure 1, Wang et al. (2019) proposes joint learning for DR, alternatively updating the error imputation and prediction models with given estimated propensities. Double learning (Guo et al., 2021) further incorporates parameter sharing between the imputation and prediction models. Bi-level optimization (Wang et al., 2021; Chen et al., 2021) can be viewed as alternately updating the prediction model and the other parameters used to estimate the loss. To the best of our knowledge, this is the first paper that proposes a algorithm to update the three models (i.e., error imputation model, propensity model, and prediction model) separately using different optimizers, which may resulting in more stable and accurate rating predictions.

We propose a stabilized doubly robust (SDR) estimator that has a weaker dependence on extrapolation and is robust to small propensities. The SDR estimator consists of the following three steps.

Step 1 (Initialize imputed errors). Pre-train imputation model $\hat{e}_{u,i}$, let $\hat{\mathcal{E}}\triangleq{|\mathcal{D}|}^{-1}\sum_{(u,i)\in\mathcal{D}}\hat{e}_{u,i}$.

Step 2 (Learn constrained propensities). Learn a propensity model $\hat{p}_{u,i}$ satisfying

Step 3 (SDR estimator). The SDR estimator is given as

where $w_{u,i}=\frac{o_{u,i}}{\hat{p}_{u,i}}\big{/}\sum_{(u,i)\in\mathcal{D}}\frac{o_{u,i}}{\hat{p}_{u,i}}$. It can be seen that SDR estimator has the same form as SNIPS estimator, but the propensities are learned differently. In SDR, the estimation of propensity model relies on the imputed errors, whereas not in SNIPS.

We provide an intuitive way to illustrate the rationale of SDR. On the one hand, if the propensities can be accurately estimated (i.e., $\hat{p}_{u,i}=p_{u,i}$) by using a common model (e.g., logistic regression) without imposing constraint (1). Then the expectation of the left hand side of constraint (1) becomes

which indicates the constraint (1) always holds as the sample size goes to infinity by the strong law of large numbers²²2This is the reason why we adopt the notation of ”asymptotically unbiased”., irrespective of the accuracy of the imputed errors $\hat{e}_{u,i}$. This implies that the constraint (1) imposes almost no restriction on the estimation of propensities. In this case, the SDR estimator is almost equivalent to the original SNIPS estimator. On the other hand, if the propensities cannot be accurately estimated by using a common model, but the imputed errors are accurate (i.e., $\hat{e}_{u,i}=e_{u,i}$). In this case, $\mathcal{\hat{E}}$ is an unbiased estimator. Specifically, $\mathcal{E}_{SDR}$ satisfies

Combining the constraint (1) and equation (2) gives

where the first term equals to 0 if $\hat{e}_{u,i}=e_{u,i}$, it implies that $\mathcal{E}_{SDR}=\hat{\mathcal{E}}$, then the unbiasedness of $\mathcal{E}_{SDR}$ follows immediately from the unbiasedness of $\hat{\mathcal{E}}$.

In addition, Step 3 is designed for two main reasons to achieve stability. First, $\mathcal{E}_{SDR}$ is more robust to extrapolation compared with DR. This is because the propensities are learned from the entire data and thus have less requirement on extrapolation. Second, $\mathcal{E}_{SDR}$ is more stable to small propensities, since the self-normalization imposes the weight $w_{u,i}$ to fall on the interval [0,1].

In summary, forcing the propensities to satisfy the constraint (1) makes the SDR estimator not only doubly robust, but also captures the advantages of both SNIPS and DR estimators. The design of SDR enables the constrained propensities to adaptively find the direction of debiasing if either the learned propensities without imposing constraint (1) or the imputed errors are accurate.

Through theoretical analysis, we note that previous debiasing estimators such as IPS (Schnabel et al., 2016) and DR-based methods (Wang et al., 2019; Guo et al., 2021) tend to have infinite biases, variances, tail bound, and corresponding generalization error bounds, in the presence of extremely small estimated propensities. Remarkably, the proposed SDR estimator doesn’t suffer from such problems and is stable to arbitrarily small propensities, as shown in the following Theorems (see Appendixes A.2, A.3 and A.4 for proofs).

Theorem 2 shows the bias of the SDR estimator consisting of a dominant term given by the difference between $\delta_{u,i}$ and its weighted average, and a negligible term of order $O(|\mathcal{D}|^{-1})$. The fact that the $\delta_{u,i}$ and its convex combinations are bounded, shows that the bias is bounded for arbitrarily small $\hat{p}_{u,i}$. Compared to the $\operatorname{Bias}\left(\mathcal{E}_{IPS}\right)={|\mathcal{D}|}^{-1}|\sum_{u,i\in\mathcal{D}}{(\hat{p}_{u,i}-p_{u,i})e_{u,i}}/{\hat{p}_{u,i}}|$ and $\operatorname{Bias}\left(\mathcal{E}_{DR}\right)={|\mathcal{D}|}^{-1}|\sum_{u,i\in\mathcal{D}}{(\hat{p}_{u,i}-p_{u,i})\delta_{u,i}}/{\hat{p}_{u,i}}|$, it indicates that IPS and DR will have extremely large bias when there exists an extremely small $\hat{p}_{u,i}$.

Note that $\sum_{\mathcal{D}\setminus(u,i)}p_{u,i}/\hat{p}_{u,i}=O(|\mathcal{D}|)$ and $\epsilon^{\prime}=O(|\mathcal{D}|^{1/2})$ in Theorem 4, it follows that the tail bound of the SDR estimator converges to 0 for large samples. In addition, the tail bound is bounded for arbitrarily small $\hat{p}_{u,i}$. Compared to the tail bound of IPS and DR, with probability $1-\eta$, we have

which are both unbounded when $\hat{p}_{u,i}\rightarrow 0$. For SDR in the prediction model training phase, the boundedness of the generalization error bound (see Theorem 5 in Appendix A.5) follows immediately from the boundedness of the bias and tail bound. The above analysis demonstrates that SDR can comprehensively mitigate the negative effects caused by extreme propensities and results in more stable predictions. Theorems 2-5 are stated under the constraint $(\ref{step2})$. If we estimate the propensities with constraint (1), but finally constraint (1) somehow doesn’t hold exactly, the associated bias, variance, and generalization error bound of SDR are presented in Appendix B.

Baselines. In our experiments, we take Matrix Factorization (MF) (Koren et al., 2009), Sparse LInear Method (SLIM) (Ning & Karypis, 2011), and Neural Collaborative Filtering (NCF) (He et al., 2017) as the base model respectively, and compare against the proposed methods with the following baselines: Base Model, IPS (Saito et al., 2020; Schnabel et al., 2016), SNIPS (Swaminathan & Joachims, 2015), IPS-AT (Saito, 2020), CVIB (Wang et al., 2020b), DR (Saito, 2020), DR-JL (Wang et al., 2019), and MRDR-JL (Guo et al., 2021). In addition, Naive Bayes with Laplace smoothing and logistic regression are used to establish the propensity model respectively.

Experimental protocols and details. The following four metrics are used simultaneously in the evaluation of debiasing performance: MSE, AUC, NDCG@5, and NDCG@10. All the experiments are implemented on PyTorch with Adam as the optimizer⁵⁵5For all experiments, we use NVIDIA GeForce RTX 3090 as the computing resource.. We tune the learning rate in $\{0.005,0.01,0.05,0.1\}$, weight decay in $\{1e-6,5e-6,\dots,5e-3,1e-2\}$, constrain parameter eta in $\{50,100,150,200\}$ for Coat and $\{500,1000,1500,2000\}$ for Yahoo! R3, and batch size in $\{128,256,512,1024,2048\}$ for Coat and $\{1024,2048,4096,8192,16384\}$ for Yahoo! R3. In addition, for the Laplacian smooth parameter in Naive Bayes model, the initial value is set to 0 and the learning rate is tuned in $\{5,10,15,20\}$ for Coat and in $\{50,100,150,200\}$ for Yahoo! R3.

Table 5.1 summarizes the performance of the proposed Stable-DR and Stable-MRDR methods compared with previous methods. First, the causally-inspired methods perform better than the base model, verifying the necessity of handling the selection bias in rating prediction. For previous methods, SNIPS, CVIB and DR demonstrate competitive performance. Second, the proposed Stable-DR and Stable-MRDR have the best performance in all four metrics. On one hand, our methods outperform SNIPS, attributed to the inclusion of the propensity model in the training process, as well as the boundedness and double robustness of SDR. On the other hand, our methods outperform DR-JL and MRDR-JL, attributed to the stabilization constraint introduced in the training of the propensity model. This further demonstrates the benefit of cycle learning, in which the propensity model is acted as the mediation between the imputation and prediction model during the training process, rather than updating the prediction model from the imputation model directly.

The debiasing performance under different stabilization constraint strength and propensity models is shown in Figure 2. First, the proposed Stable-DR and Stable-MRDR outperform DR-JL and MRDR-JL, when either Naive Bayes with Laplace smoothing or logistic regression is used as propensity models. It indicates that our methods have better debiasing ability in both the feature containing and collaborative filtering scenarios. Second, when the strength of the stabilization constraint is zero, our method performs similarly to SNIPS and slightly worse than the DR-JL and MRDR-JL, which indicates that simply using cross-entropy loss to update the propensity model is not effective in improving the model performance. However, as the strength of the stabilization constraint increases, Stable-DR and Stable-MRDR using cycle learning have a stable and significant improvement compared to DR-JL and MRDR-JL. Our methods achieve the optimal performance at the appropriate constraint strength, which can be interpreted as simultaneous consideration of accuracy and stability to ensure boundedness and double robustness of SDR.