A Data-Centric Multi-Objective Learning Framework for Responsible Recommendation Systems

However, there are two major limitations when applying FairBatch to multi-objective recommender systems. Firstly, FairBatch only considers two objectives, namely fairness and accuracy. Recommender systems require more realistic objectives to be taken into account, such as revenue, fairness, and unbiasedness. Secondly, FairBatch places excessive emphasis on the optimization of the fairness objective. It lacks a comprehensive discussion on balancing multiple objectives and controlling real-world constraints. For example, accuracy is a critical commercial metric, and it is often necessary to maintain similar performance levels when transitioning from single-objective systems to multi-objective-aware systems.

To overcome these limitations, we introduce MoRec, a trilevel data-centric framework designed to unify diverse objectives while offering controllablity. This framework comprises three interconnected levels that collaborate harmoniously, ensuring an efficient and effective process:

• •

  Outer-level - Objective Coordinator: It coordinates the relationship between goals by monitoring and adjusting the objectives to achieve a desired balance and performance.

• •

  Middle-level - Adaptive Data Sampler: Based on the coordinating signals from the outer-level, this component is in charge of updating sample weights and dynamically selecting training samples.

• •

  Inner-level - Standard Model Optimizer: This is a standard optimizer such as SGD, concentrating on the training of the backbone model with selected data samples.

To effectively capture a broad spectrum of objectives, we emphasize four core objectives to maximize: accuracy, revenue, fairness, and alignment. These categories are designed to encompass the majority of significant objectives for recommender systems. This approach enables the unification of various objectives optimization within a single data sampling framework, which can be implemented as a flexible plug-in data loader. In the following subsection, we will discuss each objective along with its respective update rules for the sampling weights $\bm{w}$. These rules are crucial for the data sampler’s optimal performance and overall effectiveness.

Accuracy. This is the fundamental objective, formulated as the negative accuracy loss as shown in Eq.(4). Intuitively, we set the sampling weights as a uniform distribution, i.e., $w_{i}^{acc}=\frac{1}{|D|}$, which is consistent with the standard accuracy-oriented model’s data loading process.

Revenue. Industrial recommendation systems typically need to consider the revenue generated for the platform. In this case, each item $e_{i}$ is associated with a profit value $r(e_{i})$, and the objective is to maximize the expected revenue of the recommended items:

Where $p(e_{i}|u_{i})$ represents the likelihood of user $u_{i}$ accepting item $e_{i}$, which can be approximated by the negative loss $-l_{acc}(\cdot)$¹¹1Explanation: In the case of the revenue objective, we regard each data sample as a positive item to be recommended. Thus, only the first part in Eq.(1) remains, and $l_{acc}(x_{i})=-logf\left(x_{i}|\bm{\theta}\right)$.. Consequently, the objective can be reformulated as maximizing the following goal:

Hence, for this objective, we set the weight of data sample $x_{i}=(u_{i},e_{i})$ as $w_{i}^{rev}\propto r(e_{i})$ and maintain fixed. Note that the data weights for both the accuracy metric and revenue metric are constants that do not require update rules.

Fairness. Fairness pertains to the performance disparity across groups, such as whether there is unfairness in the recommendation accuracy for various genders or item categories. There are multiple definitions of fairness measurement in recommendation systems, and in this paper, we adopt the Least Misery (Xiao et al., 2017) as the measurement. The least misery is denoted as the accuracy in the worst-performing group. The objective can be represented as maximizing the following goal:

where $\mathcal{O}_{acc}^{z}$ and $L^{z}$ represent the average accuracy measure and loss of group $z$, respectively. To maximize the objective, we formalize the problem as following bilevel optimization:

The update rule of $\bm{w}^{fai}$ is as follows:

where $i^{*}=\arg\max_{i}L^{z_{i}}(\bm{\theta}_{\bm{w}})$. Intuitively, the update rule elevates the sampling probability of the most disadvantaged group, i.e. group with the largest accuracy loss. Notably, $w^{fai}_{i}$ is initialized as proportional to number of samples in group $i$.

Alignment. Machine learning-based models tend to involve skew distributions in their predictive patterns. A typical phenomenon is bias amplification, in which some patterns are over-amplified in the learned model. For example, popularity amplification means that the model recommends too many popular items to users so that long-tail items have no chance to get exposed. To address the skew distribution issue, we propose the alignment objective, which aligns the model’s distribution with some pre-defined expectation distribution:

Where $P(\cdot,\bm{\theta})$ denotes the output distribution of the model and $Q(\cdot)$ represents the predefined distribution to be followed. Without loss of generality, we use item popularity for illustration and experimentation. For sample $x_{i}=(u_{i},e_{i})$, we set the exposure volume of item $e_{i}$ in the model’s recommendation list as $P(x_{i},\bm{\theta})$ and the frequency of $e_{i}$ in the training data as $Q(x_{i})$ (note that $Q(x_{i})$ can be freely designated according to real demands).

As for the update of $\bm{w}^{ali}$ to minimize $\mathcal{O}_{ali}$, the bilevel optimization is formalized as follow:

The update rule of $\bm{w}^{ali}$ is as follows:

which aims to adjust the weights of samples accordingly when they deviate from a desired level. And the initial value of $w^{ali}_{i}$ is set to $1/|D|$.

Coordinating multiple objectives presents a key challenge. (Lin et al., 2019a) generates a Pareto Frontier by setting different bounding values for the objectives, and then selects the most suitable solution from the Pareto Frontier according to real business demands. (Lin et al., 2019b; Mahapatra and Rajan, 2020) utilize preference vectors to generate a well-distributed set of Pareto solutions to choose from, representing different trade-offs among objectives. In our initial implementations, we prioritized both bounding values and preference vectors. However, we soon realized that they failed to demonstrate any advantage over the naive linear scalarization method (Experimental results can be found in Section 4.4). A possible reason is that the unified optimization through data sampling homogenizes the gradients of training supervision, facilitating the effectiveness of linear scalarization. Consequently, we ultimately selected linear scalarization for its simplicity and effectiveness.

Specifically, let the vector of weights associated with each objective be denoted as $\bm{\rho}=[\rho_{{\text{obj}_{1}}},\dots,\rho_{\text{obj}_{n}}]$, and let the losses of objectives be represented by $\bm{\ell}=[\ell_{\text{obj}_{1}},\dots,\ell_{\text{obj}_{n}}]$, with $n$ indicating the total number of objectives. The combined loss can then be calculated as $\ell_{final}=\bm{\rho}^{T}\bm{\ell}$. We can generate distributed solutions by varying the values of $\bm{\rho}$.

On the other hand, it is often important to ensure that crucial objectives, such as accuracy, do not significantly deteriorate when optimizing multiple objectives, thereby enabling a safe and smooth system upgrade. To achieve this, we draw inspiration from a method in the field of control systems – the PID (Proportional-Integral-Derivative) controller(Åström and Hägglund, 2006a). The PID is a widely-used feedback loop component in industrial control applications, designed to regulate a specific performance metric of the system to a predetermined value. This property aligns well with our objective of maintaining the model’s accuracy at a desirable level, and is thus adopted for our objectives coordinator. In the new setting, the coefficient of the accuracy objective $\rho_{\text{acc}}$ is no longer a preset, fixed value, but rather an adaptively changing one. To emphasize this, we rewrite $\ell_{final}$ as follows:

The key aspect here is determining how to adjust $\alpha_{\text{acc}}$. Inspired by ControlVAE (Shao et al., 2020), we remove the derivative term in PID, and regard $\ell_{\text{acc}}$ as the performance metric to be controlled. Loss value $\hat{\ell}$ represents the final stable value of loss in a model optimized for a single objective, it servers as the the preset value to control $\ell_{\text{acc}}$ in PI. Denote the loss value of the model at time $t$ as $l^{(t)}$, then the output of Objective Coordinator $\text{OC}\big{(}\ell^{(t)};\bm{\rho},\hat{\ell}\big{)}$ is:

where $err(t)=\hat{\ell}-\ell^{(t)}$ denotes the error at time $t$, i.e, the model’s accuracy performance gap on the current mini-batch of samples. $K_{p},K_{i}$ are the non-negative coefficients of the proportional and integral parts, respectively, and $\alpha_{\text{min}}$ is a constant reflecting the minimum value. These three variables are hyper-parameters.

The core idea of PI equation (Eq.(14)) is to apply a correction in the direction to reduce the error between the preset loss value and the current loss value. Specifically, the first term (proportional term, abbreviated as the P-term) controls the accuracy metric in the current mini-batch of samples. When $err(t)$ is negative and its absolute value is large, it indicates that the data samples are currently poorly fitted. Consequently, the P-term would be increased, promoting the model to strengthen the learning on $\ell_{\text{acc}}$. Conversely, a larger positive $err(t)$ indicates that the model may overfit those samples, prompting the P-term to decrease and reducing the weight of accuracy loss. This adjustment allows more room for the model to optimize other objectives. The second term (integral term, abbreviated as the I-term) manages accuracy from the perspective of cumulative errors, which essentially reflect the overall trend across the entire dataset rather than focusing solely on the current mini-batch’s samples. Note that $\sum_{t}err(t)$ represents the average error across all samples. If the value is positive, it suggests that the average loss is smaller than the preset value, which may potentially lead the model into an unexpected state, such as overfitting. In this case, the I-term would reduce the weight on the control metric $\ell_{\text{acc}}$. Conversely, if the model has not reached the preset state in terms of average loss, i.e., $\sum_{t}{err(t)}<0$, the I-term will be positive, assigning a stronger weight to the control metric $\ell_{\text{acc}}$. This approach stabilizes the model’s accuracy performance to the preset value throughout the training process, resulting in enhanced controllability.

The overall framework of MoRec is illustrated in Figure 1, while Figure 2(a) and Figure 2(b) present an enlarged view of the adaptive data sampler and the objective coordinator. Meanwhile, we offer an algorithmic pseudo-code in Algorithm 1. In summary, first, the objective coordinator is initialized with the preset objective priority $\bm{\rho}$ and $\hat{\ell}$, responsible for loss synthesis in line 6, denoted as the outer level. Then the sampling weights $\bm{w}$ are initialized with the training and validation set, which is mentioned in Section 3.2. Mini-batches are dynamically sampled according to weights $\bm{w}$ in line 3, and the weights $\bm{w}$ are optimized on the validation set $D_{v}$ after each training epoch in line 9, comprising the middle level. Finally, loss values corresponding to various objectives are calculated in line 5 and are synthesized with the output of the objective coordinator. The model’s parameters $\bm{\theta}$ are updated with the synthesized loss in line 8 as the inner level.

We first examine how effective is our proposed model for simultaneously optimizing four objectives. We assume that an effective method should jointly optimize multiple objectives without significantly compromising the accuracy performance. Consequently, we deem a solution to be invalid if it exhibits less than 97% accuracy compared to the base model. We generate at least six solutions for each baseline as well as our model. The most optimal solution is selected from all valid solutions based on the average relative improvement Imp. If all solutions are invalid, we opt for the one with the highest accuracy. Results are presented in Table 2 and Table 3.

All the baseline methods exhibit a relative improvement in certain objectives compared to the base model. When the base model is MF-BPR, all the baseline methods display significant average relative enhancements, despite some invalid solutions, underscoring the effectiveness of the scalarization method when the base model is not robust. However, for the more complex SASRec model, it becomes challenging to demonstrate strong overall enhancements, and some methods may even result in negative overall improvements.

On the other hand, MoRec can enhance the performance on target objectives with minimal accuracy loss, particularly on the industrial dataset. Specifically, MoRec achieves a maximum relative accuracy drop of 2.76% over three datasets, highlighting the effectiveness of our PID-based coordinator. Notably, none of the baseline models can be controlled to ensure a valid solution in different settings. Additionally, MoRec outperforms all the baseline methods in terms of average enhancements. While MoRec does not achieve state-of-the-art (SOTA) results for some individual objectives, its performance remains competitive, as it is close to the SOTA individual. Moreover, MoRec is the only model among its competitors that effectively performs on both types of base models, highlighting its superiority in terms of model-agnostic properties.

To validate the Pareto efficiency of MoRec, we generate five solutions for each method and draw the Pareto Frontier. For better visualization, we follow the two-objective setting by optimizing accuracy and revenue/fairness on two public datasets. The results are shown in Figure 3.

We observe that our MoRec exhibits great Pareto efficiency in all the four cases, especially in Electronics. While baseline methods fail to demonstrate Pareto efficiency in fairness, the reason may lie in that the loss for fairness is not designed for optimizing least misery directly and heterogeneous with accuracy loss. Furthermore, the solutions generated by MoRec have lower drop rates in accuracy and even obtain slight improvements, suggesting that PID-based objective coordinator’s capability in controlling the degradation of accuracy. As for baselines, we indeed have the similar observation with (Lin et al., 2019a) that solutions generated by MGDA are more centralized compared with PEMTL and EPO.

To investigate the importance of the proposed adaptive data sampler (DS) and PID-based objective coordinator (OC), we conduct ablation studies under the two-objective setting on the Electronics dataset, with the results presented in Figure 4. In MoRec w/o DS, we replace the data sampler with the extra loss function in Eq.(15) to model revenue. As for the objective coordinator, we replace our PID-based OC with various baseline methods, denoted by the pattern MoRec-OC-xx. Similar with the setting in Section 4.3, we generate five solutions for each variant for visualization.

First, when DS is replaced (MoRec w/o DS), the frontier in the revenue scenario remains competitive Pareto efficiency, which is not surpassed by MoRec. The reason lies in that our DS draws samples in proportion to their revenue, which is approximately equal to the weighted loss in $L^{rev}$. Nonetheless, due to the heterogeneity between accuracy and fairness loss functions, MoRec w/o DS exhibits weaker Pareto efficiency in the fairness scenario. In both scenarios, the accuracy performances are guaranteed due to the PID controller. Second, the replacement of our OC results in a failure to control accuracy performance, as most of the solutions from the variants fall outside the green area, as observed. Moreover, all the variants exhibit weaker Pareto efficiency compared to MoRec, especially in Figure 4(b), which underscores the indispensability of the OC component.

With a unified objective modeling and a PID-based objective coordinator, MoRec demonstrates a strong control effect on objective preference. To verify this, we visualize the PID’s controlling ability under the two-objective setting in Section4.3. We plot the accuracy loss, Hit, and rHit curves during the training stage in Figure6. We observe that the PID-based coordinator can effectively regulate the loss value to an expected value. The metrics on accuracy and loss exhibit an inverse relationship, meaning that the higher the expected loss value, the lower the corresponding accuracy - which is expected. In contrast, the metric on revenue increases as the expected value of accuracy loss rises, which is also expected.

Finally, we verify whether MoRec can flexibly control solutions’ generation towards specific objective preferences in four-objective setting. By setting various objective preference vectors $\bm{\rho}=[\rho_{fai},\rho_{ali},\rho_{rev}]$, we can obtain diverse solutions. Results are illustrated in Figure 6. Numbers in the figure represent the relative improvement compared to the base model. We observe a strong pattern in the relation between $\bm{\rho}$ and the resulting objectives. For instance, the preference coefficient $\bm{\rho}=[0.8,0.1,0.1]$ prioritizes the fairness metric, so its solution has a higher min-Hit than the others; preference coefficient $\bm{\rho}=[0.3,0.3,0.4]$ leads to a relatively more balanced solution among objectives.

Classical recommendation algorithms primarily focus on improving prediction accuracy by optimizing accuracy-oriented loss functions, such as Mean Square Error (MSE), Bayesian Personalized Ranking loss (BPR(Rendle et al., 2012)), Binary Cross Entropy loss (BCE), and log-softmax loss. Depending on data types and patterns in various application scenarios, numerous backbone models have been proposed to enhance the accuracy of recommender systems. For instance, matrix factorization (MF)(Johnson, 2014; Rendle et al., 2012; Mnih and Salakhutdinov, 2007; He et al., 2017) mainly focuses on latent user-item interactions for collaborative filtering learning; deep neural network-based approaches (Guo et al., 2017; Lian et al., 2018; Cheng et al., 2016; Wang et al., 2017; Song et al., 2019; He and Chua, 2017) are employed for deep feature interactions; and sequential recommendation techniques(Hidasi et al., 2015; Kang and McAuley, 2018; Tang and Wang, 2018) capture the order of user behavior history. In contrast to this line of research, the goal of this paper is not to propose a new backbone model, but rather to introduce a novel learning framework, enabling a given backbone model to be optimized for multiple diverse objectives simultaneously.

Multi-objective optimization (MOP) aims at finding a set of Pareto solutions with different trade-offs, with origins dating back to the early 1900s(Edgeworth, 1881). Multi-objective optimization methods can be broadly classified into two categories: multi-objective evolutionary algorithms (MOEAs) and scalarization. MOEAs(Deb et al., 2002; Schaffer, 2014; Fonseca and Fleming, 1993) employ various population-based heuristic search techniques to obtain solutions that are not dominated by each other, albeit at a high time cost. Scalarization methods (Chankong and Haimes, 2008; Yu, 1973) transform MOPs into single-objective problems (SOPs), with the weighted sum being the most commonly used technique. In order to obtain the Pareto efficiency, the multiple-gradient descent algorithm (Désidéri, 2012) combines scalarization with stochastic gradient descent (SGD), using the KKT condition to update the weights. MGDA(Sener and Koltun, 2018) is later improved to solve multi-task problems using the Frank-Wolfe algorithm. However, MGDA does not have a systematic way to incorporate various priorities. Recent works PEMTL(Lin et al., 2019b) and EPO(Ma et al., 2020) present methods for generating solutions tailored to specific preferences by adding extra constraints to the quadratic programming problem.

Multi-objective recommendation (MOR) aims to optimize multiple objectives simultaneously within a joint recommendation framework(Zheng and Wang, 2022; Jannach, 2022). MOEAs are designed with different heuristic search(Cai et al., 2020; Wang et al., 2016; Zuo et al., 2015; Cui et al., 2017; Pang et al., 2019; Ribeiro et al., 2012, 2014) or model hybridization(Cai et al., 2020; Ribeiro et al., 2014, 2012) strategies in balancing accuracy, diversity, long-tail performance, et al, which usually regard recommendation lists or well-trained models as solutions and do variation to generate new solutions. Especially, several scalarization methods are proposed in recent works. A two-step method(Lin et al., 2019a) built upon MGDA optimizes CTR and GMV by relaxing the quadratic programming problem as a non-negative least squares problem. And a reinforcement learning-based strategy(Xie et al., 2021) is proposed to solve the minimization optimization problem in MGDA, aiming to balance CTR and dwell time. However, existing methods only consider two or three homogeneous objectives without priorities and focus on modeling different objectives with various well-designed loss functions.