Comparative Explanations of Recommendations

Modern recommender systems fundamentally shape our everyday life (Koren et al., 2009; He et al., 2017; Sarwar et al., 2001; Rendle, 2010; Aggarwal et al., 2016; Wu et al., 2020; Cai et al., 2021). As a result, how to explain the algorithm-made recommendations becomes crucial in building users’ trust in the systems (Zhang and Chen, 2020). Previous research shows that explanations, which illustrate how the recommendations are generated (Ribeiro et al., 2016; Lundberg and Lee, 2017) or why the users should pay attention to the recommendations (Wang et al., 2018b; Sun et al., 2020; Yang et al., 2021), can notably strengthen user engagement with the system and better assist them in making informed decisions (Bilgic and Mooney, 2005; Herlocker et al., 2000; Sinha and Swearingen, 2002).

When being presented with a list of recommendations, typically sorted in a descending order, a user needs to make a choice. In other words, the provided explanations should help users compare the recommended items. Figure 1 illustrates the necessity of comparative explanations. By reading the explanations for the hotels recommended in the figure, one can easily tell why the system ranks them in such an order. But if the system provided the explanation in the dashed box for Hotel C, it would confuse the users about the ranking, e.g., Hotel C becomes arguably comparable to top ranked Hotel A; but it was ranked at the bottom of the list. This unfortunately hurts users’ trust in all three recommended hotels.

Existing explainable recommendation solutions are not optimized to help users make such comparative decisions for two major reasons. First, the explanation of a recommended item is often independently generated without considering other items in the recommendation list. As shown in Figure 1, one single low-quality generation (the one in the dashed box) might hamper a user’s understanding over the entire list of recommendations. Second, the popularly adopted neural text generation techniques are known to be flawed of its generic content output (Holtzman et al., 2019; Welleck et al., 2019). Particularly, techniques like maximum likelihood training and sequence greedy decoding lead to short and repetitive sentences composed of globally frequent words (Weston et al., 2018). Such generic content cannot fulfill the need to differentiate the recommended items. Consider the example shown in Figure 1 again, “the hotel is good” is a very generic explanation and thus not informative. Its vague description (e.g., the word “good”) and lacks of specificity (e.g., the word “hotel”) make it applicable to many hotels, such that users can hardly tell the relative comparison of the recommended items from such explanations.

In this work, we tackle the problem of comparative explanation generation to help users understand the comparisons between the recommended items. We focus on explaining how one item is compared with another; then by using a commonly shared set of items as references (e.g., items the user has reviewed before), the comparisons among the recommended items emerge. For example, if the explanations suggest item A is better than item B and item C is worse than item B, the comparison between A and C is apparent after reading the associated explanations. Our solution is designed to generically work on top of other existing recommender systems. We do not have any assumptions about how the recommendation algorithm ranks items (e.g., collaborative filtering (Sarwar et al., 2001) or content-based (Balabanović and Shoham, 1997)), but only require it to provide a ranking score for each item to our model (i.e., ordinal ranking) which reflects a user’s preference over the recommended item. This makes our solution readily applicable to explain plenty of effective recommendation algorithms deployed in practice.

We design an extract-and-refine text generation architecture (Weston et al., 2018; Guu et al., 2018) to explain the ranked items one at a time to the user, conditioned on their recommendation scores and associated reviews. We refer to the item to be explained in the ranked list as the target item, and user we are explaining to as the target user. First, the model extracts one sentence from the existing review sentences about the target item as a prototype, with a goal to maximize the likelihood of fitting the comparisons against the reviews written by the target user for other reference items. Then we refine the extracted prototype through a generative model to further polish the content for the target user. In this two stage procedure, the extraction module exploits the content already provided about the target item to ensure the relevance of generated explanations (e.g., avoid mentioning features that do not exist in the target item); and the refinement module further improve the explanation (e.g., informativeness and diversity of content) beyond the limitation of the existing content. We design a new explanation quality metric based on BLEU to guide the end-to-end training of the two modules, with a particular focus to penalize short and generic content in generated explanations.

We compared the proposed solution with a rich set of state-of-the-art baselines for explanation generation on two large-scale recommendation datasets. Besides, we also conducted extensive user studies to have the generated explanations evaluated by real users. Positive results obtained on both offline and online experiments suggested the effectiveness of comparative explanations in assisting users to better understand the recommendations and make more informed choices.

Most explainable recommendation solutions exploit user reviews as the source of training data. They either directly extract from reviews or synthesize content to mimic the reviews. Extraction-based solutions directly select representative text snippets from the target item’s existing reviews. For example, NARRE (Chen et al., 2018) selects the most attentive reviews as the explanation, based on the attention that is originally learned to enrich the user and item representations for recommendation. CARP (Li et al., 2019) uses the capsule network for the same purpose. Wang et al. (2018a) adopt reinforcement learning to extract the most relevant review text that matches a given recommender system’s rating prediction. Xian et al. (2021) extract attributes from reviews to explain a set of items based on users’ preferences. However, as such solutions are restricted to an item’s existing reviews, their effectiveness is subject to the availability and quality of existing content. For items with limited exposure, e.g., a new item, these solutions can hardly provide any informative explanations.

Generation-based solutions synthesize textual explanations that are not limited to existing reviews. One branch focuses on predicting important aspects of an item (such as item features) from its associated reviews as explanations (Wang et al., 2018b; Tao et al., 2019; He et al., 2015; Ai et al., 2018; Cai et al., 2017). For instance, MTER (Wang et al., 2018b) and FacT (Tao et al., 2019) predict item features that are most important for a user to justify the recommendation. They rely on predefined text templates to deliver the predicted features. The other branch applies neural text generation techniques to synthesize natural language sentences. In particular, NRT (Li et al., 2017) models item recommendation and explanation generation in a shared user and item embedding space. It uses its predicted recommendation ratings as part of the initial state for explanation generation. MRG (Truong and Lauw, 2019) integrates multiple modalities from user reviews, including ratings, text, and associated images, for multi-task explanation modeling.

Our work is closely related to two recent studies, DualPC (Sun et al., 2020) and SAER (Yang et al., 2021), which focus on strengthening the relation between recommendations and explanations. Specifically, DualPC introduces duality regularization based on the joint probability of explanations and recommendations to improve the correlation between recommendations and generated explanations. SAER introduces the idea of sentiment alignment in explanation generation. However, both of them operate in a pointwise fashion, i.e., independent explanation generation across items. Our solution focuses on explaining the comparisons between items. We should also emphasize our solution is to explain the comparison among a set of recommended items, rather than to find comparable items (McAuley et al., 2015; Chen et al., 2020).

There are also solutions exploiting other types of information for explainable recommendation, such as item-item relation (Chen et al., 2021), knowledge graph (Xian et al., 2019) and social network (Ji and Shen, 2016). But they are clearly beyond the scope of this work.

Item recommendation in essence is a ranking problem: estimate a recommendation score for each item under a given user and rank the items accordingly, such that the utility of the recommendations can be maximized (Rendle et al., 2012; Karatzoglou et al., 2013). Instead of explaining how the recommendation scores are obtained, our work emphasizes on explaining how the comparisons between the ranked items are derived.

To learn the explanation model, we assume an existing corpus of item reviews from the intended application domain (e.g., hotel reviews). Each review is uniquely associated with a user $u$ and an item $c$, and a user-provided rating ${r}_{c}^{u}$ suggesting his/her opinion towards the item. We group the reviews associated with user $u$ to construct his/her profile $\Omega_{u}=\{(x^{u}_{1},r^{u}_{1}),(x^{u}_{2},r^{u}_{2}),...,(x^{u}_{m},r^{u}_{m})\}$, where $x^{u}_{i}$ is the $i$-th review sentence extracted from user $u$’s reviews and $r^{u}_{i}$ is the corresponding opinion rating. $r^{u}_{i}$ can be easily obtained when the detailed aspect ratings are available (Wang et al., 2010); otherwise off-the-shelf sentiment analysis methods can be used for the purpose (interested users can refer to (Wang et al., 2018b; Zhang et al., 2014) for more details). As regards cold-start for users without reviews, generic profiles can be used instead which sample reviews from similar users clustered by other non-review-related features, such as rating history. We create the item profile as $\Psi_{c}=\{x^{c}_{1},x^{c}_{2},...,x^{c}_{n}\}$, where $x^{c}_{j}$ is the $j$-th review sentence extracted from item $c$’s existing reviews. Unlike the user profile, the item profile does not include ratings. This is because the ratings from different users are not directly comparable, as individuals understand or use the numerical ratings differently. Our solution is agnostic to the number of entries in user profile $\Omega_{u}$ and item profile $\Psi_{c}$ in each user and item.

We impose a generative process for a tuple $(x,r^{u}_{c})$ from user $u$ about item $c$ conditioned on $\Psi_{c}$ and $\Omega_{u}$. We assume when user $u$ is reviewing item $c$, he/she will first select an existing sentence from $\Psi_{c}$ that is mostly related to the aspect he/she wants to cover about the item. Intuitively, this can be understood as the user will first browse existing reviews of the item to understand how the other users evaluated this item. Then he/she will rewrite this selected sentence to reflect his/her intended opinion and own writing style. This can be considered as a set to sequence generation problem. For our purpose of explanation generation, we only concern the generation of opinionated text $x$. Hence, we take opinion rating $r^{u}_{c}$ as input, which leads us to the following formulation,

where $P_{ext}(x^{c}_{j}|r^{u}_{c},\Omega_{u})$ specifies the probability that $x^{c}_{j}$ from item profile $\Psi_{c}$ will be selected by user $u$, and $P_{ref}(x|x^{c}_{j},r^{u}_{c},\Omega_{u})$ specifies the probability that user $u$ will rewrite $x^{c}_{j}$ into $x$. We name the resulting model Comparative Explainer, or CompExp in short.

In Eq (1), $P_{ext}(x^{c}_{j}|r^{u}_{c},\Omega_{u})$ is essential to capture the comparative textual patterns embedded in user $u$’s historical opinionated text content. To understand this, we can simply rewrite its condition part: define $\Delta r^{u}_{i}=r^{u}_{c}-r^{u}_{i}$, we have $(r^{u}_{c},\Omega_{u})=\{(x^{u}_{i},\Delta r^{u}_{i})\}^{m}_{i=1}$; hence, $P_{ext}(x^{c}_{j}|r^{u}_{c},\Omega_{u})$ characterizes whether the sentence $x^{c}_{j}$ about item $c$ is qualified to characterize the desired opinion difference conditioned on user $u$’s historical content $\Omega_{u}$ and target rating $r^{u}_{c}$. For example, a negative $\Delta r^{u}_{i}$ suggests the opinion conveyed in $x^{c}_{j}$ is expected to be less positive than that in $x^{u}_{i}$. On a similar note, $P_{ref}(x|x^{c}_{j},r^{u}_{c},\Omega_{u})$ quantifies if $x$ is a good rewriting of $x^{c}_{j}$ to satisfy the desired opinion rating $r^{u}_{c}$ for item $c$ by user $u$.

One can parameterize $P_{ext}(x^{c}_{j}|r^{u}_{c},\Omega_{u})$ and $P_{ref}(x|x^{c}_{j},r^{u}_{c},\Omega_{u})$ and estimate the corresponding parameters based on the maximum likelihood principle over observations in $\Omega_{u}$. However, data likelihood alone is insufficient to generate high-quality explanations, as we should also emphasize on fluency, brevity, and diversity of the generated explanations. To realize this generalized objective, assume a metric $\pi(x|u,c)$ that measures the quality of generated explanation $x$ for user $u$ about item $c$, the training objective of CompExp is set to maximize the expected quality of its generated explanations under $\pi(x|u,c)$,

In this work, we present a customized BLUE score specifically for the comparative explanation generation problem to penalize short and generic content.

Next, we dive into the detailed design of CompExp in Section 3.1, then present our metric $\pi(x|u,c)$ for parameter estimation in Section 3.2 and 3.3, and finally illustrate how to estimate each component in CompExp end-to-end in Section 3.4.

We demonstrate empirically that CompExp can generate improved explanations compared to state-of-the-art explainable recommendation algorithms. We conduct experiments on two different recommendation scenarios: RateBeer reviews with single-ratings (McAuley et al., 2012) and TripAdvisor reviews with multi-aspect ratings (Wang et al., 2010).

We have three research questions to answer in user study: 1) does users’ judgement toward explanation quality aligns more with IDF-BLEU than BLEU; 2) do users find our comparative explanations more helpful than the baselines’; and 3) can users better perceive the comparative ranking from our explanations than the baselines’. To answer these three research questions, we design two user study tasks based on RateBeer dataset using Amazon Mechanical Turk.

The first task studies the first two research questions together. Specifically, we shuffle explanations from different models about the same recommended item and ask the participants to compare them, and then select the most helpful ones. To help participants evaluate the explanation quality, we include the original user review as the item description, towards which they can judge if the explanation are accurate or informative. For each recommended item, we ask participants to answer the following question after reading its description and candidate explanations:

• “Which of the following explanations best describe the characteristics of the given beer and help you the most to understand why you should pay attention to the recommendation?”

In this experiment, we collected 660 user responses.

The results are presented in Table 3 and 4. In Table 3, we used Cohen’s kappa coefficient to compare IDF-BLEU and BLEU’s agreement with users’ responses. For each test case, we pair explanations that the participants chose as helpful with the rest to form a set of explanation pairs. Then we use IDF-BLEU-{1,2,4} and BLEU-{1,2,4} to identify the helpful one in each pair. The kappa coefficient shows that IDF-BLEU aligns significantly better with users’ judgment in all three subcategories under paired t-test. Table 4 shows the helpfulness vote on each model and the paired t-test results of CompExp against other baselines. The helpfulness vote on CompExp is significantly higher than others, which suggests strong user preference over its generated explanations.

The second task addresses the last research question, i.e., if a user is able to perceive the ranking of recommended items from the explanations. In this task, we randomly paired items of different ratings and asked participants to identify which item is better by reading the provided explanations. We then evaluated the agreement rate between participants’ choices and the actual ranking. In particular, given the explanations of a model, the participants were required to answer the following question:

• “After reading the explanations for recommended items, which item would you like to choose? You are expected to judge the quality of the items based on the provided explanations.”

We chose SAER and NRT as baselines. Besides, we also include the ground-truth sentences from the actual user reviews as a reference. We collected 200 responses for each model.

Table 5 reports the agreement rates between the actual ranking and the ranking perceived by the participants. CompExp’s agreement rate is slightly higher than NRT and SAER, but it is far below the Ground-Truth. The Ground-Truth’s high agreement rate quantitatively confirms that the original user provided review sentences are highly comparative. This observation supports our choice of training the comparative explanation generation from paired user review sentences. And it also suggests there is still a performance gap in comparativeness for learning-based solutions to bridge. And an improved objective for optimization, e.g., include quantified pairwise comparativeness, might be a promising direction.

In this paper, we studied the problem of comparative explanation generation in explainable recommendation. The objective of our generated explanations is to help users understand the comparative item rankings provided in a recommender system. We develop a neural extract-and-refine architecture to generate such comparative explanations, with customized metrics to penalize generic and useless content in the generated explanations. Both offline evaluations and user studies demonstrated the effectiveness of our solution.

This work starts a bright new direction in explainable recommendation. Our current solution only focuses on explanation generation, by assuming a perfect recommendation algorithm (i.e., we directly used the ground-truth opinion ratings in our experiments). It is important to improve our model by co-design with a real recommendation algorithm, whose recommendation score is expected to be noise and erroneous. In addition, we still heavily depend on existing review content to guide explanation generation. It will be more meaningful to introduce actual user feedback in this process, i.e., interactive optimization of explanation generation.