Counterfactual Explainable Recommendation

Explainable recommendation is a broad research area with many different types of models, and it is difficult to cover all of the works on this direction. Since our work is more closely related with aspect-aware explainable recommendation, we mainly focus on this sub-area in the section. A more complete review of explainable recommendation can be seen in (Zhang and Chen, 2020; Gedikli et al., 2014; Tintarev and Masthoff, 2015).

Explainability of recommender systems is important because it improves the transparency, user satisfaction and trust over the recommendation system (Zhang and Chen, 2020). One representative way to generate user-friendly explanations is by modeling aspects in the recommended items. For instance, Zhang et al. (Zhang et al., 2014a) introduced an Explicit Factor Model (EFM) for explainable recommendation. It first extracts the item aspects and the user opinions on these aspects from user reviews. Then, it trains a matrix factorization-based recommendation model to generate aspect-aware explanations by aligning the latent factors with the item aspects. Chen et al. (Chen et al., 2016) and Wang et al. (Wang et al., 2018c) advanced from matrix factorization to tensor factorization models for explainable recommendation. He et al. (He et al., 2015) proposed a tripartite graph model to improve the interactivity of aspect-aware recommendation models. Gao et al. (Gao et al., 2019) proposed an explainable deep model to learn multi-level user profile and infer which level of features best captures a user’s interest. Balog et al. (Balog et al., 2019) presented a set-based recommendation technique to improve the scrutability and transparency of recommender systems. Ren et al. (Ren et al., 2017) proposed a collaborative viewpoint regression for explainable social recommendation. Wang et al. (Wang et al., 2018b) proposed a tree-enhanced embedding method to combine embedding-based and tree-based models for explainable recommendation. More recently, Chen et al. (Chen et al., 2020b) applied a residual feed-forward neural network to model the user and item explicit features and generates explainable substitute recommendations. Pan et al. (Pan et al., 2020) presented a feature mapping approach to map the latent features onto the interpretable aspects to achieve both satisfactory accuracy and explainability. Li et al. (Li et al., 2021) proposed a Personalized Transformer to generate aspect-inspired natural language explanations. Xian et al. (Xian et al., 2021) developed an attribute-aware algorithm for explainable item-set recommendation and deployed in Amazon. Some other explainable recommendation methods include knowledge graph-based explanations (Xian et al., 2019, 2020; Ai et al., 2018), neural logic explanations (Zhu et al., 2021), visual explanations (Chen et al., 2019b), natural language explanations (Li et al., 2020; Chen et al., 2019a; Li et al., 2021; Li et al., 2017), dynamic explanations (Chen et al., 2019d), reinforcement learning-based explanations (Wang et al., 2018a), conversational explanations (Chen et al., 2020a; Zhao et al., 2019), fair explanations (Fu et al., 2020), disentangled explanations (Ma et al., 2019), review-based explanations (Chen et al., 2018; Ni et al., 2019), etc.

Most of the existing approaches generate explanations based on a very similar hypothesis: If there exists an aspect that best matches between the user’s preference and the item’s performance, then this aspect would be the explanation of the recommendation. However, our method generates explanations from a counterfactual perspective: if an item would not have been recommended had it performed slightly worse on some aspects, then these aspects would be the reason for the model to recommend this item.

Counterfactual reasoning, together with logical reasoning (Shi et al., 2020; Chen et al., 2021), are two important types of cognitive reasoning approaches. Recently, counterfactual reasoning has drawn attention in explainable AI research. It has some successful applications in several machine learning fields such as computer vision (Goyal et al., 2019), natural language processing (Hendricks et al., 2018; Feder et al., 2020), and social fairness (Wachter et al., 2018; Dandl et al., 2020). In the recommendation field, recent works used counterfactual reasoning to improve both recommendation accuracy (Wang et al., 2021; Xu et al., 2021a) and explainability (Ghazimatin et al., 2020; Xu et al., 2021b; Tran et al., 2021) based on heterogeneous information networks (Ghazimatin et al., 2020), perturbation model (Xu et al., 2021b) or influence functions (Tran et al., 2021), e.g., Ghazimatin et al. (Ghazimatin et al., 2020) tried to generate provider-side counterfactual explanations by looking for a minimal set of user’s historical actions (e.g. reviewing, purchasing, rating) such that the recommendation can be changed by removing the selected actions. Tran et al. (Tran et al., 2021) adopted influence functions for identifying training points most relevant to a recommendation while deducing a counterfactual set for explanations.

A common factor between our work with prior work is that both of the proposed methods generate explanations based on extracted causalities rather than associative relationships. Yet, our work is different from prior works on two key points: 1) In terms of problem definition, prior works generate counterfactual explanations on the user-side based on user actions while our method generates counterfactual explanations on the item-side based on item aspects, which are two different types of explanations. 2) In terms of technique, our method adopts a counterfactual learning framework driven by the Occam’s Razor Principle (Blumer et al., 1987) to directly learn an explanation of small complexity and large strength, so that our desire of finding simple and effective explanation is directly encoded into the model objective.

Suppose we have a user set with $m$ users $\mathcal{U}=\{u_{1},u_{2},\cdots,u_{m}\}$ and an item set with $n$ items $\mathcal{V}=\{v_{1},v_{2},\cdots,v_{n}\}$. Let binary matrix $B\in\{0,1\}^{m\times n}$ be the user-item interaction matrix, where $B_{i,j}=1$ if user $u_{i}$ interacted with item $v_{j}$; otherwise, $B_{i,j}=0$. We use $\mathcal{R}(u,K)$ to represent the top-$K$ recommendation list for a user $u$, and we say $v\in\mathcal{R}(u,K)$ if item $v$ is recommended to user $u$ in the user’s top-$K$ list. Following the same method described in Zhang et al. (Zhang et al., 2014a), we apply the sentiment analysis toolkit¹¹1https://github.com/evison/Sentires/ built in (Zhang et al., 2014b) to extract (Aspect, Opinion, Sentiment) triplets from the textual reviews. For example, in the Cell Phone domain, the extracted aspects would include color, price, screen, battery, etc. Besides, suppose we have a total number of $r$ item aspects $\mathcal{A}=\{a_{1},a_{2},\cdots,a_{r}\}$. Same as (Zhang et al., 2014b), we further compute the user-aspect preference matrix $X\in\mathbb{R}^{m\times r}$ and the item-aspect quality matrix $Y\in\mathbb{R}^{n\times r}$. $X_{i,k}$ indicates to what extent the user $u_{i}$ cares about the item aspect $a_{k}$. Similarly, $Y_{j,k}$ indicates how well the item $v_{j}$ performs on the aspect $a_{k}$. More specifically, $X$ and $Y$ are calculated as:

where $N$ is the rating scale in the system, which is 5-star in most cases. $t_{i,k}$ is the frequency that user $u_{i}$ mentioned aspect $a_{k}$. $t_{j,k}$ is the frequency that item $v_{j}$ is mentioned on aspect $a_{k}$, while $s_{j,k}$ is the average sentiment of these mentions. For both the $X$ and $Y$ matrices, their elements are re-scaled into the range of $(1,N)$ using the sigmoid function (see Eq.(1)) to match with the system’s rating scale. Since the matrix construction process is not the focus of this work, we only briefly describe this process and readers may refer to (Zhang et al., 2014b, a) for more details. The same user-aspect and item-aspect matrix construction technique is also used in (Wang et al., 2018c; Gao et al., 2019; Le and Lauw, 2021).

With the above definitions, the objective of our counterfactual reasoning problem is to search for aspect-driven counterfactual explanations for a given black-box recommendation model.

More specifically, for a given recommendation model, if item $v_{j}$ is recommended to user $u_{i}$, i.e., $v_{j}\in\mathcal{R}(u_{i},K)$, then we look for a slight change vector $\Delta=\{\delta_{0},\delta_{1},\cdots,\delta_{r}\}$ for the item-aspect quality vector $Y_{j}$, such that if $\Delta$ is applied on item $v_{j}$’s quality vector, i.e., $Y_{j}+\Delta$, then it will change the recommendation result to make item $v_{j}$ disappear from the recommendation list, i.e., $v_{j}\notin\mathcal{R}(u_{i},K)$. All the values in $\Delta$ are either zero or negative continuous values since an item will only be removed from the recommendation list if it performs worse on some aspects. With the optimized vector $\Delta$, we can construct the counterfactual explanation for item $v_{j}$, which is composed of the aspects corresponding to the non-zero values in $\Delta$. The counterfactual explanation takes the following form, If the item had been slightly worse on [aspect(s)], then it will not be recommended. where the [aspect(s)] are selected by $\Delta$ as mentioned above. In the following, we will define the properties of $\Delta$ in more formal ways.

To better understand the counterfactual explainable recommendation problem, we introduce two concepts to motivate explainable recommendation under the counterfactual reasoning background.

The first is Explanation Complexity (EC), which measures how complicated the explanation is. In our aspect-based explainable recommendation setting, the complexity can be defined as 1) how many aspects are used to generate the explanation, which corresponds to the number of non-zero values in $\Delta$, i.e., $\|\Delta\|_{0}$, and 2) how many changes need to be applied on these aspects, which can be represented as the sum of square of $\Delta$, i.e., $\|\Delta\|_{2}$. The final complexity takes a weighted sum of the two factors:

where $\gamma$ is a hyper-parameter to control the trade-off between these two terms.

The second is Explanation Strength (ES), which measures how effective the explanation is. In our counterfactual explainable recommendation setting, this can be defined as to what extent applying the slight change vector $\Delta$ will influence the recommendation result of item $v_{j}$. This can be further defined as the decrease of $v_{j}$’s ranking score in user $u_{i}$’s recommendation list after applying $\Delta$:

where $s_{i,j}$ is the original ranking score of item $v_{j}$, and $s_{i,j_{\Delta}}$ is the ranking score of $v_{j}$ after $\Delta$ is applied to its quality vector, i.e., $Y_{j}+\Delta$.

We should note that Eq.(2) and (3) are not the only way to define explanation complexity and strength. The definition depends on what we need in practice. Our counterfactual reasoning framework introduced in Section 4 is flexible and can easily adapt to different definitions of explanation complexity and strength.

It is also worth discussing the relationship between explanation complexity and strength. Actually, complexity and strength are two orthogonal concepts, i.e., a complex explanation is not necessarily strong, and a simple explanation is not necessarily weak. There may well exist explanations that are complex but weak, or simple and strong. According to the Occam’s Razor Principle (Blumer et al., 1987), if two explanations are equally effective, we prefer the simpler explanation than the complex one. As a result, counterfactual explainable recommendation aims to seek the simple (low complexity) and effective (high strength) explanations for recommendation.

Suppose we have a black-box recommendation model $f$ that predicts the user-item ranking score $s_{i,j}$ for user $u_{i}$ and item $v_{j}$ by:

where $X_{i}$ and $Y_{j}$ are the user-aspect vector and item-aspect vector, respectively, as defined in Eq.(1); $\Theta$ is the model parameter, and $Z$ represents all other auxiliary information of the model. Depending on the application, $Z$ could be ratings, clicks, text, images, etc., and $Z$ is optional in the recommendation model. Basically, the recommendation model $f$ can be any model as long as it takes the user’s and the item’s aspect vectors as part of the input, which makes our counterfactual reasoning framework applicable to a wide scope of models.

In this work, to demonstrate the idea of counterfactual reasoning, we use a very simple deep neural network as the implementation of the recommendation model $f$, which includes one fusion layer followed by three fully connected layers. The network concatenates the user’s and the item’s aspect vectors as input and outputs a one-dimensional ranking score $s_{i,j}$. The final output layer is a sigmoid activation function so as to map $s_{i,j}$ into the range of $(0,1)$. Then, we train the model with a cross-entropy loss:

where $B_{i,j}=1$ if user $u_{i}$ previously interacted with item $v_{j}$, otherwise $B_{i,j}=0$. In practice, since $B$ is a very sparse matrix, we sample the negative samples with ratio 1:2, i.e., for each positive instance we sample two negative instances. With this pre-trained recommendation model, for a target user, we are able to recommend top-$K$ items according to the predicted ranking scores.

We build a counterfactual reasoning model to generate explanations for any item in the top-$K$ recommendation list provided by an existing recommendation model. The essential idea of the proposed explanation model is to discover slight change $\Delta$ on the item’s aspects via solving a counterfactual optimization problem which is formulated in the following.

Suppose item $v_{j}$ is in the top-$K$ recommendation list for user $u_{i}$ ($v_{j}\in\mathcal{R}(u_{i},K)$). As mentioned before, our counterfactual reasoning model aims to find simple and effective explanations for $v_{j}$, which can be shown as the following constrained optimization framework,

Mathematically, according to our definition of explanation complexity and strength in Section 3, the framework can be realized with the following specific optimization problem,

where $s_{i,j}=f(X_{i},Y_{j}\mid Z,\Theta)$, $s_{i,j_{\Delta}}=f(X_{i},Y_{j}+\Delta\mid Z,\Theta)$. In the above equation, $s_{i,j}$ is the original ranking score of item $v_{j}$, $s_{i,j_{\Delta}}$ is the ranking score of $v_{j}$ when the slight change vector $\Delta$ is applied on $v_{j}$’s aspect vector. The intuition of Eq.(7) is trying to find an explanation $\Delta$ that is both simple and effective, where “simple” is reflected by the optimization objective, i.e., explanation complexity $C(\Delta)$ is minimized, while “effective” is reflected by the optimization constraint, i.e., the explanation strength $S(\Delta)$ should be big enough to remove item $v_{j}$ from the top-$K$ list.

To realize the second goal (i.e., effective/strong enough), we take the threshold $\epsilon$ as the margin between item $v_{j}$’s score and the $K+1$’s item’s score in the original recommendation list, i.e.,

where $s_{i,j_{K+1}}=f(X_{i},Y_{j_{K+1}}\mid Z,\Theta)$ is the ranking score of the $K+1$’s item, and thus Eq.(7) can be simplified as,

In this way, item $v_{j}$ will be ranked lower than the $K+1$’s item and thus be removed from the top-$K$ list.

A big challenge to optimize Eq.(9) is that both the objective $\|\Delta\|_{2}+\gamma\|\Delta\|_{0}$ and the constraint $s_{i,j_{\Delta}}\leq s_{i,j_{K+1}}$ are not differentiable. In the following, we relax the two parts to make the equation optimizable.

For the objective, since $\|\Delta\|_{0}$ is not convex, we relax it with $\ell_{1}$-norm $\|\Delta\|_{1}$. This is shown to be efficient and provides good vector sparsity in (Candès et al., 2006; Candes and Tao, 2005), thus helps to minimize the explanation complexity in terms of the number of aspects in the explanation.

For the constraint $s_{i,j_{\Delta}}\leq s_{i,j_{K+1}}$, we relax it as a hinge loss:

and add it as a Lagrange term into the total objective. Thus, the final optimization equation for generating explanation becomes:

In Eq.(11), $\lambda$ and $\alpha$ are hyper-parameters to control the explanation strength. A sacrifice of using relaxed optimization is that we lose the guarantee that item $v_{j}$ is removed from the top-$K$ list, though the probability of removing is high due to minimizing the $L(s_{i,j_{\Delta}},s_{i,j_{K+1}})$ term. As a result, it requires a post-process to check if $s_{i,j_{\Delta}}$ is indeed smaller than $s_{i,j_{K+1}}$. We should only generate counterfactual explanations when the removal is successful. In the experiments, we will report the fidelity of our explanation model to show what percentage of items can be explained by our method.

Besides, there is a trade-off in the relaxed model: by increasing the hyper-parameter $\lambda$, the model will focus more on the explanation strength but less on the explanation complexity. We will also explore the influence of $\lambda$ and the relationship between explanation complexity and strength in the ablation study of the experiments.

In user-oriented evaluation, we adopt the user’s review on the item as the ground-truth reason about why the user purchased the item, which is similar to previous works (Li et al., 2020; Wang et al., 2018c; Li et al., 2017; Chen et al., 2019c, 2018). More specifically, from the textual review that a user $u_{i}$ provided on an item $v_{j}$, we extract all the aspects that $u_{i}$ mentioned with positive sentiment, which is defined as $P_{i,j}=\big{[}p_{i,j}^{(0)},p_{i,j}^{(1)},\cdots,p_{i,j}^{(r)}\big{]}$. $P_{i,j}$ is a binary vector, where $p_{i,j}^{(k)}=1$ if user $u_{i}$ has positive sentiment on the aspect $a_{k}$ in his/her review for item $v_{j}$. Otherwise, $p_{i,j}^{(k)}=0$. On the other hand, our model will produce the vector $\Delta=\{\delta_{0},\delta_{1},\cdots,\delta_{r}\}$, and those aspect(s) corresponding to the non-zero values in $\Delta$ will constitute the explanation.

Then, for each user-item pair, we calculate the precision and recall of the generated explanation $\Delta$ with regard to the ground-truth vector $P_{i,j}$:

where $I(\delta)$ is an identity function such that $I(\delta)=1$ when $\delta\neq 0$, and $I(\delta)=0$ when $\delta=0$. In our case, the Precision measures the percentage of aspects in our generated explanation that are really liked by the user, while Recall measures how many percentage of aspects liked by the user are really included in our explanation. We also calculate the $F_{1}$ score as the harmonic mean between the two, i.e., $F_{1}=2\cdot\frac{\text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}$. Then, we average the scores of all pairs as the final Precision, Recall, and $F_{1}$ measure.

The user-oriented evaluation only answers the question of whether the generated explanations are consistent with user’s preferences. However, it does not tell us whether the explanation properly justifies the model’s behaviour, i.e., why the recommendation model recommends this item to the user.

To test if our explanation model correctly explains the essential mechanism of the recommendation system, we use two scores, Probability of Necessity (PN) and Probability of Sufficiency (PS) (Pearl et al., 2016, p.112), to validate the explanations with model-oriented evaluation.

In logic and mathematics, necessity and sufficiency are terms used to describe a conditional or implicational relationship between two statements. Suppose we have $S\Rightarrow N$, i.e., if $S$ happens then $N$ will happen, then we say $S$ is a sufficient condition for $N$. Meanwhile, we have the logically equivalent contrapositive $\neg N\Rightarrow\neg S$, i.e., if $N$ does not happen, then $S$ will not happen, as a result, we say $N$ is a necessary condition for $S$.

Probability of Necessity (PN) (Pearl et al., 2016): In causal inference theory, probability of necessity evaluates the extent that a condition is necessary. To calculate PN for the generated explanation, suppose a set of aspects $\mathcal{A}_{ij}\subset\mathcal{A}$ compose the explanation for the recommendation of item $v_{j}$ to user $u_{i}$. The essential idea of the PN score is: if in a counterfactual world, the aspects in $\mathcal{A}_{ij}$ did not exist in the system, then what is the probability that item $v_{j}$ would not be recommended for user $u_{i}$.

Following this idea, we calculate the frequency of the generated explanations that meet the PN definition. Let $R_{i,K}$ be user $u_{i}$’s original recommendation list. Let $v_{j}\in R_{i,K}$ be a recommended item that our framework generated a nonempty explanation $\mathcal{A}_{ij}\neq\emptyset$. Then for all the items in the universal item set $\mathcal{V}$, we alter the aspect values in the item-aspect quality matrix $Y$ to 0 if they are in $\mathcal{A}_{ij}$. In this way, we create a counterfactual item set $\mathcal{V}^{*}$ which results in a counterfactual recommendation list $R_{i,K}^{*}$ for user $u_{i}$ by the recommendation algorithm. Then, the PN score is:

where $I(\mathcal{A}_{ij}\neq\emptyset)$ is an identity function such that $I(\mathcal{A}_{ij}\neq\emptyset)=1$ if the condition $\mathcal{A}_{ij}\neq\emptyset$ holds and 0 otherwise. Basically, the denominator is the total number of items that the algorithm successfully generated an explanation for, and the numerator is the number of explanations that if we remove the related aspects then it will cause the item to be removed from the recommendation list.

Probability of Sufficiency (PS) (Pearl et al., 2016): The PS score evaluates the extent that a condition is sufficient. The essential idea of the PS score is: if in a counterfactual world, the aspects in $\mathcal{A}_{ij}$ were the only aspects existed in the system, then what is the probability that item $v_{j}$ would still be recommended for user $u_{i}$.

Similarly, we calculate the frequency of the generated explanations that meet the PS definition. For all the items in $\mathcal{V}$, we alter the aspect values in the item-aspect quality matrix $Y$ to 0 if they are not in $\mathcal{A}_{ij}$. In this way, we create a counterfactual item set $\mathcal{V}^{\prime}$ which results in a counterfactual recommendation list $R_{i,K}^{\prime}$ for user $u_{i}$ by the recommendation algorithm. Then, the PS score is:

where $I(\mathcal{A}_{ij}\neq\emptyset)$ is still the identity function as above. Basically, the denominator is still the total number of items that the algorithm successfully generated an explanation for, and the numerator is the number of explanations that alone can still recommend the item to the recommendation list. Similar to the user-oriented evaluation, we also calculate the harmonic mean of PS and PN to measure the overall performance, which is $F_{NS}=\frac{2\cdot\text{PN}\cdot\text{PS}}{\text{PN}+\text{PS}}$.

We test our method on Yelp²²2https://www.yelp.com/dataset and Amazon³³3https://nijianmo.github.io/amazon datasets. The Yelp dataset contains users’ reviews on various kinds of businesses such as restaurants, dentists, salons, etc. The Amazon dataset (Ni et al., 2019) contains user reviews on products in Amazon e-commerce system. The Amazon dataset contains 29 sub-datasets corresponding to 29 product categories. We adopt four datasets of different scales to evaluate our method, which are Electronic, Cell Phones and Accessories, Kindle Store and CDs and Vinyl. Since the Yelp and Amazon datasets are very sparse, similar as previous work (Zhang et al., 2014a; Wang et al., 2018c; Xian et al., 2019, 2020), we remove the users with less than 20 reviews for Yelp dataset, and 10 reviews for Amazon dataset. The statistics of the datasets are shown in Table 1.

We compare our model with three aspect-aware explainable recommendation models. We also include a random explanation baseline to show the overall difficulty of the evaluation tasks.

EFM (Zhang et al., 2014a): The Explicit Factor Model. This work combines matrix factorization with sentiment analysis technique to align latent factors with explicit aspects. In this way, it predicts the user-aspect preference scores and item-aspect quality scores. The top-1 aligned aspect is used to construct aspect-based explanation.

MTER (Wang et al., 2018c): The Multi-Task Explainable Recommendation model. This work predicts a tensor $X\in\mathbb{R}^{m\times n\times(r+1)}$, which represents the affinity score among the users, items, aspects, and an extra dimension for the overall rating. This tensor $X$ is acquired via Tucker decomposition (Karatzoglou et al., 2010; Kolda and Bader, 2009). We should note that since the overall rating for a user on an item is predicted in the extra dimension via decomposition, which is not directly predicted by the explicit aspects, this method is not suitable for the model-oriented evaluation. As a result, we only report this model’s explanation performance on user-oriented evaluation.

A2CF (Chen et al., 2020b): The Attribute-Aware Collaborative Filtering model. This work leverages a residual feed-forward network to predict the missing values in the user-aspect matrix $X$ and the item-aspect matrix $Y$. The method originally considers both the user-item preference and the item-item similarity to generate explainable substitute recommendations. We remove the item-item factor to make it compatible with our problem setting to generate explanations for any item. Similar to (Zhang et al., 2014a), the top-1 aligned aspect will be used for explanation.

Random: For each item recommended to a user, we randomly choose one or multiple aspects from the aspect space and generate the explanation based on them. The evaluation scores of the random baseline can indicate the difficulty of the task.

We first report the fidelity of our explanation method in Table 2, which shows for what percentage of recommended items that our method can successfully generate an explanation. We notice that the fidelity of the single-aspect version is lower than that of the multi-aspect version. This indicates that for our model, using only one aspect as the explanation may not lead to enough explanation strength to effectively explain the recommendations. However, if we allow the model to use multiple aspects, we can find strong enough explanations in most cases ($80\%\sim 100\%$). The overall average number of aspects in multi-aspect explanation is $2.79$.

We then evaluate the explanations generated by CountER (original version and masked version) and the baselines. The user-oriented evaluation is reported in Table 3 and the model-oriented evaluation is reported in Table 4. We note that the random baseline performs very bad on both evaluations, which shows the difficulty of the task and that randomly choosing aspects as explanations can barely reveal the reasons of the recommendations.

For the user-oriented evaluation, the results show that when applying the mask for fair comparison, CountER outperforms all the baselines on all the datasets in terms of $F_{1}$ scores. Moreover, CountER performs better than the baselines on precision in $90\%$ cases and on recall in $80\%$ cases. We note that our model has a very huge improvement than the baselines on Yelp dataset even without mask, and the reason may be that the Yelp dataset is much denser and has more reviews than other datasets so that the user’s review bias has smaller impact. This also indicates that CountER has more advantages when the size of dataset increases.

For model-oriented evaluation, the mask limits CountER’s ability to explain the model’s behavior. However, no matter with or without the mask, our model has better performance than all the baselines according to $F_{NS}$ score. With mask, CountER has $15.63\%$ average improvement than the best performance of the baselines on $F_{NS}$. Without the mask, the average improvement is $38.49\%$.

Another observation is that the baselines commonly have higher PS scores, despite much lower on the PN scores. One possible reason is due to the mechanism of the matching based explanation methods. For an item on the recommendation list, they try to find well-aligned aspects where the user and the item both perform well. When computing the PS score, the baselines only reserve these well-aligned aspects in the aspect space for all the items, thus the recommended item will highly possibly still stay in the recommendation list, which results in a high PS score. However, when computing the PN score, though the baselines remove all these aspects, the recommended item may still perform better on other aspects compared with other items and thus still remain in the recommendation list, which results in a lower PN score. This also sheds light on why the matching based methods may not discover the true reasons behind the recommendations.

Since $\lambda$ is applied on the hinge loss $L(s_{i,j_{\Delta}},s_{i,j_{K+1}})$ (Eq.(11) and (12)), a larger $\lambda$ emphasizes more on the explanation strength and reduces the importance of complexity. As shown in Figure 2(a), when $\lambda$ increases, the model successfully generates explanations for more items, but the explanation complexity also increases because more aspects are needed to explain those “hard” items. However, the higher $\lambda$ is, the worse our model performs on the user-oriented evaluation. Besides, Figure 2(b) shows the explanation strength does not change with $\lambda$. This is because the $\alpha$ in the hinge loss controls the required margin on the ranking score to flip a recommendation. Additionally, we notice that the model-oriented performance is also independent to $\lambda$, which is the same as the explanation strength.

Based on the above results, we hypothesize that the user-oriented performance may be related to the explanation complexity, while the model-oriented performance may be related to the explanation strength. This hypothesis is justified in Section 6.6.

In this section, we study the Explanation Complexity and Explanation Strength. In Section 4.4.1, we discussed that the difficulty of removing the items in the top-$K$ list are different based on their positions. Figure 2(c) shows the mean complexity and strength of the explanations for items at different positions (i.e., from the first to the fifth). Generally speaking, it requires $1.59$ more aspects to remove the items at the first position from the recommendation list than the items at the fifth position. Because they require larger change to be removed. This is in line with the fact that the strongly recommended items have more reasons to be recommended.

In Figure 2(d), we illustrate the relationship between $F_{1}$ score and explanation complexity/strength. The scale of the marker points represent how large the $F_{1}$ score is. It shows that the user-oriented evaluation score $F_{1}$ decreases as the complexity increases, meanwhile, $F_{1}$ is relatively independent from the explanation strength.

On the contrary, in Figure 2(e), we plot the distribution of the explanations that are both necessary and sufficient (i.e., $\text{PN}_{ij}\wedge\text{PS}_{ij}=1$), as well as the distribution of explanations that are either unnecessary or insufficient (i.e., $\text{PN}_{ij}\wedge\text{PS}_{ij}=0$). We can see that as the explanation strength increases, more and more portion of the explanations are both necessary and sufficient. However, this tends to be irrelevant with the complexity of the explanations.

These observations are important because: 1) They indicate that the Explanation Complexity and Explanation Strength are two orthogonal concepts. Both of them are very important since the complexity is related to the coverage on the user’s preference and the strength is related to the model’s mechanism. 2) It legitimatizes the Occam’s Razor Principle and further justifies the motivation of our CountER model, which is to extract both simple (low complexity) and effective (high strength) explanations for recommendations.