Explanation as a Defense of Recommendation

As our focus in this work is not to design yet another recommendation algorithm, we adopted a latest neural collaborative filtering solution for the purpose (He et al., 2017). Arguably any latent factor models that explicitly learn user and item representations (Koren et al., 2009; Zhang et al., 2014a; Wang et al., 2018b) can be adopted. In this section, we will only cover the most important technical details of our recommender’s design, and leave interested readers to its original paper for more details.

We stack two Multi-Layer Perceptron (MLP) networks to predict the recommendation score $\hat{r}_{u,i}$ for a given $(u,i)$ pair. The first MLP encodes the $(u,i)$ pair to a latent sentiment vector $\mathbf{s}_{u,i}\in\mathbb{R}^{d^{r}_{s}}$, and the second MLP maps the sentiment vector $\mathbf{s}_{u,i}$ into the numerical rating $\hat{r}_{u,i}$. We refer to the first MLP as sentiment encoder and the second one as rating regressor. Instead of using the predicted score $\hat{r}_{u,i}$ to influence explanation generation, we choose to inform the explanation generator by the encoded sentiment vector $\mathbf{s}_{u,i}$. We defer the details of this design to the next section.

In the recommendation module, we define the latent embedding matrices for users and items as $P^{r}\in\mathbb{R}^{d^{r}\times|\mathcal{U}|}$ and $Q^{r}\in\mathbb{R}^{d^{r}\times|\mathcal{I}|}$ respectively, where $d^{r}$ is the dimension of the embedding vectors. The sentiment encoder concatenates the embedding vector $p^{r}_{u}$ and $q^{r}_{i}$ as its input and passes it through multiple layers with leaky ReLU activation to get the sentiment vector $\mathbf{s}_{u,i}$ encoded. Besides its use in the explanation generator, $\mathbf{s}_{u,i}$ is then mapped by the rating regressor through another set of multi-layer leaky ReLUs to get the final recommendation score ${\hat{r}}_{u,i}$.

In addition to the popularly used Minimal Squared Error (MSE) (Chen et al., 2018; Li et al., 2017) to train our recommender, we also introduce a pairwise hinge loss to improve the trained recommender’s ranking performance. Specifically, for each user $u$, we collect a set of personalized item pairs $\mathcal{B}_{u}=\{(i,j)|r_{u,i}>r_{u,j}\}$, where $i$ and $j$ are two items rated by user $u$ and one is preferred than another as observed in the training dataset. We did not use the popular BPR loss (Rendle et al., 2012), because it tends to push ratings to extreme values, which is inconsistent with our sentiment regularizer’s requirement to be explained later.

Based on the rating set $\mathcal{R}$ and personalized item pair set $\{\mathcal{B}_{u}\}_{u\in\mathcal{U}}$, the loss for recommender training is defined as:

where $\beta>0$ is a hyper-parameter to control the separation margin, i.e., it penalizes the model when the predicted difference between $\hat{r}_{u,i}$ and $\hat{r}_{u,j}$ is smaller than $\beta$, and $\lambda_{h}$ is the coefficient to control the balance between MSE loss and pairwise hinge loss.

Motivated by the success of neural language generation, we appeal to a Recurrent Neural Network (RNN) model with Gated Recurrent Units (GRUs) (Chung et al., 2014) for explanation generation. To make the generation related to the user and item, we first map the input user $u$ and item $i$ to their embeddings $p^{x}_{u}$ and $q^{x}_{i}$ with the latent matrices $P^{x}\in\mathbb{R}^{d^{x}\times|\mathcal{U}|}$ and $Q^{x}\in\mathbb{R}^{d^{x}\times|\mathcal{I}|}$ learnt by the explanation generator. We should note this set of embeddings are different from those used in the recommender (i.e., $P^{r}$ and $Q^{r}$), as they should characterize different semantic aspects of users and items (ratings vs., text). We hence use superscript $x$ to indicate variables and parameters related to explanation generator. To generate explanation text, the embeddings are concatenated and linearly converted into the initial RNN hidden state; and then the GRU generates hidden state $\mathbf{h}^{x}_{t}\in\mathbb{R}^{d^{x}_{h}}$ at position $t$ with previous state $\mathbf{h}^{x}_{t-1}$ and input word $w_{t}$, and predicts the next word $w_{t+1}$ recursively.

We initialize RNN with pretrained GloVe word embeddings $V\in\mathbb{R}^{d^{x}_{\mathbf{v}}\times|\mathcal{V}|}$ (Pennington et al., 2014).

Though similar model design has been used for explanation generation (Li et al., 2017; Sun et al., 2020), this straightforward application of RNN can hardly generate satisfactory explanations, where two issues are left open. First, a good explanation is expected to be personalized and specific about the recommendation; generic content, such as “this is a nice restaurant” can never be informative. It is important to explain the recommended item by the user’s most concerned attributes (Wang et al., 2010). Second, the sentiment carried in the explanation, especially on the mentioned attributes, should be explicit and consistent with the recommendation (as shown in our case study in Table 1). There is no guarantee that a simple RNN can satisfy both requirements.

We enhance our generator design with two gated sub-networks upon GRU to address the aforementioned issues. First, we design a sub-network, named attribute gate, to guide attribute word generation with respect to the input user-item pair and the predicted recommendation sentiment. The attribute gate is built based on a pointer network (or copy mechanism) (See et al., 2017; Zeng et al., 2016), which decides whether the current position should mention an attribute word and the corresponding distribution of attribute words based on the generation context. To make the choice of attribute word specific to the item, for each item $i$ we build an attribute set with all attribute words that appear in $i$’s associated training explanation text: $\mathcal{A}_{i}=\big{\{}a_{k}|a_{k}\in\{x_{u,i}|u\in\mathcal{U}\}\big{\}}$. To make the attribute choice depend on the already generated content, we attend on the concatenation of the current position’s RNN hidden state $\mathbf{h}^{x}_{t}$ and sentiment vector $\mathbf{s}_{u,i}$ to compute the distribution of these attribute words,

where $W^{x}_{z}\in\mathbb{R}^{(d^{x}_{h}+d^{r}_{s})\times d^{x}_{\mathbf{v}}}$ and $\mathbf{v}_{a_{k}}$ is the word embedding of attribute $a_{k}$. $\mathbf{z}_{t,k}$ is computed for every $a_{k}$ in $\mathcal{A}_{i}$, i.e., $\mathbf{z}_{t}=\{z_{t,1},z_{t,2}\\ ...z_{t,|\mathcal{A}_{i}|}\}$. $\mathbf{\zeta}_{t}$ is the resulting attribute word distribution at position $t$. For better performance, an extra linear transformation can be applied to $\mathbf{h}^{x}_{t}$ to compress it into a lower dimension before computing attention, which helps avoid overfitting attentions to the text generation context but ignoring the sentiment context.

To decide if we need to generate an attribute word using Eq (1) at position $t$, we compute the copy probability with respect to the current context $\mathbf{h}^{x}_{t}$ by $c^{x}_{t}=\sigma(W^{x}_{c}\mathbf{h}^{x}_{t}+b^{x}_{c})$, where $\sigma(\cdot)$ is the sigmoid function, $W^{x}_{c}\in\mathbb{R}^{d^{x}_{h}}$ and $b^{x}_{c}\in\mathbb{R}$. $c^{x}_{t}$ allows us to mix the vocabulary distribution predicted by GRU and attribute word choice to get our final word distribution at position $t$.

Second, we design a sentiment gate to fuse the sentiment vector $\mathbf{s}_{u,i}$ to align sentiment in the generated explanation text. Our key insight is that not all words convey sentiment, we need to choose the right word at the right place to express consistent sentiment as needed by the recommender. Similar to our attribute gate design, we apply a soft gate to decide how each position is related to the intended sentiment. At position $t$, the sentiment gate calculates a ratio $g^{x}_{t}$ with respect to the RNN’s hidden state $\mathbf{h}^{x}_{t}$. The sentiment vector $\mathbf{s}_{u,i}$ is then weighted and merged with $\mathbf{h}^{x}_{t}$,

where $W^{x}_{g}\in\mathbb{R}^{d^{x}_{h}}$ and $b^{x}_{g}\in\mathbb{R}$ produce a scalar $g^{x}_{t}$. $\mathbf{m}^{x}_{t}$ is the sentiment fused latent vector to predict the vocabulary distribution for position $t$. Because not all words are about sentiment, to better differentiate the positions where the intended sentiment needs to be expressed from the rest, we impose sparsity on the learned gate value $g^{x}_{t}$ using L1 regularization at training time. In other words, the gate is open only when necessary.

We compute the final word distribution by consolidating the outputs of the two gated sub-networks (Eq (1) and (2)). First, the sentiment fused latent vector $\mathbf{m}^{x}_{t}$ is fed through a linear layer to calculate the vocabulary distribution $\mathbf{\eta}_{t}=softmax(W^{x}_{\mathbf{v}}\mathbf{m}^{x}_{t}+b^{x}_{\mathbf{v}})$, where $W^{x}_{\mathbf{v}}\in\mathbb{R}^{|\mathcal{V}|\times d^{x}_{h}}$ and $b^{x}_{\mathbf{v}}\in\mathbb{R}^{|\mathcal{V}|}$. Second, the vocabulary distribution $\mathbf{\eta}_{t}$ and attribute word distribution $\mathbf{\zeta}_{t}$ are merged to obtain the final word distribution with respect to the copy probability $c^{x}_{t}$, i.e., $\mathbf{y}_{t}=(1-c^{x}_{t})\mathbf{\eta}_{t}+c^{x}_{t}\mathbf{\zeta}_{t}$, where the value of $w_{k}$ in $\mathbf{\zeta}_{t}$ is $0$ if $w_{k}$ is not an attribute word.

The objective for explanation generation is to minimize the negative log-likelihood loss (NLL) on the training explanation set $\mathcal{X}$,

where $\mathbf{y}_{t}(w_{t})$ is the resulting probability of word $w_{t}$ and $\lambda_{g}$ is the coefficient for the L1 regularization of the sentiment gate values.

Though our sentiment gate design (Eq (2)) introduces predicted sentiment from the recommender to the explanation generator, it is still insufficient to guarantee sentiment alignment, for three major reasons. First, word-based NLL training cannot maintain the whole sentence’s sentiment. For example, as the number of sentiment words in an explanation is less than the number of non-sentiment words, the training is affected more by those non-sentiment words. This weakens its prediction quality on sentiment words. Second, the explanation generator might utilize the sentiment vector differently as the recommender does, so that the recommendation rating might diverge from the sentiment carried by the explanation. Third, the generation process at the inference stage works differently from the training stage (Ranzato et al., 2015): at inference time, the previously decoded word is used as the input for the next word prediction, instead of the ground-truth word as at the training time. Hence, the learnt text pattern might not be fully exploited at the inference time.

We introduce the sentiment regularizer to close the loop between the recommender and explanation generator. It uses a stand-alone sentiment regressor to predict the sentiment rating $\hat{r}^{x}$ on the generated explanation text $\hat{x}_{u,i}$ for user-item pair $(u,i)$, and requires the explanation generator to match the rating $\hat{r}_{u,i}$ from the recommender accordingly. We do not have any particular assumption about the sentiment regressor; and any state-of-the-art regression models can be leveraged (Pang and Lee, 2007). In this work, we employed an MLP on top of a bidirectional RNN text encoder with inner attention for rating regression, and denote it as $f^{R}(x)\to r^{x}$. We pre-train this regressor based on ground-truth $\{\mathcal{R},\mathcal{X}\}$ in the training set; and fix the learnt model thereafter.

To enforce sentiment alignment by the predicted ratings, we introduce a new loss to the training of our explanation generator,

where $P(\hat{x}|u,i)$ is the probability of generating $\hat{x}$ for the given $u$ and $i$. We should note this loss is not necessarily restricted to the observed $(u,i)$ pairs in the training set; instead, it could be any pairs of them, since both the recommender and explanation generator can generate output on any given $(u,i)$ pair. It thus enables data augmentation for sentiment alignment.

However, because the word distribution is categorical, the generation of $\hat{x}$ is not differentiable. It makes direct optimization with respect to Eq (3) infeasible. As a result, we appeal to Gumbel softmax (Jang et al., 2016) to obtain approximated gradient of sampling from a categorical distribution. Briefly, Gumbel softmax reparameterizes the randomness in sampling by a gumbel distribution and simulates a relaxed one-hot vector with softmax. As we need a strict one-hot vector to represent each single word, we adopt the Straight-Through (ST) Gumbel softmax estimator (Jang et al., 2016). For each $(u,i)$ pair in Eq (3), we back-propagate the gradient from $L^{a}$ to the explanation generator to improve the quality of sentiment alignment on the whole sequence.

Unfortunately, this new sentiment alignment loss might also attract the generation process to produce unreadable sequences, which however match the intended sentiment ratings. For example, the sentiment regressor may give a very positive rating to an unnatural sentence “good good good good”, when the recommender also happens to predict a high rating for this item. Giving a higher weight to the NLL loss $L^{x}$ in explanation learning cannot address this issue, as it cannot inform why a particular sequence should not be generated.

To improve the readability of our generated explanation, we introduce a text discriminator $f^{D}$, which learns to differentiate the authentic explanations from the generated ones, to guide the explanation generation as well. Our design allows any text classifier. In this work, we used an MLP binary classifier on top of a bidirectional RNN encoder for the purpose. We train the discriminator using cross-entropy loss with the ground-truth explanations $x$ as positive and the generated explanations $\hat{x}$ as negative,

Correspondingly, another objective of explanation generation is to fool the discriminator, i.e., the adversarial loss,

This loss also requires sampled explanations $\hat{x}$ as the input, like the alignment loss defined in Eq (3). The same Gumbel softmax sampling technique is used for end-to-end training.

As we pointed out before, addressing the sentiment alignment issue in training alone is still insufficient, we introduce a constraint-driven decoding strategy to enhance the alignment at the inference stage as well. Similarly as in training, we use MSE to quantify the difference between the rating predicted from the explanation text and that from the recommender. But at the inference stage, since the explanation generator has been trained and fixed, the discrepancy can only be minimized by varying the generation of explanation text, e.g., trial and error.

Because the sentiment regressor can only be applied to a complete sequence, the search space is too large to enumerate by the generator. Hence, we treat generating explanation $\hat{x}$ at inference time as a sequence of decision making, where each action is to generate a word $w_{t}$ at position $t$, given its already generated prefix as state. But we do not have feedback on the actions, until we complete $\hat{x}$; and the return for taking the series of actions can be measured by $Q(\hat{x};\hat{r}_{u,i})=[\hat{r}_{u,i}-f^{R}(\hat{x})]^{2}$. To find a policy that minimizes return (since we want to reduce the discrepancy), we need to estimate the value function under each state. This is a well studied problem in reinforcement learning, and it can be effectively addressed by Monte Carlo Tree Search (MCTS) (Kocsis and Szepesvári, 2006). Basically, we estimate the value function using our trained explanation generator for roll-out. When at position $t$ for generating $\hat{x}_{u,i}$, we will sample $n$ complete sequences for every action $w$ using the current prefix $\{w_{1},w_{2},\dots,w_{t-1}\}$, following the distribution specified by the explanation generator: $\hat{X}_{u,i,t}(w)=\big{\{}\hat{x}_{k}=MCTS_{u,i}(w_{1},w_{2},\dots,w_{t-1},w)\big{\}}^{n}_{k=1}$. Then the value of taking action $w$ at position $t$ can be estimated by,

Based on the estimated values, we can take the action that minimizes the value. A recent study (Holtzman et al., 2019) suggests that top-k sampling oftentimes avoids bland and repetitive content compared to more commonly used greedy decoding strategies, such as top-1 or beam search. Therefore, we integrate our MCTS with top-k sampling, i.e., at each decoding position $t$, we sample $k$ most likely words according to word distribution $\mathbf{y_{t}}$ and then use MCTS to select the one that minimizes the estimated value under given state.

A vanilla implementation of MCTS is expected to be expensive and slow in our problem, as it needs to complete the sequence at each position from an RNN model for multiple times. Fortunately, our sentiment gate design provides a short path for efficient sampling: as sentiment is only carried by a small number of words, there is no need to conduct such expensive sampling procedure at every position. Instead, we only need to perform MCTS at positions where sentiment is expressed. Hence, we set a threshold on the sentiment gate’s value to decide when to perform MCTS. When the gate’s value is below the threshold, we will directly sample from the top-k words of the explanation generator’s prediction.

Putting together the three components in our proposed explainable recommendation solution SAER, the overall objective of our model training is formulated as:

where $\Theta$ is the complete set of model parameters, and $\{\lambda_{r},\lambda_{x},\lambda_{a},\lambda_{c}\}$ are the corresponding coefficients to control the relative importance of each component in model training. We also include an L2 regularization for the model parameters $\Theta$, weighted by its coefficient $\lambda_{n}$. The parameters are then effectively estimated end-to-end with stochastic gradient optimizer of Adam (Kingma and Ba, 2014).

However, due to our model’s complex structure, it is challenging to fully unleash the optimizer’s potential on its own. Therefore, we split the whole training process into five stages. First, estimate the sentiment regressor on $\{\mathcal{X},\mathcal{R}\}$, as it does not depend on the other parts of our model. Second, pre-train the recommender on $\{\mathcal{U},\mathcal{I},\mathcal{R}\}$ till convergent. This step is essential to learn a good sentiment encoder whose output will be used to inform the explanation generator. Third, freeze the recommender and train the generator on $\{\mathcal{U},\mathcal{I},\mathcal{A},\mathcal{X}\}$ with negative log-likelihood loss only. We found in our experiments that generation learning was more difficult than recommendation learning. First training the explanation generator separately can help align the training of both modules later. Fourth, after the separate training converges, start joint training of the recommender and explanation generator. This step allows the model to align the sentiment representation from both modules. At last, freeze the recommender, and turn on the sentiment regularizer to further improve the explanation generator. At this stage, the explanation discriminator and generator are trained in turn.

We evaluate the recommendation quality both in terms of rating prediction (by RMSE and MAE) and item ranking performance (by NDCG@{3,5,10} (Järvelin and Kekäläinen, 2017)). The results are shown in Table 3. SAER demonstrates better performance in all metrics on both datasets. In particular, thanks to the introduced hinge loss for pairwise ranking, SAER demonstrates improved ranking performance against all baselines, which only modeled recommendation as a rating prediction task. The performance difference among NCF, NRT and SAER is worth noting. Although their rating prediction modules all use MLP, NRT and SAER additionally leverage the content information for improved recommendation quality. Improvements from SAER against NARRE and NRT demonstrate that our sentiment vector and corresponding soft gate design better distill and exploit review data for joint learning. Again, as our focus in this work is not on improving recommendation quality, but more on explanation. Next, we will dive into our extensive evaluations about the generated explanation.

We evaluate the quality of our generated explanations from three perspectives: text quality, attribute personalization, and sentiment alignment. We introduce two variants of our model to better analyze the effects of our sentiment regularizer and constrained decoding strategy. 1) SAER (topk), it removes sentiment regularization and decodes by top-k sampling, such that sentiment alignment is only introduced by the soft gates, without the alignment loss, nor the constrained decoding; 2) SAER (reg + topk), it uses sentiment regularization (i.e., the alignment loss) and decodes by top-k sampling, such that sentiment alignment is only enforced at training time.