Generating Diversified Comments via Reader-Aware Topic Modeling and
Saliency Detection

To begin with, we state the problem of news comment generation as follows: given a news title $T=\{t_{1},t_{2},\ldots,t_{m}\}$ and a news body $B=\{b_{1},b_{2},\ldots,b_{n}\}$, the model needs to generate a comment $Y=\{y_{1},y_{2},\ldots,y_{l}\}$ by maximizing the conditional probability $p(Y|X)$, where $X=[T,B]$. As shown in Figure 2, the backbone of our work is a sequence-to-sequence framework with attention mechanism (Bahdanau, Cho, and Bengio 2014). Two new components of reader-aware topic modeling and saliency information detection are designed and incorporated for better comment generation. For reader-aware topic modeling, we design a variational generative clustering algorithm for latent semantic learning and topic mining from reader comments. For reader-aware saliency information detection, we introduce a saliency detection component to conduct the Bernoulli distribution estimating on news content. Finally, the obtained topic vectors as well as the selected saliency information are incorporated into the decoder to conduct comment generation.

The backbone of our work is a sequence-to-sequence framework with attention mechanism (Bahdanau, Cho, and Bengio 2014). A BiLSTM encoder is used to encode the input content words $X$ into vectors. Then a LSTM-based decoder generates a comment $Y$ conditioning on a weighted content vector computed by attention mechanism. Precisely, a word embedding matrix is used to convert content words into embedding vectors. Then these embedding vectors are fed into encoder to compute the forward hidden vectors via:

$$\overrightarrow{\mathbf{h}_{i}}=LSTM_{f}^{e}\left(\mathbf{x}_{i},\overrightarrow{\mathbf{h}}_{i-1}\right),$$

(1)

where $\overrightarrow{\mathbf{h}_{i}}$ is a $d$-dimension hidden vector and $LSTM^{e}_{f}$ denotes the LSTM unit. The reversed sequence is fed into $LSTM^{e}_{b}$ to get the backward hidden vectors. We concatenate them to get the final hidden representations of content words. And the representation of news is : $\mathbf{h}^{e}=\left[\overrightarrow{\mathbf{h}_{|X|}};\overleftarrow{\mathbf{h}_{0}}\right]$.

The state of decoder is initialized with $\mathbf{h}^{e}$. For predicting comment word $y_{t}$, the hidden state is first obtained by:

$$\mathbf{h}_{t}=LSTM_{d}\left(\mathbf{y}_{t-1},\mathbf{h}_{t-1}\right),$$

(2)

where $h_{t}\in\mathbb{R}^{2d}$ is the hidden vector of comment word and $LSTM_{d}$ is the LSTM unit. Then we use attention mechanism to query the content information from source input. The weight of each content word is computed as follows:

	$\displaystyle e_{ti}$	$\displaystyle=\mathbf{h}_{t}^{\top}W_{a}\mathbf{h}^{e}_{i},$		(3)
	$\displaystyle\alpha_{ti}$	$\displaystyle=\frac{\exp\left(e_{ti}\right)}{\sum_{i{\prime=1}}^{|X|}\exp\left(e_{ti{\prime}}\right)},$

where $W_{a}\in\mathbb{R}^{2d\times 2d}$, $\mathbf{h}^{e}_{i}$ is the hidden vector of content word $i$, $\alpha_{ti}$ is the normalized attention score on $x_{i}$ at time step $t$. Then the attention-based content vector is obtained by: $\tilde{\mathbf{h}}^{e}_{t}=\sum_{i}\alpha_{ti}\mathbf{h}^{e}_{i}$. The hidden state vector is updated by:

$$\tilde{\mathbf{h}}_{t}=\operatorname{tanh}(W_{c}[\mathbf{h}_{t};\tilde{\mathbf{h}}^{e}_{t}]).$$

(4)

Finally, the probability of the next word $y_{t}$ is computed via:

$$p\left(y_{t}|y_{t-1},X\right)=\operatorname{softmax}\left(linear\left(\tilde{\mathbf{h}}_{t}\right)\right),$$

(5)

where $linear(\cdot)$ is a linear transformation function.

During training, cross-entropy loss $\mathcal{L}_{ce}$ is employed as the optimization objective.

Reader-aware topic modeling is conducted on all the comment sentences, aiming to learn the reader-aware topic representations only from the comments. To achieve this goal, we design a variational generative clustering algorithm which can be trained jointly with the whole framework in an end-to-end manner.

Since this component is a generative model, thus for each comment sentence $Y$ (we employ Bag-of-Words feature vector $\mathbf{y}$ to represent each comment sentence), the generation process is: (1) A topic $c$ is generated from the prior topic categorical distribution $p(c)$; (2) A latent semantic vector $\mathbf{z}$ is generated conditionally from a Gaussian distribution $p(\mathbf{z}|c)$; (3) $\mathbf{y}$ is generated from the conditional distribution $p(\mathbf{y}|\mathbf{z})$. According to the generative process above, the joint probability $p(\mathbf{y},\mathbf{z},c)$ can be factorized as:

$$p(\mathbf{y},\mathbf{z},c)=p(\mathbf{y}|\mathbf{z})p(\mathbf{z}|c)p(c).$$

(6)

By using Jensen’s inequality, the log-likelihood can be written as:

	$\displaystyle\log p(\mathbf{y})$	$\displaystyle=\log\int_{\mathbf{z}}\sum_{c}p(\mathbf{y},\mathbf{z},c)d\mathbf{z}$		(7)
		$\displaystyle\geq E_{q(\mathbf{z},c|\mathbf{y})}\left[\log\frac{p(\mathbf{y},\mathbf{z},c)}{q(\mathbf{z},c|\mathbf{y})}\right]=\mathcal{L}_{\mathrm{ELBO}}(\mathbf{y}),$

where $\mathcal{L}_{\mathrm{ELBO}}$ is the evidence lower bound (ELBO), $q(\mathbf{z},c|\mathbf{y})$ is the variational posterior to approximate the true posterior $p(\mathbf{z},c|\mathbf{y})$ and can be factorized as follows:

$$q(\mathbf{z},c|\mathbf{y})=q(\mathbf{z}|\mathbf{y})q(c|\mathbf{z}).$$

(8)

Then based on Equation 6 and Equation 8, the $\mathcal{L}_{\mathrm{ELBO}}$ can be rewritten as:

	$\displaystyle\mathcal{L}_{\mathrm{ELBO}}(\mathbf{y})=$	$\displaystyle E_{q(\mathbf{z},c|\mathbf{y})}\left[\log\frac{p(\mathbf{y},\mathbf{z},c)}{q(\mathbf{z},c|\mathbf{y})}\right]$		(9)
	$\displaystyle=$	$\displaystyle E_{q(\mathbf{z},c|\mathbf{y})}[\log p(\mathbf{y},\mathbf{z},c)-\log q(\mathbf{z},c|\mathbf{y})]$
	$\displaystyle=$	$\displaystyle E_{q(\mathbf{z},c|\mathbf{y})}[\log p(\mathbf{y}|\mathbf{z})+\log p(\mathbf{z}|c)$
		$\displaystyle+\log p(c)-\log q(\mathbf{z}|\mathbf{y})-\log q(c|\mathbf{z})]$
	$\displaystyle=$	$\displaystyle E_{q(\mathbf{z},c|\mathbf{y})}[\log p(\mathbf{y}|\mathbf{z})$
		$\displaystyle-\log\frac{q(\mathbf{z}|\mathbf{y})}{p(\mathbf{z}|c)}-\log\frac{q(c|\mathbf{z})}{p(c)}]$
	$\displaystyle=$	$\displaystyle E_{q(\mathbf{z}|\mathbf{y})}[\log p(\mathbf{y}|\mathbf{z})$
		$\displaystyle-\sum_{c}q(c|\mathbf{z})\log\frac{q(\mathbf{z}|\mathbf{y})}{p(\mathbf{z}|c)}$
		$\displaystyle-D_{KL}(q(c|\mathbf{z})||p(c))]$

where the first term in Equation 9 is the reconstruction term, which encourages the model to reconstruct the input. The second term aligns the latent vector $\mathbf{z}$ of input $\mathbf{y}$ to the latent topic representation corresponding to topic $c$. $q(c|\mathbf{z})$ can be regarded as the clustering result for the input comment sentence $Y$. The last term is used to narrow the distance between posterior topic distribution $q(c|\mathbf{z})$ and prior topic distribution $p(c)$.

In practical implementations, the prior topic categorical distribution $p(c)$ is set to uniform distribution $p(c)=\frac{1}{K}$ to prevent the posterior topic distribution $q(c|\mathbf{z})$ from collapsing, that is, all comments are clustered into one topic. $p(\mathbf{z}|c)$ is a parameterised diagonal Gaussian as follows:

$$p(\mathbf{z}|c)=\mathcal{N}\left(\mathbf{z}|\boldsymbol{\mu_{c}},\operatorname{diag}\left(\boldsymbol{1}\right)\right),$$

(10)

where $\boldsymbol{\mu_{c}}$ is mean of Gaussian for topic $c$, which is also used as the latent topic representation of topic $c$. Inspired by VAE (Kingma and Welling 2014; Bowman et al. 2016), we employ a parameterised diagonal Gaussian as $q(\mathbf{z}|\mathbf{y})$:

	$\displaystyle\boldsymbol{\mu}$	$\displaystyle=l_{1}(\mathbf{h}),\log\boldsymbol{\sigma}=l_{2}(\mathbf{h}),$		(11)
	$\displaystyle q(\mathbf{z}|\mathbf{y})$	$\displaystyle=\mathcal{N}\left(\mathbf{z}|\boldsymbol{\mu},\operatorname{diag}\left(\boldsymbol{\sigma}^{2}\right)\right),$

where $l_{1}(\cdot)$ and $l_{2}(\cdot)$ are linear transformations, $\mathbf{h}$ is obtained by the comment encoder, which contains two MLP layers with $\operatorname{tanh}$ activation functions. In addition, a classifier with two MLP layers is used to predict topic distribution $q(c|\mathbf{z})$. $p(\mathbf{y}|\mathbf{z})$ is modeled by the decoder, which is a one-layer MLP with $\operatorname{softmax}$ activation function.

After training, $K$ reader-aware topic representation vectors $\{\boldsymbol{\mu}_{i}\}_{1}^{K}$ are obtained only from the whole reader comments corpus in the training set. And these reader-aware topics can be used to control the topic diversity of the generated comments.

Reader-aware saliency detection component is designed to select the most important and reader-interested information from the news article. It conducts Bernoulli distribution estimating on each word of the news content, which indicates whether each content word is important or not. Then we can preserve the selected important words for comment generation.

Specifically, the saliency detection component first uses a BiLSTM encoder to encode title words and the last hidden vectors of two directions are used as the title representation $\mathbf{h}^{te}$. Then we use two-layer MLP with $\operatorname{sigmoid}$ activation function on the final layer to predict the selection probability for each content word $x_{i}$ as follows jointly considering the title information:

$$p_{\theta}(\beta_{i}|x_{i})=\operatorname{MLP}(\mathbf{h}^{e}_{i},\mathbf{h}^{te}),$$

(12)

where $\mathbf{h}^{e}_{i}$ is the hidden vector obtained by BiLSTM-based news content encoder in Section The Backbone Framework 2.2. The probability $\beta_{i}$ determines the probability (saliency) of the word be selected, and it is used to parameterize a Bernoulli distribution. Then a binary gate for each word can be obtained by sampling from the Bernoulli distribution:

$$g_{i}\sim Bernoulli\left(\beta_{i}\right)$$

(13)

Content words with $g_{i}=1$ are selected as context information to conduct the comment generation. Thus, the weight of each content word in attention mechanism and the weighted source context in the backbone framework in Section The Backbone Framework 2.2 are changed as follows:

	$\displaystyle\hat{\alpha}_{ti}$	$\displaystyle=\frac{g_{i}\odot\exp(e_{ti})}{\sum_{i\prime=1}^{|X|}g_{i{\prime}}\odot\exp(e_{ti^{\prime}})},$		(14)
	$\displaystyle\tilde{\mathbf{h}}^{e}_{t}$	$\displaystyle=\sum_{i}\hat{\alpha}_{ti}\mathbf{h}^{e}_{i},$

where $\tilde{\mathbf{h}}^{e}_{t}$ is the selected saliency information of news content and it will be used for comment generation.

However, the sampling operation in Equation 13 is not differentiable. In order to train the reader-aware saliency detection component in an end-to-end manner, we apply Gumbel-Softmax distribution as a surrogate of Bernoulli distribution for each word selection gate (Xue, Li, and Zhang 2019). Specifically, the selection gate produces a two-element one-hot vector as follows:

$$\begin{array}[]{c}\mathbf{g}_{i}=\text{one\_hot}\left(\arg\max_{j}p_{i,j},j=0,1\right)\\ p_{i,0}=1-\beta_{i},p_{i,1}=\beta_{i}\end{array}$$

(15)

we use the Gumbel-Softmax distribution to approximate to the one-hot vector $\mathbf{g}_{i}$:

$\displaystyle\begin{array}[]{c}\hat{\mathbf{g}}_{i}=\left[\hat{p}_{i,j}\right]_{j=0,1}\\ \hat{p}_{i,j}=\frac{\exp\left(\left(\log\left(p_{i,j}\right)+\epsilon_{j}\right)/\tau\right)}{\sum_{j\prime=0}^{1}\exp\left(\left(\log\left(p_{i,j\prime}\right)+\epsilon_{j\prime}\right)/\tau\right)},\end{array}$

(18)

where $\epsilon_{j}$ is a random sample from Gumbel(0, 1). When temperature $\tau$ approaches 0, Gumbel-Softmax distribution approaches one-hot. And now we use $g_{i}=\hat{\mathbf{g}}_{i,0}$ instead of Equation 13. Via this approximation, we can train the component end-to-end with other modules. In order to encourage the saliency detection component to turn off more gates and select less words, a $l_{1}$ norm term over all gates is added to the loss function as follows:

$$\mathcal{L}_{sal}=\frac{\|\mathcal{G}\|_{1}}{|X|}=\frac{\sum_{i}g_{i}}{|X|}.$$

(19)

Given the learned $K$ reader-aware topic vectors, we need to select an appropriate topic for current article to guide the comment generation. Therefore, a two-layers MLP with $\operatorname{softmax}$ activation function is used to predict the selection probability of each topic as follows:

$$p(c|X)=MLP(\mathbf{h}^{e}).$$

(20)

During training, the true topic distribution $q(c|\mathbf{z})$ (Equation 8) is available and is used to compute a weighted topic representation by:

$$\tilde{\boldsymbol{\mu}}=\sum_{c}^{K}q(c|\mathbf{z})\odot\boldsymbol{\mu_{c}}.$$

(21)

After getting the topic vector $\tilde{\boldsymbol{\mu}}$ and the selected saliency information $\tilde{\mathbf{h}}^{e}_{t}$, we update the hidden vector of the backbone decoder in Section The Backbone Framework 2.2 as follows:

$$\tilde{\mathbf{h}}_{t}=\operatorname{tanh}(W_{c}[\mathbf{h}_{t};\tilde{\mathbf{h}}_{t}^{e};\tilde{\boldsymbol{\mu}}]).$$

(22)

Then $\tilde{\mathbf{h}}_{t}$ is used to predict next word as Equation 5.

In the inference stage, $p(c|X)$ is used to get the topic representation. So in order to learn $p(c|X)$ during the training stage, a KL term $\mathcal{L}_{top}=D_{KL}(q(c|\mathbf{z})||p(c|X))$ is added into the final loss function.

Finally, considering all components the loss function of the whole comment generation framework is as follows:

$$\mathcal{L}={\lambda_{1}\mathcal{L}_{\mathrm{ELBO}}}+\lambda_{2}\mathcal{L}_{sal}+\lambda_{3}\mathcal{L}_{ce}+\lambda_{4}\mathcal{L}_{top}.$$

(23)

where $\lambda_{1},\lambda_{2},\lambda_{3}$, and $\lambda_{4}$ are hyperparameters to make a trade-off among different components. Then we jointly train all components according to Equation 23.

Table 1 summarizes the statistics of the three datasets.

The following models are selected as baselines:

Seq2seq (Qin et al. 2018): this model follows the framework of seq2seq model with attention. We use two kinds of input, the title(T) and the title together with the content (TC).

GANN (Zheng et al. 2017): the author proposes a gated attention neural network, which is similar to Seq2seq-T and adds a gate layer between encoder and decoder.

Self-attention (Chen et al. 2018): this model also follows seq2seq framework and use multi-layer self multi-head attention as the encoder and a RNN decoder with attention is applied. We follow the setting of Li et al.(2019) and use the bag of words as input. Specifically the words with top 600 term frequency are as the input.

CVAE (Zhao, Zhao, and Eskenazi 2017): this model uses conditional VAE to improve the diversity of neural dialog. We use this model as a baseline for evaluating the diversity of comments.

For each dataset, we use a vocabulary with the top 30k frequent words in the entire data. We limit maximum lengths of news titles, news bodies and comments to 30, 600 and 50 respectively. The part exceeding the maximum length is truncated. The embedding size is set to 256. The word embedding are shared between encoder and decoder. For RNN based encoder, we use a two-layer BiLSTM with hidden size 128. We use a two-layer LSTM with hidden size 256 as decoder. For self multi-head attention encoder, we use 4 heads and two layers. For CVAE and our topic modeling component, we set the size of latent variable to 64. For our method, $\lambda_{1}$ and $\lambda_{3}$ are set to 1, $\lambda_{2}$ and $\lambda_{4}$ are set to $0.5\times 10^{-3}$ and 0.2 respectively. We choose topic number $K$ from set [10, 100, 1000], and we set $K=100$ for Tencent dataset and $K=1000$ for other two datasets. The dropout layer is inserted after LSTM layers of decoder and the dropout rate is set to 0.1 for regularization. The batch size is set to 128. We train the model using Adam (Kingma and Ba 2014) with learning rate 0.0005. We also clamp gradient values into the range $[-8.0,8.0]$ to avoid the exploding gradient problem (Pascanu, Mikolov, and Bengio 2013). In decoding, top 1 comment from beam search with a size of 5 is selected for evaluation.

Automatic evaluation results on three datasets are shown in Table 2. On most automatic metrics, our method outperforms baseline methods. Compared with Seq2seq-TC, Seq2seq-T and GANN perform worse in all metrics. This indicates that news bodies are important for generating better comments. The results of Self-attention and CVAE are not stable. Compared with Seq2seq-TC, Self-attention performs worse in Yahoo dataset and close in other datasets. CVAE performs better in Tencent dataset and worse in other dastasets compared with Seq2seq-TC. Compared with other methods, our method improves Distinct-4 and M-Distinct scores significantly. This demonstrates that our method can generate diversified comments according to different topics and salient information. While different comments of one article of Seq2seq-T, Seq2seq-TC, GANN and Self-attention come from the same beam, the M-Distinct scores of these methods are lower than ours. Although CVAE can generate different comments for one article by sampling on a latent variable, it obtains a worse M-Distinct score than ours. This demonstrates that the semantic change of generated comments is small when sampling on a latent variable. Our method generates comments by selecting different topic representation vectors and salient information of news, thus has a higher M-Distinct score.

Table 3 reports human evaluation results in three datasets. Because Seq2seq-T and GANN are not using news bodies and perform worse in automatic metrics, we compare our method with other methods. Our method achieves the best Total scores in three datasets. Specifically, our method mainly improves scores on Relevance and Informativeness. This shows that our method can generate more relevant and informative comments by utilizing reader-aware topic modeling and saliency information detection. However, our method performs worse in term of Fluency. We find that baselines tend to generate more generic responses, such as “Me too.”, resulting in higher Fluency scores.

We conduct ablation study to evaluate the affection of each component and show the results in Table 4. We compare our full model with two variants: (1) No Topic Modeling: the reader-aware topic modeling component is removed; (2) No Saliency Detection: the reader-aware saliency detection is removed. We can see that our full model obtains the best performance and two components contribute to the performance. No topic modeling drops a lot in Distinct and M-Distinct. This shows that the reader-aware topic modeling component is important for generating diversified comments. With saliency detection, the performance gets better and this indicates that it is useful to detect important information of news for generating diversified comments.

In order to evaluate the reader-aware topic modeling component, we visualize the learned latent semantic vectors of comments. To this end, we use t-SNE (Maaten and Hinton 2008) to reduce the dimensionality of the latent vector z to 2. The one with the highest probability in topic distribution $q(c|\textbf{z})$ is used as the topic of a comment. We first randomly select 10 topics from 100 topics in Tencent dataset and then plot 5000 sampled comments belonging to these 10 topics in Figure 3. Points with different colors belong to different topics. In addition, we plot topic vectors of these 10 topics. We can see that these latent vectors of comments are grouped into several clusters and distributed around corresponding topic vectors. This shows that reader-aware topic modeling component can effectively cluster comments and topic vectors can be used to represent topics well. What’s more, we collect comments on each topic to observe what each topic is talking about. The top 10 frequent words (removing stop words) of some topics are shown in Table 5. We can see that Topic 1 is talking about intensity of emotional expression, such as “a little”, “really”, and “too”. Appearance and Food are discussed in Topic 37 and Topic 62 respectively.

In order to further understand our model, we compare comments generated by different models in Table 6. The news article is about the dressing of a female star. Seq2seq-TC produces a general comment while Self-attention produces an informative comment. However, they can not produce multiple comments for one article. CVAE can achieve this by sampling on a latent variable, but it produces same comments for different samples. Compared to CVAE, our model generates more relevant and diverse comments according to different topics. For example, “It is good-looking in any clothes” comments on the main story of news and mentions detail of news, “wearing clothes”. What’s more, comparing the generated comment conditioning on a specific topic with corresponding topic words in Table 5, we find that the generated comments are consistent with the semantics of the corresponding topics.