Extractive and Abstractive Sentence Labelling of Sentiment-bearing Topics

Automatically learning topic labels from data.   In early work, Mei et al. [6] proposed generating topic labels using either bigrams or noun phrases extracted from a corpus and ranked the relevance of candidate labels to a given topic using KL divergence. Candidate labels were scored using relevance functions which minimised the similarity distance between the candidate labels and the topic words. The top-ranked candidate label was then chosen as the topic’s label. Mao et al. [13] labelled hierarchical topics by investigating the sibling and parent-child relations among the topics. Their approach followed a similar paradigm to [6], which ranked candidate labels by measuring the relevance of the labels to a reference corpus using Jensen-Shannon divergence. More recently, Cano et al. [12] proposed an approach to labeling topics based on multi-document summarisation. They measured the term relevance of documents to topics and generated topic label candidates based on the summarisation of a topic’s relevant documents.

To summarise, the topic labelling approaches in the above-mentioned works were engineered for labelling standard LDA topics rather than sentiment-bearing topics, and mostly fall into one of the following linguistic modalities: a single term [11], a small set of terms (e.g., the top-$n$ topic words) [11], or phrases [14, 7]. These linguistic modalities, while sufficient for expressing facet subjects of standard topics, are inadequate to express complete thoughts or opinions due to their linguistic constraints, i.e. lacking a subject, a predicate or both. Our goal is to address this gap and study the problem of automatically labelling sentiment-bearing topics with more appropriate and descriptive sentence labels.

Sentence fusion is a text-to-text generation technique that transforms overlapping information from a cluster of similar sentences into a single sentence. One of the most popular strategies for sentence fusion relies on merging the dependency trees of input sentences to produce a graph representation which will be linearised in a separate stage [20, 21, 22, 23].

Barzilay and McKeown [20] introduced a sentence fusion framework for multi-document summarisation. Their approach provides a means to capture the main information in a cluster of related sentences in comparison to the common method which just selects the sentence closest to the centroid. Approaches to sentence fusion typically use dependency graphs to produce an intermediary syntactic representation of the information in a cluster of sentences [20, 21, 22]. The graph is then linearised in different ways to generate fused sentences, which are then ranked using a language model or language-specific heuristics to filter out ill-formed sentences [24]. This type of approaches of fusing sentences mimic the strategies used by humans as reported in the analysis of human-generated summaries in [25]. The sentence fusion task has since been widened to encompass other variants such as combining two sentences to produce a single sentence that either conveys the common information shared by the two sentences, or all information of sentences but without redundancy [26, 27, 22, 28, 23]. More recent works [29, 30, 31] on sentence fusion consider not only pairs of sentences but also larger sentence clusters using multi-sentence compression, where the compression is performed by selecting the high-scoring paths in a weighted bigram graph [29, 30, 31].

The graphical model of JST is shown in Figure 1, in which $D$ denotes a collection of documents, $N_{d}$ a sequence of words in document $d$, $S$ the number of sentiment labels, $T$ the total number of topics, and $\{\alpha,\beta,\gamma\}$ the Dirichlet hyperparameters (cf. [2] for details). The formal definition of the generative process in JST corresponding to the graphical model shown in Figure 1 is as follows. First, one draws a sentiment label $l$ from the per-document sentiment proportion $\bm{\pi}_{d}$. Following that, one draws a topic label $z$ from the per-document topic proportion $\bm{\theta}_{d,l}$ conditioned on sentiment label $l$. Finally, one draws a word from the per-corpus word distribution $\bm{\varphi}_{l,z}$ conditioned on both the sentiment label $l$ and topic label $z$.

• •

  For each sentiment label $l\in\{1,...,S\}$

  • –

    For each topic $j\in\{1,...,T\}$

    • *

      draw $\bm{\varphi}_{lj}\sim\text{Dir}(\lambda_{l}\cdot\beta^{T}_{lj})$

• •

  For each document $d\in\{1,...,D\},$

  • –

    Choose a distribution $\pi_{d}\sim\text{Dir}(\gamma)$

  • –

    For each sentiment label $l$ under document $d$

    • *

      Choose a distribution $\theta_{d,l}\sim\text{Dir}(\alpha)$

  • –

    For each word $w_{i}$ in document $d$

    • *

      Choose a sentiment label $l_{j}\sim\text{Mult}(\pi_{d})$

    • *

      Choose a topic $z_{i}\sim\text{Mult}(\theta_{d,l_{i}})$

    • *

      Choose a word $w_{i}\sim\text{Mult}(\varphi_{l_{i}z_{i}})$

The problem to be solved by the JST model is the posterior inference of the variables, which determine the hidden sentiment-bearing topic structures that can best explain the observed set of documents. Formally, a sentiment-bearing topic is represented by a multinomial distribution over words denoted as $p(w|l,z)\equiv\varphi_{w,l,z}$, where $w$, $l$, and $z$ represent the word, sentiment label and topic label indices, respectively. In particular, the approximated per-corpus sentiment-topic word distribution is

where $V$ is the corpus vocabulary size, $N_{l,z,w}$ is the number of times word $w$ appeared in topic $z$ with sentiment label $l$, $N_{l,z}$ is the number of times words are assigned to $z$ and $l$. Recalling that $\varphi_{w,l,z}$ is a multinomial distribution, we have $\sum_{w\in V}\varphi_{w,l,z}=1$.

The original JST model can only learn topic–word and topic–document associations as it operates on bag-of-words features at the document-level, with the corpus sentence structure being ignored. Therefore, we propose a new computational mechanism that can uncover the relevance to a sentiment-bearing topic of the underlying sentences in the corpus. To achieve this, we first preserve the sentence structure information for each document during the corpus preprocessing step (see Section 4 for more details). Second, modelling topic-sentence relevance is essentially equivalent to calculating the probability of a sentence given a sentiment-bearing topic $p(\text{sent}|l,z)$, i.e., the likelihood of a sentence (from the corpus) associating with a given sentiment-bearing topic. The posterior inference of JST, based on Gibbs sampling [32, 2], can recover the hidden sentiment label and topic label assignments for each word in the corpus. Such label-word assignment information provides a means to re-assembling the relevance between a word and a sentiment-bearing topic. By leveraging the sentence structure information and gathering the label assignment statistics for each word of a sentence, we can derive the probability of a sentence given a sentiment-bearing topic as

where

Note that the per-corpus sentiment-topic word distribution $\varphi_{w,l,z}$, defined in Eq. 1, is obtained via the posterior inference using Gibbs sampling. Also $p(l,z)$ is discounted as it is a constant when comparing sentencial labels for the same sentiment-bearing topic. A summary of our mechanism of calculating $p(\text{sent}|l,z)$ is given in Algorithm  1.

Given a sentiment-bearing topic, one intuitive approach for selecting the most representative sentence label is to rank the sentences in the corpus according to the topic-sentence relevance probability $p(\text{sent}|l,z)$ derived in the previous section. One can then select the sentence with the highest probability as a label for the topic. We found that a high-degree of relevance, however, cannot be taken as the only criterion for label selection as it might result in the selection of short sentences that do not provide enough information about the topic or that cover either thematic or sentiment information alone. To address these issues and to select the most representative sentence labels covering sentiment-coupled aspects which reveal opinions, it is important to consider the occurrences of both aspect and sentiment, and the balance between them. To this end, we designed a sentence label selection criterion which jointly considers relevance as well as aspect and sentiment co-coverage. This provides the basis for building an algorithm for ranking sentence labels in terms of how well they can describe the opinion encoded in a sentiment-bearing topic.

Given a sentiment-bearing topic, we define the sentence scoring function below:

Here, $Rel(s|t_{l,z})$ is the relevance score between a sentence $s$ and a given sentiment-bearing topic $t_{l,z}$ as described in Section 3.2. $Cov(s|t_{l,z})$, the aspect and sentiment co-coverage score, encodes two heuristics: (i) sentence labels covering either sentiment or aspects information alone will be significantly down-weighted; and (ii) labels which cover salient aspects of the topic coupled with sentiment will be given high weightage. Parameter $\alpha$ sets the relative contributions of relevance and the co-coverage of aspect and sentiment, and was empirically set to 0.4. Let $i$ be a word from sentence $s$, $a_{i}$ a binary variable that indicates whether word $i$ is an aspect word, $o_{i}$ a binary variable that indicates whether word $i$ is a sentiment word, $w_{l,z,i}$ the importance weight of word $i$ given topic $t_{l,z}$, and $t_{l,z,n}^{i}$ a binary variable which indicates the presence of word $i$ in the top-$n$ words of sentiment-bearing topic $t_{l,z}$. Formally, the aspect and sentiment co-coverage score is formulated as

where the weight $w_{l,z,i}$ essentially equals to $\varphi_{l,z,i}$, i.e., the marginal probability of word $i$ given a sentiment-bearing topic $t_{l,z}$. We identify whether word $i$ is an aspect word or a sentiment word as follows. In the preprocessing, we perform parts of speech (POS) tagging on the experimental datasets (detailed in Section 4). Words tagged as nouns are regarded as aspects words whereas sentiment words are the ones that have appeared in the MPQA sentiment lexicon¹¹1http://www.cs.pitt.edu/mpqa/. Recalling that a multinomial topic is represented by its top-$n$ topic words with the highest marginal probability, we further constrain that word $i$ must also appear in the top-$n$ topic words of sentiment-bearing topic $t_{l,z}$. This ensures that a good sentence label should cover as many important sentiment coupled aspects of a topic as possible. In the case where a sentence contains either aspect or sentiment information alone (i.e., $A(s|t_{l,z})=0$ or $S(s|t_{l,z})=0$), the resulting aspect and sentiment co-coverage score would be zero, thus down-weighting the corresponding sentence. Finally, a sentence label $\hat{s}$ for a sentiment-bearing topic $t_{l,z}$ can be obtained by:

Our second type of approach to topic labelling employs sentence fusion techniques for generating abstractive labels. We hypothesise that abstractive sentence labels would be more succinct by removing redundant information of input sentences, while maintaining the essence of sentiment-bearing topics as much as possible. Specifically, we explore two well established sentence fusion algorithms [29, 30] for the fusion task, both of which search for an optimal weighting function in noisy graphs to identify readable and informative compressions.

We evaluated the effectiveness of the sentence label selected by our approach on three real-world datasets. Two datasets are publicly available, i.e., Amazon reviews for kitchen and electronic products, where each dataset contains 1000 positive and 1000 negative reviews²²2https://www.cs.jhu.edu/ mdredze/datasets/sentiment/ [34]. We have also collected a movie review dataset for Internal Affairs and The Departed from IMDb³³3http://www.imdb.com/. These two movies virtually share the same storyline but with different casts and productions: Internal Affairs is a 2002 Hong Kong movie, while The Departed is a 2006 Hollywood movie.

In the JST model implementation, we set the symmetric prior $\beta$=0.01 [32], the symmetric prior $\lambda=(0.05\times L)/S$, where $L$ is the average of the document length, $S$ the is total number of sentiment labels, and the value of 0.05 on average allocates 5 percent of probability mass for mixing. The asymmetric prior $\alpha$ is learned directly from data using maximum-likelihood estimation [35] and updated every 50 iterations during the Gibbs sampling procedure. We empirically set the model with 2 sentiment labels and 10 topics for each sentiment label, resulting 20 sentiment-bearing topics in total for each of the three evaluation datasets. In the preprocessing, we first performed automatic sentence segmentation⁴⁴4http://www.nltk.org/ on the experimental datasets in order to preserve the sentence structure information of each document, followed by parts of speech tagging on the datasets using the Stanford POS tagger [36]. Punctuation, numbers, and non-alphabet characters were then removed and all words were lowercased. Finally, we trained JST models with the preprocessed corpus following the standard procedures described in [2], with the MPQA lexicon [37] being incorporated as prior information for model learning.

• 3

  Very good label, a perfect description of the topic;

• 2

  Reasonable label, but does not completely capture the topic;

• 1

  Label is semantically related to the topic, but would not make a good topic label; and

• 0

  Label is completely inappropriate, and unrelated to the topic.

The most relevant review of the sentiment-bearing topic (calculated using $p(d|l,z)$, the probability of a document given a sentiment-bearing topic) was also provided to annotators so that they can have a better context for topic understanding. To ensure there was no bias introduced in this step, we checked whether the most relevant review contains our sentence label. This only occurred in 7.5% of cases. For the remaining 92.5%, sentence labels appeared in only the second to the fifth most relevant reviews, not shown to the user. This suggest that our experimental setup is not biased towards our system as the most relevant review rarely contains the most representative sentence label.

We recruited 30 Computing Science postgraduate students to participate in the evaluation task. Recalling that 20 sentiment-bearing topics were extracted for each dataset and that there are 5 systems to compare (i.e., our proposed approach and the four baselines) and 3 datasets, the total number of labels to rate is 300. To avoid the evaluation task being overly long, we broke the evaluation task into 3 sessions carried out on different days, with each session evaluating labels from one dataset only. Completing one session took around 18 minutes for each participant on average.

We want to stress that these baselines are fair. To our knowledge, we are the first to study the problem of labelling sentiment-bearing topics, so there is no directly comparable system available which can generate sentence labelling for sentiment-bearing topics. In the past, top-$n$ topic terms have been commonly used for manually interpreting sentiment-bearing topics for a variety of sentiment-topic models (i.e., it’s not just exclusively used for standard topic models), and that is why we propose an automatic labelling approach here. For the phrase label baselines, they can capture a fair amount of sentiment information from sentiment-bearing topics based on the adjective/adverb phrases extracted. The sentence label baselines are also strong baselines as they both make use of the probability distributions from the JST model, with the centroid baseline further addressing the diversity issue.

Our labelling approach also shows better performance than the sentential baselines. For instance, both sentential baselines were unable to adequately interpret the negative sentiment-bearing topics in the Kitchen dataset. For example, the Centroid baseline “I ordered … received only the bottom piece.” captures the thematic information (i.e., product) but fails to capture the sentiment information (e.g., worth). The Top-prob baseline captures negative sentiment information (i.e., disappointed) but does not include any thematic information (e.g., money, product). A user who read the Top-prob baseline (i.e., “Heres why I am sorely disappointed”) is likely to have some confusions in understanding what causes the disappointment in the label, e.g., whether it is due to money or product. Our approach (i.e., Sent-label) also performs better in the Electronics dataset by giving more descriptive information about products. Take the positive sentiment-bearing topics for example, the labels generated by Mei07 and Aletras14 capture two different thematic aspects i.e., sound and headphones, respectively. For the Top-prob label, it captures positive sentiment about sound quality, whereas the Centroid baseline label seems less relevant to the topic. In contrast, our proposed approach shows that the topic conveys the opinion of iPod sound quality being amazing.

To summarise, our experimental results show that our extractive sentence labelling approach outperforms four strong baselines and demonstrates the effectiveness of our sentence labels in facilitating topic understanding and interpretation.

Table 6 shows some examples of extractive (i.e., Sent-label) and abstractive (i.e., PathGraph and Keyphrase) labelling for two sentiment-bearing topics extracted from the Kitchen dataset.

To summarise, our experimental results show that the Keyphrase algorithm outperforms the PathGraph algorithm in terms of both grammaticality and informativity. In addition, the abstractive labels generated based on the compression algorithms are more informative than the extractive label, with some cost of reduced grammaticality.