Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning

We investigated a few summarization datasets. As shown in Table 1, different datasets consider different evaluation dimensions. We observed that these dimensions can be roughly divided into three classes: the semantic quality (Semantic), the linguistic quality (Linguistic), and other dimensions that can be hardly classified (Else). In this paper, we design our method to cover both dimensions of semantic quality and linguistic quality.

To better evaluate the semantic quality, we utilize the contextualized embeddings of BERT Devlin et al. (2019). BERT takes in a sequence which always starts with a special classification token [CLS] as input, and outputs the representation of this sequence. Each token has its own hidden state. The hidden state corresponding to [CLS] is supposed to aggregate information from the whole sequence. We design our evaluation model as follows.

Formally, let $S_{x}$ and $S_{d}$ be the sequence of tokens in the summary $x$ and the source document $d$ from which $x$ is generated. A sequence of tokens is encoded into a sequence of token embeddings $H$ by the BERT encoder.

$$H_{x}=\text{BERT}(S_{x})$$

(1)

$$H_{d}=\text{BERT}(S_{d})$$

(2)

In order to avoid the requirement of a reference summary, similar to Sun and Nenkova (2019), we measure the semantic quality of the target summary $x$ by calculating the semantic similarity between $x$ and its source document $d$. Thus the semantic quality score is:

$$S\_Score(x)=Sim(H_{d}^{0},H_{x}^{0}),$$

(3)

where $Sim$ refers to cosine similarity, $H^{0}$ denotes the hidden state corresponding to token[CLS].

For a summary $x$ and its sequence of tokens $S_{x}$, the exact operations to obtain its linguistic quality score are as follows.

We first use the BERT encoder to get the representation of the summary $x$.

$$H_{x}=\text{BERT}(S_{x}),$$

(4)

where $H_{x}\in\mathbb{R}^{N\times K}$, $N$ is the sequence length and $K$ means the hidden size of the BERT encoder. Then we calculate the probability of the sequence based on this representation.

$$P_{x}=\text{softmax}\bigg{(}W_{1}^{\top}\big{(}\sigma(W_{0}^{\top}H_{x})\big{)})\bigg{)},$$

(5)

where $W_{0}\in\mathbb{R}^{K\times K}$ and $W_{1}\in\mathbb{R}^{K\times V}$ denotes two weight parameters and we omit biases here. $V$ stands for the vocabulary size. $\sigma$ is an activation function, which is GELU in our experiments. A softmax operation is applied to every token’s embeddings to predict a probability distribution at each position in the sequence. Here we use $p_{x}^{i}$ to represent the probability of the $i$-th token to be the same as $S_{x}^{i}$. Motivated by the perplexity, the linguistic quality of $x$ can be calculated as:

$$L\_Score(x)=\frac{1}{|x|}\sum_{i}^{n}{log~{}p_{x}^{i}}$$

(6)

In order to capture both the linguistic and semantic aspects, we develop our final metric by linearly combining the S_Score and L_Score. We call it LS_Score, which is a trade-off between the semantic score and linguistic score.

$\displaystyle LS\_Score(x)=\alpha L\_Score(x)+\beta S\_Score(x)$

(7)

The $\alpha$ and $\beta$ are used to scale the L_Score and the S_Score. In our experiments we fix $\alpha=0.01$ and $\beta=1$ to scale the L_Score and the S_Score.

To alleviate the requirement of reference summaries as well as given human evaluation scores, we develop a new unsupervised training framework via contrastive learning. Intuitively, for a given good summary, if we make some noise, e.g. disordering the words/sentences, we can easily create a summary with worse quality. Then we can compare these two summaries to get a contrastive loss. In practice, we can use human generated summaries in the training data as the ”good” summaries, however, they can also be replaced with other good machine-generated summaries. We do not require any reference summaries in the test phase, i.e. for a candidate summary without known reference summaries we can also predict a score for it. That increases the flexibility and generalizability of our evaluation method.

Given a base summary $r$, assume we make some noise and get a set of negative samples $\hat{X_{r}}$, we formulate a ranking loss function as follows:

$\displaystyle Loss=\sum_{r\in\mathcal{R}}\sum_{\hat{x}\in\hat{X}_{r}}max(0,1-(LS\_Score(r)-LS\_Score(\hat{x})))$

(8)

where $\mathcal{R}$ denotes the set of original summaries in the training set and $\hat{X}_{r}$ is the set of corresponding noisy variants of a training sample $r$. For a batch of $(r,\hat{X_{r}})$, we obtain their scores predicted by an evaluation model and then update model parameters (including fine-tuning BERT) using the gradients of the loss function . In this way, we train the model to better distinguish between good and bad summaries.

Since we evaluate the summaries from two different aspects, for each aspect we create different types of noisy samples. For semantic quality, one straightforward strategy is to randomly remove some words or sentences in the original summary to get a new negative sample. Obviously the created new summary will encounter information loss compared to the original one, so its evaluator score will be lower. In our experiments, we randomly select 20% words (with no consideration of word types) to delete. We do not delete entire sentences because most of the summaries have only very few sentences (as shown in Table 3, the average number of sentences in a reference is 1.43 and 3.88 in Newsroom and CNN/Daily Mail, respectively), thus deleting sentences will cause too much information loss, which doesn’t benefit the model’s ability to distinguish good from bad.

In addition, we do not want the generated summaries to have too much redundant information. So we create another type of negative samples by adding redundant sentences. The redundant sentences are chosen randomly from the original document. Firstly we extract sentences from the original document. Then we filter out the sentences that are most similar to each sentence in the reference. At last, we randomly sample the redundant sentences from the remaining sentences in the reference.

For linguistic quality, the negative samples can be generated by either disordering the words/sentences or deleting words. Both of the operations will lead to loss of coherence or fluency. So the negative sampling strategy in this case is as follows: 1) randomly rotating the order of sentences or the order of words within a sentence. 2) randomly deleting some of the words in the original summary. Note that the second strategy is also used in generating noisy samples for semantic quality, but our LS_Score combines both semantic and linguistic quality, so we do not explicitly discriminate the two aspects for this type of negative samples.

Table 2 shows three examples of our negative samples, each of which represents one type of negative samples respectively. By differentiating the original summaries and the negative samples we enforce our evaluator to capture various aspects of the summary quality. The trained evaluator can then be used for evaluating summaries with unknown references. In our experiments, we generate only one negative sample per type of operations for each base summary, i.e. each base summary has 3 negative samples.

The encoder in our experiments to convert token sequence into embeddings is BERT Devlin et al. (2019). We simply use a pretrained BERT model bert-base-uncase which has 12 layers, a hidden size of 768, 12 attention heads and 110M parameters in total.¹¹1https://github.com/google-research/bert Our model is implemented based on the HuggingFace Transformers.²²2https://github.com/huggingface/transformers The max length of sequence we use for BERT encoding is 512, so we truncate the sequence longer than 510 tokens (despite the special tokens [CLS] and [SEP]).³³3Our code is publicly available at https://github.com/whl97/LS-Score.git.

We conduct empirical studies on two benchmark single-document summarization datasets. These datasets both have original documents, their corresponding human-authored summaries (i.e. references) and also some model-generated summaries that are manually rated in several dimensions, so we can compare different evaluation methods by their correlation with human ratings.

Newsroom. Proposed by Grusky et al. (2018), this summarization dataset includes 1.3 million documents and human-written summaries. In this corpus, there are only 420 summaries with human ratings. These summaries are generated by 7 different extractive or abstractive summarization systems. Each document-summary pair is evaluated by three human raters in four dimensions (coherence, fluency, informativeness, and relevance). We take the mean score of three raters as the groundtruth human score for each summary. We use these summaries with human ratings as our test data. In order to prevent information leakage in the training process, we select our training data (108,802 document-reference pairs) with no overlapped reference summaries with the test data. It means we do not use any reference summaries in our test data for training. The data statistics are shown in Table 3.

CNN/Daily Mail. This dataset was first proposed by Hermann et al. (2015) using news documents for question answering research and was subsequently extended to the area of summarization by Nallapati et al. (2016). Chaganty et al. (2018) provided human scores for 2,513 references and system-generated summaries in three dimensions (overall, fluency and redundancy). We use 1,996 summaries generated by 4 systems for testing and 10,932 document-reference pairs for training. Similarly, there is no overlap of reference summaries between the training data and test data. Table 3 shows the data statistics of the training data.

For both datasets, in the training data, we randomly selected 95% of sentence-pairs for training and the remaining 5% for validation.

We adopt the following metrics as our baselines. Since this paper focuses on unsupervised approaches, we do not compare with the metrics training with human ratings.

ROUGE. This metric has been the most frequently used automatic metric for summarization evaluation. It evaluates the quality of a summary by comparing it to a human-authored reference. The essence of the comparison is to measure the overlapping units (such as n-gram or word sequences) between the summary and the reference Lin and Och (2004).

METEOR. Proposed by Banerjee and Lavie (2005), this metric evaluates a candidate string by measuring the harmonic mean of unigram-precision and unigram-recall between the candidate string and a reference string.

BERTScore. This metric was proposed by Zhang et al. (2020), it utilizes token-level contextual embeddings generated by a pretrained language model (here we use BERT). The evaluation score is calculated by computing similarity between the embeddings of the summary to evaluate and the reference. The BERTScore includes three metrics $R$ (recall), $P$ (precision)and $F$ (F1 score).

WMS/SMS/S+WMS. Kusner et al. (2015) proposed word mover’s distance (WMD) to calculate the minimum cost of moving a sequence into the other. They treat each sequence as a bag of words and each word is represented by its word embeddings. The WMD can then be transformed into a similarity (WMS) Clark et al. (2019). On the basis of WMS, Clark et al. (2019) (2019) designed to measure the similarity of two sequences by calculating sentence mover’s distance to enhance the ability of evaluating multi-sentence texts. They introduced two metrics: sentence mover’s distance (SMS) and sentence and word mover’s distance (S+WMS). SMS uses sentence embeddings instead of word embeddings and represents each sequence as a bag of sentences and S+WMS combines both sentence and word embeddings and represents each sequence as a bag of both sentences and words.

MoverScore. Also inspired by WMD, Zhao et al. (2019) represented both the reference and the candidate text as a sequence of n-gram embeddings and calculate the WMD between two sequences. We report the result of the best models described in their paper that use a BERT pretrained on MNLI dataset to generate the n-gram embeddings and PMeans as the aggregator.

BERT+Cos+Ref. This metric uses BERT as the encoder and calculates the cosine similarity between the embeddings of the reference and the candidate summary.

BERT+Cos+Doc. This metric is similar to BERT+Cos+Ref, but it measures the similarity between the source document and the candidate summary. This is the only reference-free metric in the baselines.

The usual practice of evaluating a summarization evaluation metric is to measure its average summary-level correlation with human judgements, i.e. to measure the correlation between the predicted scores and the human scores across all the test summaries. We evaluate our methods on the aforementioned two datasets. We implemented our final model ( LS_Score with contrastive learning), as we introduced in 3.3. For each dataset, we train our models on the document-reference pairs in the training data, and test on the machine-generated summaries without comparing with reference summaries.