Intrinsic Knowledge Evaluation on Chinese Language Models

Linguistic knowledge is fundamental to language understanding. To examine the linguistic knowledge encoded in pre-trained LMs, we propose two language probing tasks to address both the syntactic and semantic regularities.

Besides learning linguistic knowledge, tasks like Question Answering (QA) and Reading Comprehension (RC) often require real-world knowledge beyond contexts. We investigate world knowledge in common sense and encyclopedia facts.

In general, our tasks have two forms: (1) multiple-choice questions for semantic knowledge; and (2) cloze questions for function word, commonsense and fact inferences. Unlike existing probing tasks that use additional classifiers, our tasks allow pre-trained LMs to make predictions directly from their contextual representations. This design mitigates influences of the tuning process (Hewitt and Liang, 2019), hence can enable a fairer overall evaluation.

The word sense similarity task uses sentences in CTC corpus²²2http://www.aihanyu.org/basic_v2/index.html, a Chinese textbook corpus with manual sense labeling on polysemous words. To balance the data set, we allow at most $10$ questions per meaning and $50$ questions per word, resulting in $5790$ questions from $372$ words.

For syntactic knowledge, we extract sentences that have function words from CTC corpus as well, in the end created $(3776,10132,5887,5711,3839)$ clozes for type (C, D, P, U, ND) respectively.

Regarding world knowledge, we present common sense using templates and factual triples in natural contexts just as described above. Target items—prefer object$>$relation (if has)$>$subjects—are masked to create cloze questions. Common sense adapts from the Chinese part of ConceptNet (Speer et al., 2017), in which $3111$ pairs from $13$ relation types are usable after professional manual inspection. For facts, we built $1062$ clozes using a Chinese Knowledge Base built upon encyclopedia called CN-DBpedia (Xu et al., 2017).

In these data sets, target words only appear once in their contexts, and sentences within a data set never repeat. We end up obtained four data sets counting to $39,308$ questions in total. Table 1 is a summary of these knowledge data sets. See more details of manual checks and illustrated examples in Appendix B and C. For evaluation results, unless otherwise stated, we report the average score if multiple classes occur³³3Calculate prediction accuracy firstly within each category, then averaged across classes..

Many Chinese language models are publicly available at CLUE (Xu et al., 2020). To reduce computing cost and keep candidates comparable in size, we test the models in BASE-size: BERT (Devlin et al., 2019), BERT-wwm, BERT-wwm-ext, and RoBERTa-wwm-ext (Cui et al., 2019a)⁴⁴4More specified model configurations list in Appendix C.

Also, to compare the performance between training objectives, we implement four variants:

BERT (MLM+NSP): Masked Language Model and Next Sentence Prediction as in BERT.

RoBERTa (MLM): Remove NSP, often benefit from longer sequences and fewer topic conflicts.

SpanBERT (MLM+SBO): Add a span-shrunk version of the Subject Boundary Objective (Joshi et al., 2020). We mask single tokens instead of spans to control variables.

ALBERT (MLM+SOP): Replace NSP with the Sentence Order Prediction (Lan et al., 2019), guide bi-spans on inter-logic rather than topic conformity.

Training from scratch costs, so we initialize them with BERT-base-chinese and train for $500,000$ additional steps at a $32$ batch size using Baidu Baike corpus. Other settings align with BERT.

Table 2 shows the results of off-the-shelf models, on our intrinsic knowledge tasks and four extrinsic NLP tasks. Our three observations read as follows.

First, pre-trained Chinese LMs using natural contexts capture the linguistic and factual knowledge well. However, they do poorly on commonsense questions, probably because they are not fully capable of storing relationally-structured knowledge (Poerner et al., 2019). For another, template-based clozes might be too short triggers for models to yield informative contextual representations. To further verify our hypothesis, we bucket fact clozes based on text length, study the trend depending on lengths, then illustrate in Figure 2.

Second, pre-process with whole-word-masking and pre-train with additional data, consistently enhance the performance in most intrinsic tasks. Removing the NSP objective further helps. RoBERTa-wwm-ext, who integrates all of the three advantages, scores the highest on intrinsic tasks.

Third, by comparing the intrinsic tasks against extrinsic ones, we observe that the intrinsic tasks ring more sensitive to changes in model structures and training data. Among extrinsic tasks, only in Reading Comprehension that models consistently improve with upgraded masking strategy and training corpora. XNLI doesn’t lead to any difference between BERT-wwm-ext and RoBERTa-wwm-ext. The other two classification tasks vary trivially across four models. These results suggest that our intrinsic tasks can reflect model discrepancies in more elaborate ways than extrinsic ones, further, unveil knowledge encoding from various aspects.

Note that these intrinsic knowledge evaluations can also shed light on the structural design of LMs. Since most off-the-shelf models vary diversely (in the corpus, parameters, and more), we focus on a single factor, the training objectives. We implemented four BERT variants and make a preliminary study of their effects on knowledge encoding abilities. As shown in Table 3. MLM (RoBERTa) strikes the best in all knowledge aspects. Boundary information (SBO) barely helps, for it may suit spans better than single tokens. NSP surpasses SOP in most cases, showing a priority of topic conformity at bi-span training.