Overview of CTC 2021: Chinese Text Correction for Native Speakers

Chinese text correction (CTC) is a challenging task in natural language processing and it has attracted more and more concerns recently. We organized the CTC competition in 2021, with a focus on text errors produced by native Chinese speakers. In particular, our task is defined as to detect various errors in the text from native speakers and return the corrected texts.

The previous research on text errors in Chinese is mainly studied using texts written by Chinese-as-a-second-language (CSL) learners, with a focus on grammatical errors or spelling errors Wu et al. (2013); Yu et al. (2014); Tseng et al. (2015); Zhao et al. (2018); Rao et al. (2020); Wang et al. (2021). However, most of the errors in Chinese learner texts seldom appear in texts written by native speakers. Therefore, CTC 2021 collects texts written by native Chinese speakers from the Internet to evaluate the performance of the trained models. These texts are more complex, and the errors are more diverse, including spelling error, grammatical error, and Chinese semantic error. In addition, the corresponding corrections are more rigorous.

The goal of the task is to develop techniques to automatically detect and correct errors made by native Chinese speakers. We provide large-scale pseudo training data, and we release the validation data in which errors have been annotated by native speakers. Blind testing data is used to evaluate the outputs of the participating teams using a common scoring script and evaluation metric.

A total of 124 teams signed up for the task, 42 of them reached the final, and 20 of them submitted final systems. This overview paper provides detailed descriptions of the task and it is organized as follows. Section 2 gives the task definition. Section 3 presents a detailed introduction of the data sets and our pseudo data construction method. Section 4 provides the evaluation metric and Section 5 reports the results of the participants’ approaches. Conclusions are finally drawn in Section 6.

CTC 2021 evaluates Chinese text correction performance on Internet texts written by native Chinese speakers. The participants should detect and correct errors in given texts written by native Chinese speakers. Each character or punctuation mark occupies 1 spot for counting location. The input instance is given a unique passage number “PID”. If the text contains no spelling errors, the checker should return “PID, -1”. If an input text contains at least one error, the output format is “PID [, location, error type, detect word, correct word] +”, where the symbol “+” indicates there is one or more instance of the predicting element “[, location, error type, incorrect word, correct word]”. “Location” denotes the start location of incorrect word. “Error type” can be divided into three coarse categories and seven fine-grained categories, shown in Table 1. “Incorrect word” and “correct word” respectively denote the continuous incorrect characters and its correct version. Table 2 presents some examples. There are two errors in Ex. 1, the $20^{th}$ character "轮" should be "论", Location “-1” denotes that there is no error in Ex. 2, in Ex. 3 missing a character "供" in location 13, and the $26^{th}$ or $27^{th}$ character "都" is redundant.

This section presents the pseudo training data, validation data, and testing data in our task. The texts used in our task were collected from the Internet, including education, science, technology, and other types of data, the collected texts were written by native speakers. Table 3 shows statistics for the data set.

Table 5 shows the confusion matrix used for performance evaluation. In the matrix, TP (True Positive) is the number of errors that are correctly identified by the checker; FP (False Positive) is the number of wrongly identified errors that are non-existent; TN (True Negative) is the number of sentences without any errors which are correctly identified as such; FN (False Negative) is the number of errors which are not detected.

Correctness is determined at two levels. The error type does not affect the evaluation results.

(1) Detection level: all locations and incorrect characters in a given text should be completely identical with the gold standard. It is noteworthy that there are some errors shown with different location and incorrect character, but the correction text is right, this is also right detection. For example, “PID=0011-3, 26, redundant error, 都, ,” is completely identical with “PID=0011-3, 27, redundant error, 都, ,”.

(2) Correction level: all locations, incorrect characters, and corresponding corrections should be completely identical with the gold standard, or the correction text is completely identical with the gold standard.

The following metrics are measured at both levels with the help of the confusion matrix.

$Precision=TP/(TP+FP)$

$Recall=TP/(TP+FN)$

$F1=2*Precision*Recall/(Precision+Recall)$

Finally, we calculate overall F1 by weighted detection level F1 and correction level F1, defined as:

$F1_{overall}=0.8*F1_{detection}+0.2*F1_{correction}$.

Take for example, testing input with gold standards and the system may output the result shown in Table 6, the evaluation tool will yield the following performance.

Detection-level:

$Pre.=0.6(=3/5)$

$Rec.=0.4286(=3/7)$

$F1=0.5(=2*0.6*0.4286/(0.6+0.4285))$

Correction-level:

$Pre.=0.4(=2/5)$

$Rec.=0.2857(=2/7)$

$F1=0.3333(=2*0.4*0.2857/(0.4+0.2857))$

Overall:

$F1=0.4667(=0.8*0.5+0.2*0.333)$

A total of 124 teams signed up for the task, 42 of them reached the final, and 20 of them submitted final systems. The 20 submitting system teams are all from universities and industries in China. In the official testing phase, each participating team was allowed to submit at most three runs that adopt different models or parameter settings, the highest score of the three times will be the final score of the team.

In total, we had received 36 runs. Of the 20 submission teams, 16 team systems can work normally. Table 7 shows the submission statistics for the 16 work normally teams and their final score. we can see that Chinese text correction for native speakers is a challenging task. There remain large gaps between submitting systems and gold standards. In detail, S&A gets the best detection F1 score of 0.6800 and best correction F1 score of 0.6460.

We collected the top 6 participants’ reports of their systems. Table 8 summarizes their approaches and the usage of linguistic resources for this task. We can observe that most of the participants adopt pre-trained language models (e.g. Devlin et al., 2018; Liu et al., 2019; Clark et al., 2020), sequence tagging (Omelianchuk et al., 2020), multimodal information of the Chinese characters (Xu et al., 2021), sequence to sequence models (Vaswani et al., 2017), and rule-based models. In addition to the CTC 2021 pseudo training data, some public linguistic resources are used by participants, such as Weixin corpus¹¹1https://github.com/nonamestreet/weixin_public_corpus, Wikimedia, SIGHAN, NLPCC2018, Lang-8, and WuDaoCorpora 2.0²²2https://resource.wudaoai.cn/home.

This paper provides an overview of CTC 2021, including task design, data preparation, evaluation metrics, and evaluation results. The final results show that it is still a challenging task which deserves more concern. In order to provide a good communication platform for researchers, industrial practitioners and NLP enthusiasts, we create CTC2021 leaderboard, release the pseudo training data and gold standards validation data. We hope this task can facilitate and expedite future development in this research area.

CTC 2021 was hosted by the Chinese Association for Artificial Intelligence, sponsored by iFLYTEK CO.LTD., organized by the Joint Laboratory of HIT and iFLYTEK Research (HFL).