Enriching the Korean Learner Corpus with Multi-reference Annotations and Rubric-Based Scoring

The KoLLA learner corpus addresses the challenge of annotating and analyzing Korean post-positional particle errors, a major difficulty for learners of Korean as a L2 [15]. These particles, akin to English prepositions, serve multiple functions, including marking grammatical roles (e.g., subject and object), semantic roles (e.g., agent and location), and discourse functions. The agglutinative nature of Korean and the tendency for particles to combine with nominals without whitespace complicate their correct use for learners. The KoLLA corpus annotates essays written by college-level Korean learners, grouped by proficiency (beginner and intermediate) and learner type (heritage and non-heritage). The annotation scheme is multi-layered, addressing spacing and spelling errors to ensure proper segmentation of words before particle errors are classified. Errors are categorized into four types: omission, substitution, addition, and ordering. To determine the correct particles for annotation, the corpus adopts the principle of “minimal interaction,” focusing on grammatical forms expected in formal Korean writing. Additionally, it accounts for cases where multiple particles may be equally valid, providing a set of possible correct forms for robust system evaluation. Developed using 100 learner essays and tools like EXMARaLDA [21], the corpus achieves high inter-annotator agreement rates, validating its guidelines and methodology. By supporting automatic particle error detection, KoLLA aims to advance the analysis of morphologically rich learner languages. The resource also contributes to broader computational linguistic efforts by offering insights into annotation best practices for error detection and language learning applications.

The dataset used in this study is based on the original KoLLA corpus, which consists of 100 essays written by Korean L2 learners. These essays were segmented into 1,419 individual sentences to align with the differing evaluation levels required by AES and GEC systems. While AES operates at the essay level, GEC is typically conducted at the sentence level, thus necessitating this segmentation.

To support multi-reference evaluation for GEC, each of the 1,419 sentences is accompanied by two human-generated corrections. The first correction is sourced from the original KoLLA corpus, while the second was newly created by a trained linguist following detailed annotation guidelines. This dual-reference design enhances the resource’s ability to capture the range of valid corrections—especially important in a morphologically rich language like Korean. For AES, two annotators independently assigned rubric-based scores to all 100 essays, evaluating dimensions such as content, coherence, grammatical accuracy, and vocabulary use.

The present version constitutes a substantial enrichment of the original KoLLA corpus, rather than an entirely new resource. While the learner-produced texts are drawn from the original KoLLA dataset, this version introduces additional multi-reference GEC annotations and double-coded AES scoring, following best practices in learner corpus construction and evaluation. The enriched KoLLA resource thus supports more comprehensive investigations into both grammatical error correction and essay scoring for Korean as a foreign language.

The KoLLA corpus, consisting of open writing assignments from Korean L2 learners, serves as the foundation for our enhanced multi-reference GEC dataset. The original corpus includes 1,419 sentences written by beginner and intermediate-level learners, segmented from essays to facilitate sentence-level GEC annotation, given the difficulty of automatic boundary detection in the presence of learner errors.

To create a multi-reference GEC resource, we augmented the original KoLLA corrections with a second human-generated correction for each sentence. This design follows precedents in English and Chinese GEC corpora (e.g., FCE, MuCGEC), where multiple references are employed to capture the inherent ambiguity and variability of learner errors. For Korean, such variability is even more pronounced due to its morphosyntactic flexibility, honorific system, and case-marking ambiguity. In agglutinative languages, a single correction may fail to represent the full range of valid alternatives, potentially penalizing correct but divergent model outputs.

The second layer of annotation was produced by a trained linguist holding an M.A. in Korean Language Education. This annotator followed detailed guidelines to ensure fluency, preserve the learner’s intended meaning, maintain structural fidelity, and avoid unnecessary rewriting. Although the second set of corrections was provided by a single annotator, quality assurance focused on internal consistency through strict adherence to annotation guidelines.

The resulting dataset thus provides two reference corrections per sentence, offering a more realistic and flexible evaluation framework for Korean GEC. Multi-reference annotation improves the corpus’s ability to capture correction variability and supports more realistic evaluation settings for Korean GEC.

Grammatical error typology provides a principled framework for systematically identifying, classifying, and analyzing learner errors, serving as a foundation for building and evaluating GEC systems. Building on the original MRU framework (Missing, Replacement, and Unnecessary) introduced by errant [1], we extend it for Korean by incorporating language-specific error types and annotation strategies, formalized in Algorithm 1.

A key extension involves the treatment of word boundary errors, which are prevalent in agglutinative languages such as Korean. We introduce two fine-grained categories: M:WB (missing word boundaries) and U:WB (unnecessary word boundaries). M:WB is applied when multiple lexical items are erroneously fused into a single token (typically due to omitted spacing), while U:WB is used when excessive spacing disrupts a morphosyntactically valid token.

Spelling errors are labeled as R:SPELL when a content word is incorrectly realized orthographically or phonologically. We further expand the framework to explicitly address errors involving functional morphemes—especially postpositions (ADP) and verbal endings (PART). If the associated content word is correct, such errors are marked according to their context: a missing postposition is annotated as R:POS_i $\rightarrow$ POS_j+ADP, while an unnecessary postposition is marked as R:POS_i+ADP $\rightarrow$ POS_j. Substitution of one postposition for another is marked as R:POS+ADP_i $\rightarrow$ POS+ADP_j. When the associated content word is itself misspelled or incorrect, the error is subsumed under R:SPELL, and the morpheme is not annotated independently.

To mitigate segmentation mismatches and annotation inconsistencies, preprocessing was applied to normalize common variants in particle use and compound verb boundaries. This step reduced word boundary misalignment errors to under 5%.

Finally, word order errors are annotated as R:ORDER when the source and target sentences contain the same lexical items but differ in sequence, indicating syntactic reordering without lexical substitution.

Algorithm 1 formalizes this annotation logic and accounts for the structural and orthographic complexities characteristic of Korean grammatical errors.

Figure 1 presents annotations from two annotators for the sentence “비행기 음식이 안 막였습니다” (‘The airplane food didn’t …’). The noun phrase 음식이 eumsig-i (‘food.nom’) is replaced with 음식을 eumsig-eul (‘food.acc’), annotated as R:NOUN+ADP $\rightarrow$ NOUN+ADP, reflecting a case marker substitution from nominative to accusative. The verb 막였습니다 magyeossseubnida, a misspelling based on phonetic and orthographic similarity, is corrected by Annotator 0 as 먹었습니다 meogeossseubnida (‘ate’) and annotated as R:SPELL. Annotator 1, however, interprets the intended meaning differently and replaces the verb with 맞았습니다 maj-assseubnida (‘agreed’), also marked as R:SPELL.

This example highlights a central challenge in grammatical error annotation: distinguishing true grammatical errors from alternative, semantically divergent but grammatically valid expressions. While both corrections are linguistically plausible, they lead to different interpretations of the sentence—either I didn’t eat the airplane food or The airplane food didn’t agree with me—underscoring the importance of consistency and clarity in annotation guidelines.

These annotation strategies and enhancements make the KoLLA corpus a robust and linguistically informed resource for Korean GEC. By addressing the structural nuances of the language and capturing correction variability, this enriched dataset offers a valuable foundation for developing and evaluating GEC systems tailored to Korean L2 writing.

The relationship between first language (L1) and L2 writing has been extensively explored by L2 writing researchers [11, 13, 22]. Recent studies have found that although there are differences in L1 and L2 writing in terms of organizational patterns, overall quality, and grammatical errors, there are also significant similarities between L1 and L2 writing [10]. For example, Fathman \BBA Kobayashi [5] discovered that many Japanese learners of English used a specific-to-general organizational pattern (inductive), which contrasts with American L1 students’ general-to-specific writing styles (deductive). However, advanced Japanese ESL students tended to favor the deductive organizational pattern, indicating that L2 writing experience or a certain proficiency level leads to the adoption of organizational patterns preferred in L1 English writing. Similarly, Hirose (2003) compared L1 (Japanese) and L2 (English) organizational patterns in the argumentative writing of Japanese EFL student-writers. The study revealed that a majority of students employed deductive-type organizational patterns in both L1 and L2, challenging the stereotype that Japanese writers naturally favor inductive structures. This finding suggests that with increased proficiency and exposure to L2 writing conventions, learners may adopt organizational patterns aligned with those of native speakers.​Other studies have maintained that not only the quality of writing but also the process of writing might be similar between L1 and L2 writing [3], thus justifying the extensive use of L1 process-oriented models of writing in investigating L2 writing [7, 8].

Following these findings, this study employs a rubric traditionally used to assess Korean L1 speakers’ writing samples for evaluating Korean L2 learners’ writing. The rubric includes three overarching categories—expression, structure, and content—each with multiple subcategories that provide clear, operationalized criteria for assessment. This rubric is widely used in Korean language education settings and was originally developed and refined by experts in L1 writing assessment to ensure clarity, objectivity, and comprehensiveness in evaluating student performance.

The first category, expression, includes three subcategories: grammatical accuracy, appropriateness of vocabulary use, and appropriateness of expressions. Grammatical accuracy evaluates the correctness of sentence structures, ensuring that sentences are grammatically correct. Appropriateness of vocabulary use measures whether the vocabulary used is suitable for the given context and whether there is a variety of vocabulary used throughout the text. Appropriateness of expressions assesses the variety and suitability of expressions and whether the length of sentences is appropriate.

The second category, structure, consists of four subcategories: appropriateness of internal structure, organization of essay structure, coherency of paragraphs, and appropriateness of length. Appropriateness of internal structure measures how cohesively the thesis and supporting sentences are connected. Appropriateness of intra-structure assesses whether the essay includes an introduction, body, and conclusion paragraphs, and whether the length of each paragraph is appropriate. Coherency of paragraphs evaluates the use of conjunctions and transition sentences to ensure smooth flow between ideas. Appropriateness of length checks if the essay meets the required word limit.

The third category, content, includes clarity of topic, detailed explanation, and creativity of thought. Clarity of topic measures whether the topic is clearly presented in the essay and whether each sentence relates to the topic. Detailed explanation assesses whether the explanation is detailed and varied. Creativity of thought examines the uniqueness, logic, and presentation of new and different viewpoints.

Table 1 shows the full rubric used for scoring, including the three main categories—expression, structure, and content—and their respective subcategories and descriptions.

To ensure that the application of this rubric constituted a standardized assessment procedure, we implemented a multi-step annotator training and calibration process. Two annotators with M.A. degrees in Korean Language Education and extensive teaching and rating experience independently assessed student essays using the rubric. Prior to rating, they underwent two training sessions led by the researcher. In the first session, the rubric categories were thoroughly explained and five sample essays from each proficiency level were collaboratively scored. Discrepancies were discussed in depth until consensus was reached, thereby aligning interpretations of the rubric criteria. In the second session, annotators independently scored essays and reconvened with the researcher to resolve any inconsistencies in interpretation through discussion and clarification.

This process mirrors best practices in standardized assessment, particularly in language testing, where inter-annotator reliability, operationalized rubrics, and calibration sessions are critical for maintaining scoring consistency across annotators and contexts. After independent grading, inter-annotator agreement was measured using Cohen’s kappa.

Table 2 presents the results of Cohen’s kappa, a statistical measure of inter-annotator agreement. This measure accounts for agreement occurring by chance and is commonly used in annotation tasks to evaluate the consistency and reliability of multiple annotators. In our study, we compute Cohen’s kappa across four learner groups: foreign beginner (FB), foreign intermediate (FI), heritage beginner (HB), and heritage intermediate (HI), along with the overall agreement across all groups. The results show strong agreement across all groups, with the highest agreement observed for foreign beginners (0.8253) and the lowest for heritage intermediate learners (0.7245), likely reflecting the relative complexity and fluency of sentences in this group. Overall, the aggregated kappa score of 0.8241 indicates high inter-annotator consistency in our error annotation framework.

Taken together, the use of a detailed, pre-established rubric, combined with structured annotator training and verification of inter-annotator reliability, meets key criteria for a standardized assessment procedure. This provides a strong foundation for using the resulting dataset in further research on L2 Korean writing development.

In summary, the adoption of a detailed rubric and a carefully implemented annotation protocol ensures a high level of scoring reliability and transparency. These procedures establish the rubric-scored KoLLA corpus as a valuable benchmark for evaluating Korean L2 writing proficiency and developing automated essay scoring systems grounded in pedagogically meaningful criteria.

The enhancements to the KoLLA corpus provide insights into grammatical error patterns in Korean L2 writing, with particular emphasis on the benefits of multi-reference GEC. To the best of our knowledge, the multi-reference KoLLA corpus represents the first resource of this kind specifically tailored for Korean L2. It addresses inherent linguistic variability in learner writing, thereby enabling more comprehensive and contextually appropriate evaluations.

Single-reference GEC datasets may overlook natural variability in learner language, potentially penalizing alternative yet valid corrections. In contrast, multi-reference annotations capture the diversity of valid grammatical structures, which is particularly critical given Korean’s syntactic and morphological flexibility. Korean’s syntactic flexibility frequently allows multiple grammatical structures or synonyms to express equivalent meanings without compromising grammatical correctness. Consequently, corrections for case markers, verb endings, and compound verbs often exhibit considerable variability, each reflecting a valid linguistic choice dependent on discourse context.

Lexical flexibility also contributes notably to correction diversity, with many corrections allowing synonymous substitutions without altering intended meaning. This characteristic underscores the importance of flexible evaluation frameworks in GEC systems, prioritizing meaning preservation rather than strict grammatical conformity.

In subsequent analyses, we plan to further investigate specific instances of word-boundary inconsistencies and lexical alternatives by conducting a quantitative error analysis, examining correlations with proficiency levels and evaluating how these variations affect automated correction model performance.

Beyond grammatical error correction, the introduction of rubric-based scoring provides a broader assessment framework for evaluating Korean L2 writing proficiency. Unlike GEC, which focuses primarily on grammatical accuracy at the sentence level, rubric-based scoring assesses higher-level attributes of writing quality, including coherence, structure, and content development. By incorporating rubric scores aligned with the Korean National Language Institute guidelines, the KoLLA corpus establishes a standardized framework for evaluating L2 writing performance.

Analysis of the rubric scores indicates a strong correlation between grammatical accuracy and overall writing proficiency. Essays with fewer grammatical errors consistently received higher rubric scores, suggesting that error-free writing contributes to improved coherence and clarity. This trend underscores the interdependence of grammatical competence and discourse-level writing skills. Additionally, lexical diversity emerges as a distinguishing factor among proficiency levels. Essays that exhibit a wider range of vocabulary usage tend to receive higher scores, reinforcing the role of lexical variation in language proficiency assessment.

Structural complexity is another key indicator of writing ability, as higher-scoring essays demonstrate stronger paragraph organization, better use of discourse markers, and smoother transitions between ideas. However, despite improvements in grammar and structure at higher proficiency levels, rubric scores reveal that certain discourse-level errors persist even among intermediate learners. Challenges in logical sequencing, underdeveloped arguments, and inappropriate rhetorical structures suggest that targeted instructional interventions are needed to address discourse-level deficiencies in L2 writing.

The rubric-based scoring also highlights differences in proficiency levels across various essay types. Argumentative essays tend to exhibit more discourse errors, particularly in structuring logical arguments and supporting claims with appropriate evidence. In contrast, personal essays often score higher in fluency and coherence but may suffer from lexical repetition. These distinctions indicate that different types of writing require tailored instructional approaches to address specific linguistic challenges.

The integration of multi-reference GEC and rubric-based scoring in the KoLLA corpus has significant implications for both Korean L2 learning and the development of automated NLP systems. The dataset provides detailed insights into common learner errors, allowing language educators to design targeted curricula that address frequent grammatical challenges such as postposition omission, honorific mismatches, and discourse structure issues. The refined annotation framework offers a valuable resource for the development of instructional materials aimed at improving grammatical competence and writing fluency.

For GEC model development, the inclusion of multi-reference annotations reduces penalization for alternative but grammatically acceptable corrections, making it more suitable for training flexible and context-aware correction models for Korean. By incorporating diverse correction strategies, these models can better accommodate the inherent linguistic variability present in learner writing. Moreover, the multi-reference dataset provides a benchmark for evaluating future GEC systems, ensuring that corrections account for multiple valid grammatical structures rather than rigidly enforcing a single reference.

In the context of AES, rubric-based scores serve as a foundational dataset for training AI-driven assessment models that align closely with human grading standards. By integrating grammatical, lexical, and discourse-level features, these models can provide more nuanced feedback to learners, assisting in both formative and summative assessments of writing proficiency. The incorporation of rubric-based assessments into computational models presents an opportunity to develop more sophisticated evaluation systems capable of providing personalized and structured feedback.

Future research should explore the scalability of multi-reference GEC datasets by incorporating additional annotators to capture a wider range of linguistic variations. Expanding the corpus with discourse-level error annotations could further enhance its utility for both GEC and AES research. Additionally, the integration of rubric scoring into neural network-based assessment models offers promising directions for improving automated feedback systems in Korean L2 writing instruction.

Overall, the findings from this study emphasize the necessity of incorporating both multi-reference grammatical error detection and rubric-based scoring in Korean learner corpora. These enhancements provide a more comprehensive framework for analyzing L2 writing, bridging the gap between error correction, writing assessment, and instructional design.

The enhanced KoLLA corpus demonstrates the value of integrating both multi-reference grammatical error annotations and rubric-based scoring for a more holistic evaluation of Korean L2 writing. Multi-reference GEC captures the linguistic variability and ambiguity intrinsic to Korean, while rubric-based scoring offers a structured and pedagogically grounded method for assessing broader aspects of writing proficiency. Together, these resources support both computational development and instructional practice, paving the way for more accurate, fair, and informative evaluation of learner language.