TransGAT: Transformer-Based Graph Neural Networks for Multi-Dimensional Automated Essay Scoring

Research on holistic essay scoring has progressed from traditional machine learning methods to deep neural networks and, more recently, to Graph Convolutional Networks (GCNs). Traditional machine learning methods relied on handcrafted features such as vocabulary, sentence length, grammatical errors, and syntactic complexity (Madala et al., 2018; Sharma and Goyal, 2020). Advanced features, including parse tree depth and type-token ratio, have also been explored (Salim et al., 2019; Janda et al., 2019; Doewes and Pechenizkiy, 2020).

Deep learning methods introduced word embeddings to automate feature extraction, initially using static embeddings like GloVe and Word2Vec with CNNs, GRUs, LSTMs, and Bi-LSTMs (Chen et al., 2020; Li et al., 2018; Cai, 2019; Zhu and Sun, 2020; Xia et al., 2019; Muangkammuen and Fukumoto, 2020; Chen and Li, 2018). Attention mechanisms further enhanced performance (Chen and Li, 2018). Transformer-based models, such as BERT, RoBERTa, and DeBERTa, provided contextualized embeddings that outperformed static embeddings in AES tasks and yielded improved results when combined with LSTMs, CNNs, Capsule Networks, and Bi-LSTMs (Wangkriangkri et al., 2020; Wang, 2023; Sharma et al., 2021; Yang and Zhong, 2021; Yang et al., 2020; Mayfield and Black, 2020; Susanto et al., 2023; Beseiso et al., 2021). Hybrid approaches integrated both static and contextual embeddings to enhance performance (Li et al., 2023; Beseiso and Alzahrani, 2020; Zhou et al., 2021). However, most deep learning models primarily focus on local word sequences without explicitly capturing global word relationships (Yao et al., 2019).

GCNs have emerged as a promising method for capturing global relationships within textual data (Ren et al., 2022; Bhatti et al., 2023). Early studies applied GCNs to short-answer scoring, leveraging one-hot encoding, TF-IDF, and Word2Vec embeddings (Tan et al., 2023; Ait Khayi and Rus, 2020). Later research extended GCNs to essay scoring using GloVe embeddings (Ma et al., 2021). However, most prior work has primarily focused on holistic scoring, evaluating overall essay quality without addressing specific writing aspects.

Initial research in AES primarily utilized traditional machine learning algorithms combined with manually engineered features. In recent years, however, numerous studies have shown that neural network methods can more effectively capture the deep semantics of essays. For example, Li et al. (Li and Ng, 2024b) trained a simple neural network using extracted features as input and achieved state-of-the-art performance. Mathias et al. (Mathias and Bhattacharyya, 2020) compared a feature-engineering system, a string kernel-based model, and an attention-based neural network using GloVe pre-trained word embeddings to automatically extract features for scoring various essay traits, finding that the neural network delivered the best results. The introduction of deep learning further automated feature extraction through word embeddings, with models such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs), Long Short-Term Memory networks (LSTMs), and Bidirectional LSTMs (BiLSTMs) showing promising results (Hussein et al., 2020; Ridley et al., 2021; Shin and Gierl, 2022; Do et al., 2023; Chen and Li, 2023).

More recently, many AES studies have utilized transformer-based pre-trained language models—such as BERT, DistilBERT, RoBERTa, and DeBERTa—to obtain essay embeddings, followed by model fine-tuning to optimize performance (Lee and Nam, 2021; Xue et al., 2021; Ormerod, 2022; Lohmann et al., 2024; Sun and Wang, 2024; Chen et al., 2024). Several studies have enhanced these transformer models by integrating them with additional deep-learning architectures. For example, Lee et al. (Cho et al., 2024) incorporated a CNN with the output from BERT to improve local feature extraction, while another study (Lee et al., 2023) combined RoBERTa with an enhanced Bidirectional GRU (BiGRU) to boost scoring accuracy. Recent work has also begun to explore the use of large language models (LLMs), which have shown strong performance in AES tasks (Xiao et al., 2025).

However, most of these approaches primarily focus on local word sequences and often overlook the global relationships between words (Yao et al., 2019). To address this limitation, GCNs have emerged as a promising technique for capturing global dependencies within textual data (Ren et al., 2022; Bhatti et al., 2023). Building on this, the present study integrates Transformer-based contextual embeddings with GCNs for multidimensional essay scoring. Transformer models, such as BERT and DeBERTa, provide rich, context-dependent representations of essays that effectively capture semantic features. By feeding these embeddings into a GCN, the proposed hybrid approach leverages the strengths of both architectures: the deep semantic understanding from Transformers and the global relational modeling from GCNs. This combination enhances predictive performance and contributes to more accurate and robust AES outcomes.

Transformer-based language models like BERT have demonstrated remarkable performance in NLP tasks (Gillioz et al., 2020), building upon the Transformer architecture originally proposed by Vaswani et al. (Vaswani, 2017). Unlike traditional sequential models, the Transformer uses an encoder-decoder structure that processes entire input sequences simultaneously. Although it does not inherently encode word order, it incorporates positional encoding to retain sequence information, ensuring that word representations vary based on their positions. The self-attention mechanism allows the model to capture dependencies between all tokens in a sequence (Vaswani, 2017).

These models are highly effective at learning general-purpose language representations from large-scale unlabeled corpora, which can be transferred to various downstream tasks through transfer learning. This process involves unsupervised pre-training on extensive text data, followed by supervised fine-tuning on task-specific datasets to adapt the model to the target domain. This strategy often yields superior performance compared to training models from scratch (Schomacker and Tropmann-Frick, 2021).

This study evaluates the performance of three Transformer-based pre-trained models on the AES task, using the large version of each model for comparison. Each model was fine-tuned on the AES datasets to generate embeddings tailored to the data, which then served as the foundation for further processing.

1. 1.

   BERT, short for ”Bidirectional Encoder Representations from Transformers,” is a foundational model in natural language processing that introduced a deep bidirectional approach to language understanding. The large version of BERT, known as BERT-large, consists of 24 layers and 340 million parameters. It is pre-trained on large corpora using masked language modeling and next-sentence prediction tasks, allowing it to capture context from both directions in a sentence. BERT-large has achieved strong performance across a wide range of NLP benchmarks (Devlin et al., 2019).

2. 2.

   RoBERTa, short for ”Robustly Optimized BERT Pre-training Approach,” replicates the original BERT by optimizing hyperparameters and significantly increasing the size of the training data. The RoBERTa-large model, with 24 layers, is pre-trained on five diverse English-language datasets totaling over 160GB of text. RoBERTa has achieved state-of-the-art results in several downstream tasks across multiple benchmarks (Liu et al., 2019).

3. 3.

   DeBERTa, short for ”Decoding-enhanced BERT with Disentangled Attention,” further refines the BERT and RoBERTa models by introducing two novel techniques: a disentangled attention mechanism and an enhanced masked decoder. The large version of DeBERTa, DeBERTa-large, improves pretraining efficiency and model performance (He et al., 2021).

The model consists of four interconnected modules: a Transformer module, a syntactic dependency module, a GAT module, and a fusion module. The Transformer module leverages a fine-tuned Transformer model (described earlier) to extract contextualized embeddings from the input text, which are then passed through a dense layer to generate essay-level predictions. At the same time, the syntactic dependency module employs the Stanza toolkit to parse the syntactic structure of the text and to represent it as an adjacency matrix. In the GAT module, the contextualized word embeddings obtained from the fine-tuned Transformer model are used as node features, while the edges of the graph are defined by syntactic dependencies extracted by the syntactic dependency module. This graph is processed through two GAT layers, followed by a global mean pooling to generate a graph-level representation for prediction. Finally, the outputs of the Transformer and the GAT modules are fused to produce the final analytic scores.

Both the Transformer and Transformer-GAT models are trained using Mean Squared Error (MSE) loss:

where:

• 1.

  $y_{i}$ is the ground-truth score for the $i$-th analytic trait,

• 2.

  $\hat{y}_{i}$ is the predicted score for the $i$-th analytic trait,

• 3.

  $i$ indexes the six traits: cohesion, syntax, vocabulary, phraseology, grammar, and conventions,

• 4.

  $\mathcal{L}_{\text{MSE}}$ is the average loss across all six traits.

The English Language Learner Insight, Proficiency, and Skills Evaluation (ELLIPSE) corpus  (Crossley et al., 2023) is a publicly available dataset containing approximately 6,500 essays written by English Language Learners (ELLs) in grades 8–12. These essays were collected from statewide standardized assessments in the United States and span 29 distinct writing prompts. Each essay is annotated with both holistic and analytic proficiency scores, including evaluations of cohesion, syntax, vocabulary, phraseology, grammar, and conventions. On average, the essays contain around 430 words, most ranging between 250 and 500. Scoring is performed on a scale from 1.0 to 5.0, in 0.5-point increments. Two trained human raters evaluated each essay to ensure scoring reliability. This corpus addresses the limitations of previous analytic datasets, such as ASAP++ (Mathias and Bhattacharyya, 2018), which extended the original ASAP dataset to support analytic scoring but retained many of its constraints.

Quadratic Weighted Kappa (QWK) is a statistical metric used to assess the level of agreement between predicted and actual scores, making it especially well-suited for evaluating AES systems. Unlike basic accuracy measures, QWK accounts for the extent of disagreement between predicted and true scores, offering a more detailed evaluation of a model’s performance (Ramnarain-Seetohul et al., 2022).

The QWK score is computed using the following formula (Ramnarain-Seetohul et al., 2022):

where:

• 1.

  $O_{i,j}$ denotes the observed agreement matrix, which reflects how often each predicted score aligns with the actual score.

• 2.

  $E_{i,j}$ refers to the expected agreement matrix, indicating the level of agreement that would be expected by random chance.

• 3.

  $w_{i,j}$ represents the weight matrix, usually defined as:

  $$w_{i,j}=\frac{(i-j)^{2}}{(N-1)^{2}},$$

  (11)

  where $i$ and $j$ correspond to the predicted and actual score categories, and $N$ is the total number of distinct score levels.

QWK scores range from -1 to 1. A score of 1 reflects perfect agreement between the predicted and actual values, while a score of 0 suggests agreement no better than random chance. Negative values imply systematic disagreement. QWK incorporates a quadratic penalty, which means that larger differences between predicted and actual scores are penalized more heavily. This feature is especially important in educational settings like AES, where accurate scoring is essential, and significant errors can undermine the model’s credibility. As outlined in Table 1, QWK scores can be interpreted to assess the strength of agreement, with values below 0 indicating no agreement and those between 0.81 and 1.00 suggesting near-perfect agreement (Ramnarain-Seetohul et al., 2022).

The experimental setup includes configurations for both Transformer-based models and GAT models. The maximum input sequence length was determined based on model capacity: $1024$ tokens for DeBERTaV3 to leverage its ability to process longer inputs, and $512$ tokens for other models, in line with their architectural constraints. The tokenizers were extended to recognize special tokens representing paragraph breaks (\n) and double spaces, which are important for preserving the linguistic structure of essays. Training and evaluation were conducted using a batch size of $4$, and the models that achieved the best performance on the validation set were saved for subsequent analysis.

The GAT model comprised two layers and four attention heads and employed the LeakyReLU activation function. Both model types were optimized using the AdamW optimizer. The learning rate was set to $1e{-5}$ for the Transformer models and $1e{-3}$ for the Transformer-GAT model. A weight decay of $1e{-2}$ was applied only to the Transformer models. Both setups utilized a cosine learning rate scheduler and were trained for six epochs. No dropout was applied during training, as this was found to improve regression performance.

Extensive experiments were conducted by systematically varying several key factors to optimize model performance. These included using different dependency parsing tools—such as spaCy and Stanza—to construct the graph structure, tuning dropout rates to prevent overfitting, applying distinct learning rates for the Transformer and GAT components to balance their training, experimenting with various optimizers (e.g., Adam, AdamW), and testing different learning rate schedulers. Additionally, the number of GAT layers and attention heads, choice of activation functions (e.g., ReLU, LeakyReLU), and batch sizes were adjusted. These variations helped identify the optimal configuration for each analytic scoring dimension. The best results from these experiments are presented in Table 2, which reports QWK scores on the ELLIPSE dataset across six analytic scoring dimensions: Cohesion, Syntax, Vocabulary, Phraseology, Grammar, and Conventions.

RoBERTa-large-GAT achieved the highest average QWK score of $0.854$, reflecting its top performance in four out of six analytic dimensions among the GAT variants: Vocabulary ($0.825$), Phraseology ($0.861$), Grammar ($0.877$), and Conventions ($0.859$). In contrast, the DeBERTaV3-large-GAT model demonstrated strong performance in the remaining two dimensions, Cohesion ($0.877$) and Syntax ($0.853$), but its lower score in Conventions ($0.744$) contributed to a reduced overall average.

These findings underscore RoBERTa’s ability to capture complex linguistic features, making it more suitable for this task than DeBERTaV3. One possible explanation is that RoBERTa was pre-trained on a significantly larger corpus (160GB of text) compared to DeBERTaV3’s 78GB, which may enhance its effectiveness on AES tasks—despite DeBERTaV3 generally outperforming RoBERTa in other benchmarks (He et al., 2020).

Table 3 presents a comparative analysis of the proposed TransGAT method against two baseline studies, as these are the only studies that utilized the ELLIPSE dataset.

The baseline studies, represented by Sun et al. (Sun and Wang, 2024) and Chen et al. (Chen et al., 2024), employ RoBERTa-base and DeBERTaV3-base models, respectively. Sun et al.’s RoBERTa-base achieve relatively strong performance across most dimensions, with a notably high score in Vocabulary ($0.84$). Conversely, Chen et al.’s DeBERTaV3-base, despite being more recent, performs relatively lower across all dimensions, indicating a less effective adaptation to this task.

The TransGAT method, which integrates GAT with Transformer-based models, consistently improves performance over the baseline models. Specifically, the RoBERTa-large-GAT variant shows the strongest results among the baseline studies, achieving the highest scores in three dimensions: Phraseology ($0.861$), Grammar ($0.877$), and Conventions ($0.859$). Meanwhile, the DeBERTaV3-large-GAT variant excels in Cohesion ($0.877$) and Syntax ($0.853$), surpassing the RoBERTa-large-GAT model in these dimensions.

Although the TransGAT method generally outperforms baseline models across most analytic dimensions, it shows relatively lower performance in the Vocabulary dimension compared to Sun et al.’s RoBERTa-base model ($0.84$). This could be attributed to the fact that vocabulary scoring often relies heavily on surface-level lexical richness and diversity, which the baseline RoBERTa-base model may capture more directly due to its simpler architecture and focus on token-level representations. In contrast, the integration of GAT in TransGAT emphasizes relational and contextual information across essays, which may reduce attention to details lexical features critical for vocabulary scoring.

Overall, the results highlight the advantage of incorporating GAT with Transformer architectures. The TransGAT method effectively captures richer contextual and relational information between essays, enabling superior modeling of complex linguistic features critical to AES. This is particularly evident in the RoBERTa-large-GAT model’s performance, which outperforms the baseline RoBERTa-base model across almost all analytic dimensions.