star092304/ielts-writing-task2-transgat

TransGAT for Automated Essay Scoring

This project implements the TransGAT model for Automated Essay Scoring, leveraging a combination of Transformer architectures and Graph Attention Networks (GATs). It is based on the research presented in the referenced paper.

Multi-Dimensional Scoring (IELTS Task 2 Criteria)

Unlike traditional systems that output a single holistic score, this model performs Multi-Dimensional Automated Essay Scoring. It evaluates essays across the 4 official IELTS Writing Task 2 assessment criteria:

• —Task Achievement (TA): Evaluates how well the essay addresses all parts of the prompt and presents a clear, well-supported position.
• —Coherence and Cohesion (CC): Assesses the logical structure of the essay, paragraphing, and the effective use of cohesive devices (linking words).
• —Lexical Resource (LR): Evaluates the range, accuracy, and appropriateness of the vocabulary used.
• —Grammatical Range and Accuracy (GRA): Measures the variety of grammatical structures deployed and the correctness of sentence construction.

The TransGAT architecture models these specific criteria by capturing both semantic features (via Transformers) and structural/relational dependencies within the text (via Graph Attention Networks).

Dataset

The model is trained and evaluated on the IELTS Writing Task 2 Evaluation dataset. You can find the dataset on Hugging Face here: chillies/IELTS-writing-task-2-evaluation 👉 [View Clean Dataset](https://huggingface.co/star092304/ielts-writing-task2-transgat/tree/main/data)

Training Results

Below are the training metrics from the two phases of model training:

Phase 1: Initial Training Metrics

[image]

Phase 2: TransGAT Metrics

[image]

Usage

You can use this model either by cloning the repository and using the high-level TransGATScorer interface, or by loading the model directly using Hugging Face's transformers library.

Option 1: Using the High-Level Interface (Recommended)

The repository includes an inference.py script that handles the end-to-end pipeline, including text tokenization, dependency graph parsing via Stanza, and score denormalization.

In a Python Notebook / Script:

# 1. Download inference.py, modeling_transgat.py, and graph_utils.py into your workspace
# 2. Run the following code:
from inference import TransGATScorer

# Load the scorer (It will automatically download the model weights and initialize Stanza)
scorer = TransGATScorer("star092304/ielts-writing-task2-transgat")

# Predict scores for an essay
essay_text = "In recent years, climate change has become one of the most pressing global issues..."
result = scorer.predict(essay_text)

print(result)
# Output: {'TA': 7.0, 'CC': 6.5, 'LR': 7.0, 'GRA': 6.5, 'OverallBand': 7.0}

Via Command Line Interface (CLI):

python inference.py --model_dir "star092304/ielts-writing-task2-transgat" --essay "Your essay text here"

Option 2: Loading Directly via Transformers

If you want to integrate the core model into your own custom training or evaluation pipeline, you can load it directly using AutoModel.

⚠️ Note: Since TransGAT requires a dependency graph as an additional input, you will need to manually extract node features (graphx), edge indices (graphedgeindex), and batch indicators (graphbatch) before feeding them into the model.

import torch
from transformers import AutoTokenizer, AutoModel

# Load tokenizer and custom TransGAT model
model_name = "star092304/ielts-writing-task2-transgat"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

model.eval()

# Prepare text inputs
essay = "Your IELTS essay text here..."
inputs = tokenizer(essay, max_length=512, truncation=True, padding="max_length", return_tensors="pt")

# [Required] You must construct the graph inputs (graph_x, graph_edge_index, graph_batch)
# based on your dependency parser (e.g., Stanza) before running the forward pass.
with torch.no_grad():
    outputs = model(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        graph_x=graph_x,                  # Node features aligned with word embeddings
        graph_edge_index=graph_edge_index, # Graph structure edges
        graph_batch=graph_batch            # Graph batch mapping
    )
    
    # Raw normalized predictions
    logits = outputs["logits"] 

Dependencies

To run the inference smoothly, please ensure you have the following libraries installed:

pip install torch transformers stanza numpy

Reference Paper

This implementation is inspired by and based on the following research paper:

TransGAT: Transformer-Based Graph Neural Networks for Multi-Dimensional Automated Essay Scoring Hind Aljuaida, Areej Alhothalia, Ohoud Al-Zamzamia, Hussein Assalahib

View paper