namquangstudy/aaie-gft-full-fg-8k-dpo

aaie-gft-full-fg-8k-dpo

DPO fine-tune of `moinsaj/aaie-gft-full-fg-8k` — a criterion-grounded educational feedback-generation (FG) model with YaRN 8x context scaling (8,192-token context) — trained to prefer honest, criterion-grounded feedback over feedback that either hallucinates a rubric criterion that doesn't exist, or is generic/non-grounded instead of specific. Trained with TRL's DPOTrainer on preference pairs built from a rubric-grading dataset; see the training code and full data-preparation details at dpo-fg/ in the source repo.

Request/response contract

This model expects a single user turn (no system/multi-turn) with a compact JSON input block and a trailing Response: marker:

Generate criterion-grounded educational feedback for the supplied assignment, rubric, and
student submission. Return JSON only. Use every rubric criterion identifier exactly once.

Input:
{"assignment_description": "...", "assignment_types": ["essay"], "domain": "...",
 "rubric": {"rubric_id": "...", "rubric_origin": "...", "criteria": [{"criterion_id": "c1",
 "name": "...", "description": "...", "assignment_excerpt": "", "performance_descriptors": {},
 "weight": null}, ...]}, "submission": {"final_submission": "..."}}

Response:

Expected output is one JSON object with four top-level keys in this order: criterion_scores (int 0-10 or null per criterion id), criterion_status ("assessed"/"not_assessable"), criterion_feedback (non-empty string per criterion id), and overall_feedback (academic_quality/next_steps/strengths/weaknesses, all non-empty strings). The criterion-id set in each map must exactly match the input rubric's criterion ids.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "namquangstudy/aaie-gft-full-fg-8k-dpo"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID)

user_content = (
    "Generate criterion-grounded educational feedback for the supplied assignment, rubric, "
    "and student submission. Return JSON only. Use every rubric criterion identifier exactly "
    "once.\n\nInput:\n{...see contract above...}\n\nResponse:"
)
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": user_content}], tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt")
im_end_id = tokenizer.convert_tokens_to_ids("<|im_end|>")
out = model.generate(
    **inputs, max_new_tokens=1280, do_sample=False,
    eos_token_id=[im_end_id, tokenizer.eos_token_id], pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Explicitly passing eos_token_id=[im_end_id, tokenizer.eos_token_id] matters — some downstream tooling defaults only to tokenizer.eos_token_id, which does not include the chat template's actual turn-end token (<|im_end|>) on every environment; passing both avoids ungoverned generation past the intended stop point.

Training procedure

TRL DPOTrainer: policy + a frozen reference model, both initialized from moinsaj/aaie-gft-full-fg-8k. Learning rate 5e-7, β=0.1, 2 epochs, batch size 1 with gradient accumulation 16, max_length=4096. 300 preference pairs (270 train / 30 eval), each pair sharing the same student submission and rubric, differing only in the feedback JSON returned — one criterion-grounded and contract-valid, the other either hallucinating an extra criterion id or giving generic, non-grounded feedback.

Result: reward accuracy converged to 1.0 on held-out pairs, reward margin grew from 0.28 to 6.65 over training, eval loss dropped from 0.10 to 0.013.

Known limitation

Validating both the base model and this DPO'd checkpoint against the response contract above (10 held-out samples each, 1280-token generation budget) found 0/10 fully contract-valid outputs for either model. Most failures are outputs that stop cleanly but still include an invented criterion id not present in the input rubric (sometimes not even present in that same response's own criterion_scores); the rest hit the token budget on genuinely detailed feedback. This is a pre-existing gap in the base model, not something this DPO run introduced or fixed — the training data's only hallucination pattern is a single fixed extra id, which was not diverse enough to teach general "never invent an id" behavior. The DPO objective itself was learned correctly (reward accuracy 1.0) but does not generalize to this failure mode. Do not treat this checkpoint's JSON output as reliably schema-valid without an application-level validator, and consider decode-time constraints (grammar-guided generation restricted to the real rubric's criterion ids) if strict validity is required.

Framework versions

• —TRL: 1.4.0
• —Transformers: 4.57.1
• —Pytorch: 2.7.1+cu118
• —Datasets: 4.8.5
• —Tokenizers: 0.22.1

Citations

Cite DPO as:

@inproceedings{rafailov2023direct,
    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
    year         = 2023,
    booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
    url          = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
    editor       = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
}

Cite TRL as:

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}