Modelpublic

tbooy/Qwen2.5-3B-Instruct-Sheldon-RLAIF-grpo-v2

sourceHugging Faceotherupdated 6d agoView on Hugging Face

0likes269downloads

Model Card

Qwen2.5-3B-Instruct-Sheldon-RLAIF-grpo-v2 (merged bf16 weights)

RLAIF-stage model of a Harvard CS 2881R project: GRPO with an LLM-judge reward on top of tbooy/Qwen2.5-3B-Instruct-Sheldon-SFT-touchup-v4 (itself v3b + a data-patch touch-up), the persona being Dr. Sheldon Cooper and the protected STEM task GSM8K.

Files: merged bf16 safetensors + tokenizer (step 109 adapter merged into the touch-up model).

Reward R = G * (2 * P + F) - rules - 0.5 * false_claim - batch_tax: a persona-blind task gate G (asks satisfied, refusal, harm, self-contradiction, false claims about the user), a pairwise persona score P from sibling comparisons judged in both presentation orders on nine items, a small form bonus F, 17 deterministic rule penalties from the v3b audit (canon table, fabricated corrections, format constraints, loops, preamble, truncation, tool voice, ...) and a rolling-window batch diversity tax. Judge: GPT-5.6 Luna via OpenRouter, calibrated first (v3b beats base 0.98, no position bias on that pair; see rlaif/judge/calibration_luna_v2.md).

Training: TRL 1.13 GRPOTrainer, LoRA r=32/alpha=64, 16 prompts x 8 completions per step, 400-token rollouts (T 0.8), DAPO loss, KL beta 0.04 to the seed, lr 1e-5 constant, 109 steps (4-hour wall-clock cap) on one H100 with vLLM colocated; 5,714 RL prompts (persona, short, constraint, two-turn, verdict, AI-identity, sensitive), no references.

eval	touch-up (seed)	grpo-v2
GSM8K test, strict, greedy	63.9	64.0 (63.9-64.9 at every checkpoint)
mean deterministic rule penalty (502 held-out prompts)	1.003	0.870
repetition-loop term / replies hitting the 400-token cap	0.191 / 18.9%	0.131 / 17.5%
LLM-judge pairwise win rate vs the seed (200 held-out prompts, both orders)	–	0.50 [0.45, 0.55]

Honest summary: at this learning rate and step budget the policy moved too little for the judge to tell it from its seed (a 1e-6 run moved nothing at all); the deterministic monitors improved modestly and GSM8K was preserved. The write-up recommends 3e-5 to 5e-5 with a verifiable GSM8K anchor slice for the next run.

Code, reward, judge prompts and write-up: https://github.com/TBOO-Y/cs2881r-sheldon-sft (rlaif/, CHECKPOINT2.md in the course submission). Companion repo: tbooy/Qwen2.5-3B-Instruct-Sheldon-RLAIF-grpo-v2-LoRA (adapter + checkpoints).