benjkoch/Qwen2.5-3B-Instruct-Sheldon-RLAIF-v2

Qwen2.5-3B-Instruct-Sheldon-RLAIF-v2

Checkpoint 2 of Harvard CS2881R Assignment 1. A Sheldon Cooper persona LoRA for Qwen/Qwen2.5-3B-Instruct, produced by GRPO against a constitutional AI judge (RLAIF) starting from the checkpoint 1 SFT adapter.

• —Run: rlaif-v2, epoch 2 · learning rate 2.5e-5 · 250 optimizer steps over 1,000 prompts
• —Starting point: zachchxn/Qwen2.5-3B-Instruct-Sheldon-SFT-v2 (checkpoint 1 SFT)
• —Reward: a fixed Qwen/Qwen2.5-14B-Instruct judge comparing pairs of sampled answers under one sampled principle from a 7-principle Sheldon constitution, scored in both A/B orders
• —Regularization: explicit KL penalty (beta 0.02) against the frozen SFT policy
• —Code: https://github.com/harvard-cs2881f26/hw1-chen-koch-zhou

This repository holds the LoRA adapter only; load it on top of the base model above.

Results, honestly

RLAIF did not produce a detectable persona improvement over the SFT baseline.

The paired persona change is +0.0019 with a 95% bootstrap interval of [-0.0064, +0.0103], and an independent Mistral-Small-24B-Instruct-2501 judge scored it at 0.500 against SFT — 4 wins, 4 losses, 92 ties on 100 held-out prompts. Math accuracy is statistically unchanged.

Full analysis, including what we think went wrong and the measured ~42% position bias in the judge, is in `docs/CHECKPOINT2.md`.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "benjkoch/Qwen2.5-3B-Instruct-Sheldon-RLAIF-v2")

The persona is prompted, not baked in: use the style guide from `prompts/sheldon_system.txt` as the system message, which is how the model was trained and evaluated.