submarat/gpt2-small-fineweb-edu-10b-dpo

GPT-2 Small — DPO (preference-optimized) toy assistant

Direct Preference Optimization on top of the SFT model, using UltraFeedback (prompt/chosen/rejected) with TRL's DPOTrainer.

• —Interactive demo: https://huggingface.co/spaces/submarat/gpt2-fineweb-chat
• —Code: https://github.com/submarat/gpt2-small-repro (posttraining/)
• —Write-up: https://submarat.github.io/a-toy-assistant-sft-and-dpo/

What DPO did here (an honest note)

DPO raised the reward margin (chosen preferred at ~0.63) and shifted the style toward longer, more elaborate, "more assistant-like" responses — the known UltraFeedback length/verbosity preference. But at 124M, that extra length is capacity the model doesn't have, so it often fills the space with confident hallucination, and on some factual prompts it regressed vs. the SFT model. A small, honest demonstration of the reward-vs-truth gap / alignment tax. For a cleaner toy assistant, prefer the SFT model.

Same Alpaca prompt format and usage as the SFT model.

Training

• —Base: the SFT model
• —Data: UltraFeedback (15k pairs), Alpaca-formatted prompts
• —1 epoch, effective batch 32, LR 5e-6, β 0.1, bf16