Modelpublic

andreayhchen/sherlock-qwen2.5-3b-dpo-v1

sourceHugging Faceupdated 8d agoView on Hugging Face

0likes281downloads

Model Card

sherlock-qwen2.5-3b-dpo-v1

Qwen2.5-3B-Instruct post-trained to solve MATH (levels 1-3) problems in the voice of Sherlock Holmes. CS2881R Assignment 1, week 2 (RLAIF). Full weights: the LoRA adapters are already merged in.

Lineage: Qwen/Qwen2.5-3B-Instruct -> SFT (andreayhchen/sherlock-qwen2.5-3b-sft-v1, merged) -> DPO on judge-labelled pairs

Training: TRL DPOTrainer, 619 pairs built from the scored GRPO rollouts (score gap >= 2, matched on use of "Watson" and on correctness), 2 epochs, LoRA r=16, lr 5e-5, beta 0.1.

Reward / labels: an LLM judge (claude-sonnet-5, thinking disabled) scoring persona 1-10 against a written rubric. Responses breaking the format contract (no Final answer: \boxed{} line, outside 80-500 tokens, bullets/headers) got reward 0. Correctness was not rewarded.

Prompt format (required): no system prompt; the user message is the problem followed by \n\nPlease reason step by step, and put your final answer within \boxed{}.

Held-out results (200 MATH test problems, levels 1-3, greedy; persona = mean judge score):

model	persona	accuracy
base Qwen2.5-3B-Instruct	-	80%
SFT	5.27	59%
GRPO, 60 steps	5.80	56%
DPO	7.28	60%

Known issue: the judge adds about one point for the name "Watson" alone, and both RL stages increased its use.

Code: https://github.com/andreach3n/cs2881r-assignment1