Modelpublic

lorenzocazzador/math-reviewer-simpo-r1-distill-7b

sourceHugging Facemitupdated 4mo agoView on Hugging Face

0likes10downloads

Model Card

Math Reviewer (SimPO) — DeepSeek-R1-Distill-Qwen-7B LoRA

LoRA adapter that turns `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B` into the math-solution reviewer of a self-refining coding/math agent loop: it reads a problem and a candidate solution, then scores and critiques it to drive the next refinement round.

Trained with SimPO (sigmoid_norm, β = 2.0) on mined preference pairs. SimPO removed the length bias of an earlier DPO variant and improved the reviewer's scoring accuracy by +5.16 pp over the base model, while keeping its free-text reviews coherent and deployable.


Base model	`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`
Method	SimPO (LoRA, r = 32, α = 64)
Role	math-solution reviewer / scorer in a self-refining agent loop

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
repo = "lorenzocazzador/math-reviewer-simpo-r1-distill-7b"

model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, repo)
tokenizer = AutoTokenizer.from_pretrained(repo)

Adapter produced for a master's thesis on self-refining coding/math agents.