Modelpublic

atakle/socratic-tutor-judge-v9-1.7b

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes15downloads

Model Card

Socratic Tutor Judge (v9) — Qwen3-1.7B (MLX)

A verdict-only, recall-first safety judge for AI math tutors. Given (problem, reference solution, conversation, candidate tutor message) it returns a JSON {verdict, reasoning} classifying whether the message leaks the answer or the pivotal key step. It is the detector stage of a two-model guardrail (judge → rewriter); pair it with `atakle/socratic-tutor-rewriter-v4-1.7b`.

Behavior spec: a message is adequate iff it neither states the final answer nor hands over the single key step/insight that trivializes the problem — it scaffolds with a question or calibrated hint.

Verdicts (5-way): adequate · gives_final_answer (leak) · gives_away_key_step (leak) · mismatched_calibration · vague_unhelpful. Safety axis: LEAK = the two leak labels.

Results (held-out frozen set, n≈300)

model	5-way	safety-binary	leak recall	leak F1
base Qwen3-1.7B	26.8%	68.5%	51%	53%
this model (v9)	64.1%	77.5%	90.4%	73.7
Claude Opus (frontier ref)	68.5%	87.9%	84.6%	83.0

Recall-first by design: the judge only triggers a rewrite, so a missed leak is the real harm and a false flag is cheap. On that metric the 1.7B ties/edges Claude Opus (90.4% vs 84.6%). Scale isn't the lever — a 4B trained on the identical data reaches ~93% (within noise).

Usage (MLX)

python

from mlx_lm import load, generate
model, tok = load("atakle/socratic-tutor-judge-v9-1.7b")
# System prompt = the project's verdict-task prompt (split_common.VERDICT_SYSTEM);
# user turn = problem + solution (ref only) + conversation + candidate message.
# Output: {"verdict": "...", "reasoning": "..."}

Trained via QLoRA (rank 16) on `atakle/socratic-tutor-data` (tier-2 minimal-pair augmented). Base: Qwen3-1.7B (Apache-2.0). Thesis: behavior from data, not scale.