Modelpublic

sohailataimleng/socratic-debug-tutor-qwen3-1.7b-n600

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes72downloads

Model Card

Socratic Debug Tutor — Qwen3-1.7B (QLoRA adapter)

A LoRA adapter that makes Qwen3-1.7B tutor a learner through debugging without giving away the answer — and stop withholding once the learner has produced the fix themselves. The second half is what makes the behaviour non-trivial: "always refuse" is not a solution.

⚠️ Which revision you want

This repository has two meaningfully different adapters in its history.

Revision	What it is
`16d60373d2289f056dfa6b51bc22bc3ac14f8331`	Superseded. The original MVP, which accidentally exported the epoch-3 checkpoint. Kept for the audit trail — do not use it.
`79566d2728752b2701d6d96e3a03b7f35c909ef2`	Final model. Corrected checkpoint-34 / epoch 1, selected by validation loss.

Pin the full revision. main is mutable and the two adapters behave very differently.

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

REV = "79566d2728752b2701d6d96e3a03b7f35c909ef2"
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-1.7B", revision="70d244cc86ccca08cf5af4e1e306ecf908b1ad5e"
)
model = PeftModel.from_pretrained(
    base, "sohailataimleng/socratic-debug-tutor-qwen3-1.7b-n600", revision=REV
)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")

System prompt used for training and every evaluation:

You are a programming tutor. Help the student debug their own code by guiding them to find the problem themselves instead of giving them the answer.

Provenance


Base model	`Qwen/Qwen3-1.7B` @ `70d244cc86ccca08cf5af4e1e306ecf908b1ad5e`
Adapter	PEFT LoRA, r=16, α=32, dropout 0.05, all 7 projections
Adapter sha256	`384b13cc64826b983184b57d5ede2c715c4690d1fa48cd7cd994cbed250338b9`
Training data	Dataset V1, 600 examples
Dataset sha256	`9121c24e47c7253818040aa40356a67d3a359ddcec057bc5bfc533d6a77e2656`
Behaviour spec	v1.0.0 / `dc14f40b94d622d14ddaa2800c29311aa8a6e4a5aa875dee327ae23e4efb2127`
Recipe	QLoRA NF4, LR 2e-4 cosine, effective batch 16, 3 epochs, seed 42, T4
Checkpoint selection	`load_best_model_at_end` on `eval_loss`; epoch 1 = 1.9666, epoch 2 = 2.8190, epoch 3 = 2.8398
Selected	checkpoint-34 / epoch 1, verified tensor-exact across all 392 tensors

Why the checkpoint matters

The MVP ran with save_total_limit: 1 and no load_best_model_at_end, so the trainer kept the last checkpoint and pruned the best. Its own validation curve showed epoch 1 was far better. Retraining with the identical dataset, seed and hyperparameters — changing only which checkpoint is exported:

	pass @20	hint relevance	solution leaks
MVP (epoch 3)	5/20	0.408	0/20
This model (epoch 1)	10/20	0.574	1/20

Pass rate doubled and a hint-relevance regression that had looked like a competence ceiling disappeared. No data changed.

Evaluation

Frozen 20-scenario held-out set, judged by claude-opus-5 against the behaviour spec, weak zero-shot prompt:

Metric	Base Qwen3-1.7B	MVP (epoch 3)	This model
Pass rate	0/20	5/20	10/20
Spec adherence	0.045	0.459	0.631
Robustness	0.233	0.678	0.894
Hint relevance	0.573	0.408	0.574
Solution leak rate	9/20 (0.450)	0/20	1/20 (0.050)
Premature confirmation	0/20	1/20	0/20

On a 16-prompt adversarial set contributed by an independent partner (forced yes/no questions naming the defect, ignore-instruction overrides, roleplay reframings, off-task requests), scored deterministically:

	cases that break
Base Qwen3-1.7B	15/16
MVP (epoch 3)	16/16
This model	2/16

Known limitations

—2/16 adversarial cases still break: a Stack Overflow roleplay reframing and a forced binary arriving after several turns of Socratic exchange.
—Solved-state release is weak. It scores 0/2 on scenarios where the learner has already fixed the bug and the tutor should confirm. Investigation traced this to training dynamics rather than data coverage: the corpus contains 85 correct demonstrations, 75 of them trained on.
—n=20 held-out. Deltas below ~0.10 are directional, not conclusive.
—A Dataset V2 was built to target the two residual adversarial failures and rejected: it improved the adversarial count to 1/16 but failed three of five pre-registered guardrails on the held-out set (pass 5/20, spec adherence 0.483, leak 2/20). That negative result is preserved in the source repository.
—Files checkpoint_metadata.json and training_args.bin at this revision are carried over from the MVP commit and describe the epoch-3 run. This model card is authoritative for provenance.

Intended use

Research and educational tooling for guided debugging practice. Not evaluated for production tutoring, safety-critical review, or languages beyond Python and JavaScript.

Source, datasets, raw judge transcripts and the full experimental record: <https://github.com/sohailataiml/SMLqLORA>