DavidBShan/gsm8k-4b-nothink-opd-20260709

Qwen3.5-4B · GSM8K · non-thinking (SFT → On-Policy Distillation, kimi-k2.6 teacher)

A LoRA adapter for Qwen/Qwen3.5-4B that solves grade-school math word problems (GSM8K) in a single, non-thinking turn — concise natural-language chain-of-thought followed by a #### <answer> marker. No <think> blocks; the model reasons in the visible answer.

Trained on the Freesolo flash platform:

1. 1.SFT on a GLM-5.2 oracle (concise natural-language CoT, verified-correct solutions).
2. 2.On-Policy Distillation (OPD) continued in place from the SFT adapter. The OPD teacher was selected by a head-to-head benchmark of three teachers (glm-5.2, deepseek-v4-pro, kimi-k2.6) distilling into the same SFT base — kimi-k2.6 won, giving the highest-accuracy student and the only statistically-significant improvement over SFT.

Results

Greedy (temperature 0), served, on the full GSM8K test split (1319 questions). The model is non-thinking; the openai/gpt-5.5 baseline is called identically (same prompt, same max_tokens, greedy). Accuracy = exact final-answer match.

Two findings. (1) OPD with the kimi-k2.6 teacher improves on SFT by +1.9 pts at mt112 (p=0.034, McNemar) — a significant lift, unlike the other teachers. (2) The efficient non-thinking 4B beats GPT-5.5 across the tight-budget regime: significantly at mt104 (+3.7 pts, p=0.006) and mt96 (+8.6 pts), and it edges ahead even at mt112. At a fixed token budget GPT-5.5's hidden reasoning is truncated (11–15 % of its answers come back empty at mt96–104), while the 4B's concise CoT fits. Above ~mt115 GPT-5.5's fuller reasoning pulls ahead (0.829 @ mt128). This is the classic "token-efficient non-thinking model beats a truncated thinking model at a fixed tight budget" result — not a claim that the 4B out-reasons GPT-5.5 in general.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3.5-4B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "DavidBShan/gsm8k-4b-nothink-opd-20260709")

SYSTEM = ("Solve the math problem. Show brief essential arithmetic steps "
          "(no long explanations), then give the final answer on its own line as '#### <answer>'.")
q = "Natalia sold clips to 48 friends in April, then half as many in May. How many clips total?"
msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": q}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=104, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
# She sold 48/2 = 24 clips in May, so 48+24 = 72 total.\n#### 72

Serve max_new_tokens ≈ 104 to reproduce the reported operating point. The exact system prompt above is the one used in training and evaluation.

Limitations

• —Budget-saturated at ~0.72–0.76: the 4B does not use extra token budget productively, so it cannot match GPT-5.5's looser-budget scores (0.829 @ mt128) or an unbounded thinking model (~0.95). Its advantage is confined to tight budgets.
• —GSM8K-style arithmetic word problems only; single-turn, non-thinking.
• —LoRA adapter (rank 16, SFT→OPD continued in place) — requires the Qwen/Qwen3.5-4B base at load.

Training data

Derived from GSM8K (train split) with GLM-5.2-generated chain-of-thought SFT targets. Companion dataset: `DavidBShan/gsm8k-4b-nothink-opd-20260709`.