Kevletesteur/quant-vs-api-parity-harness

🔬 quant-vs-api-parity-harness

Is your local quant actually as good as the full-precision API? This toolkit answered that question for a 284B MoE compressed to 2.9 bpw — verdict: indistinguishable on the real serving path (90.8% token-identical, 240/240 paired-QA parity, deep-derivation parity). 🎯 But the real product is the method: it caught 9 bugs in our own instruments before they could lie to us — including one that had us convinced the quant had lost factual recall when it hadn't. 🪤

Used to produce the results in DeepSeek-V4-Flash-0731-StrixHalo-Verified-GGUF. Model-agnostic in spirit: bring any llama.cpp server + any OpenAI-compatible reference API.

🚀 Install

python3 -m venv venv && . venv/bin/activate
pip install sympy jinja2                 # that's all — stdlib otherwise
echo "sk-..." > deepseek_api_key.txt     # your reference-API key (chmod 600!)
# a llama.cpp server on 127.0.0.1:8080 serving the quant you want to test

🧭 The method, in the order that matters

1️⃣ Validate your instruments FIRST (scorers.py) — non-negotiable

python3 scorers.py   # 67 synthetic cases — must print 67/67 before ANY model run

Answer extractors, SymPy equivalence oracle (with convention lists + decimal tolerance), degeneration detector, tool-call deep-equal. Each shipped with synthetic test cases: correct answers in hostile formats (LaTeX \pm, \boxed{}, thousand separators, mid-line), plausible wrongs, truncations, repetition loops. This step caught 2 real bugs in our own scorers on day one. If you skip it, your benchmark will eventually lie to you. 🐛

2️⃣ Run the gates (G2-G6) — they decide what your data can mean

3️⃣ Measure, cheapest-strongest first

🧬 P1 — token-level teacher forcing (p1_*.py) — the best instrument here. Generate greedy reference texts via the API (with top_logprobs), then force that exact token sequence through your local model one token at a time (cache_prompt=true, n_probs=20 → linear cost). Record top-1 agreement, the reference token's local rank, NLL. No judge, no regex — the oracle is the distribution itself. Bootstrap over TEXTS, never tokens (autocorrelation). Compare token ids, not strings.

📐 T2 — multi-step derivations, SymPy oracle (t2_*.py) — the compounding-error regime. Parametric bank (truths computed, never typed — our own self-test caught a hand-typed constant that was wrong 🙃), fixed FINAL: <expr> line, equivalence by simplify(candidate − truth) == 0. Truncated ≠ wrong. McNemar on paired outcomes. Log reasoning-trace lengths both sides and report them first.

🎯 T1/T1b — paired exact-answer bank + calibration (t1_items.py, t1b_passe.py). 240 generated integer-answer items, two conditions (forced / UNSURE-allowed), answer-token logprob both sides → AUROC(confidence → correctness) and the metric that actually matters for routing: P(answers ∧ wrong).

⏱️ Speed (reference/nmax_depth_bench_reference.sh): warm-up excluded, medians of ≥3, report t/s AND ms/eval (under speculation, t/s is not machine speed), nested prompts + prompt cache to pay prefill once. Our headline finding: the speculative `n_max` optimum inverts with context depth (3 at short ctx → 2 from ~16k).

🥇 The cardinal rule (it cost us our biggest false finding)

Evaluate QUALITY only through the real serving path — `/v1/chat/completions`. Raw /completion with an auto-injected <think> can close the reasoning instantly at temp 0 (10-character traces!) and answer reflexively. We published an entire "the quant lost factual recall" conclusion, then replayed the failed items through the chat endpoint: 4/4 correct. The deficit was our benchmark's serving path, not the quant. Full story and eight more traps in [`NEGATIVE_RESULTS.md`](./NEGATIVE_RESULTS.md) — read it before you optimize anything. 🪤

📁 Files

MIT. Benchmarked on AMD Strix Halo (gfx1151, ROCm 7.1, 115 GB unified) against the official DeepSeek API — but the method owes nothing to that hardware. Validate your instruments, equalize your serving paths, and let the distributions be the judge. 🔍