patrickbdevaney/dspark-mtp-draft-head-s3recap-p25-b0.1

deepseek-v4-flash-0731-cuda

A from-scratch pure-CUDA inference server for `0xSero/DeepSeek-V4-Flash-0731-REAP` (K160, native MXFP4) with embedded DSpark self-speculative decoding, MLA + DSA attention, hand-tuned for Jetson AGX Thor, `sm_110a`.

No Python on the hot path. Every kernel gated against a PyTorch oracle before it is trusted.

## Why Local frontier-adjacent intelligence, on hardware you own, fast enough to do long-horizon agentic work unattended. Neither half is rare alone — frontier capability is available through an API, fast local decode is available on small models. Both at once, on one box, with no network in the loop, is not. The four categories this engine runs fastest — long context 30.77, tool/JSON format 29.98, multi-turn 28.97, code edit 26.81 tok/s — are precisely the shapes agentic coding produces. Speculation pays where continuation is constrained, and agentic work is constrained by definition. Every gain here is lossless: emitted tokens identical to base AR, checked on every run. That invariant is what keeps "fast" and "frontier" from becoming a trade. [`NORTH_STAR.md`](NORTH_STAR.md) — the full argument, including the one number in it that is still inherited rather than measured.

Where the programme is (2026-08-26)

The work runs as four phases, in this order, on one box that can hold one model at a time. Full text and per-phase gates: `PRODUCTION_PLAN.md`.

Which head wins — three different answers, and they are not interchangeable

auto-ace0p5 is the highest-acceptance head this box has produced and it is correctly not promoted: +1.23 % over the incumbent, against a 3.5 % bar set from the measured cross-run spread on identical config. Promotion and release are also different gates — CURRENT_BEST still points at the older v1.0-s3 because the promoted head fails 3 of the 6 release floors, giving back ground on long-context and reasoning even while winning the mean by +18 % over run-0. Neither pointer is currently backed by a rule-passing block-5 measurement, and saying so is more useful than moving the symlink to whichever head is newest. Full ladder, rejects included: `HEAD_REGISTRY.md`.

Why phase 1 is called exhausted. Nine consecutive arms have failed to clear the promotion bar, and the best of them moved acceptance +1.2 %. The decisive negative is agentic-p25-b0.1: a purpose-built 3,071-prompt agentic corpus, 2.0x the training data of every other arm, which measured tau 3.6812 — worse than the 1,472-sequence incumbent. Data scale is not the binding constraint, and recipe search around this corpus family has converged. What remains on acceptance is structural, and the two structural options already tried are closed: typical acceptance won and shipped; DDTree is correct but not faster here, because the regime is depth-dominated.

Every head is archived, winners and rejects alike, under arms/<name>/ — 24 of them, each with mtp_trained.safetensors (BF16 training master), head_card.json (sha256 of every file, measured tau, full training history), train_metrics.json and eval.log. A rejected head is a measured point on the acceptance-vs-corpus curve and is worth exactly as much as a winner when choosing what to try next.

The target: 31–35 tok/s

22.66 → 28.38 tok/s is banked (+25.3 %). The remaining ladder is written down rung by rung, with what each is worth and what it costs: `DECODE_ENDGAME.md`, full mechanisms in `ROADMAP.md`.

This headline was 35–42 until 2026-08-23. The C(k) sweep measured its largest rung — adaptive block width, estimated at +20–25 % — at an oracle upper bound of +1.8 %. The estimate was wrong by an order of magnitude, so the target came down with it. See `wiki/negative-results.md` §4m.

The levers already used do not repeat. Twelve draft-head arms at block 5 put their top five within 1.3 % of each other against a 3.5 % promotion bar — ce/tv swept three ways, β bracketed on both sides, anchor shape bracketed at pow 1 and 2, HASS and the confidence loss term both retired. At ~13.8 tok/s per unit `tau`, an excellent further arm is worth +1.5 tok/s. That is why rung 1 is data rather than another hyperparameter.

What C(k) actually found. k\ does vary — {4, 5, 7, 8} across the suite — so the lever is real, just small. An engine that read each prompt's best width off the table with hindsight would gain +1.8 % on the suite mean; a live engine must predict k\ per position from the confidence head (AUC 0.88) and would realise a fraction of that. Four of nine prompts already sit at their optimum, and the two directions cancel: the agentic categories want wider (multiturn 7, agenticformat 8) while control, code_gen and explanation want narrower (4). The sweep also re-confirmed, on a fresh measurement, that 5 is the best fixed width (27.98 tok/s suite mean, against 27.75 at width 4 and monotone decay above 5).

Why the ceiling is low, and it is not the head's fault. tau's ceiling is the draft width, so at a fixed 5 even a perfect head is worth 1.30× — and perfect is impossible, because acceptance is bounded by the target's entropy, not by our ignorance. Varying the width was the one lever that removes that bound. It has now been measured, and it does not remove much: the width the target's entropy supports is close to 5 almost everywhere. The remaining headroom is in the kernels, not in the speculator.

Prefill is not on the ladder and may matter more than all of it. 90.8 tok/s and ~2.2 min TTFT at 12 k contribute nothing to tok/s, and a three-minute time-to-first-token makes throughput academic for an agentic harness.

Where the numbers are today

The shipped speculator is `s3recap-p25-b0.1`, promoted at tau 3.8413 against a same-width incumbent of 3.6888. Every candidate, rejects included, is in `HEAD_REGISTRY.md` with its weights archived under ~/model-backups/heads/; the programme that produced it is `wiki/draft-head-finetuning.md` §9. Nothing is ever deleted from the archive — a refused head is still a measured point on the acceptance curve, and two of this project's rulers turned out to be wrong after the fact.

τ is not comparable across block widths. τ counts tokens committed per target forward and its ceiling is the draft width, so 3.84/5 and 3.84/6 are not the same measurement. Ladder 2.1 moved the served width from 6 to 5 — which is what config.json's own dspark_block_size always said — and that alone re-prices every τ recorded before 2026-08-21. s3 reads 3.8438 at width 6 and 3.6888 at width 5, same weights.

One caveat that belongs next to the headline. Trained heads win the frozen suite and can lose on held-out continuation drafting against a true paired control (F116/F117: training helps where the head is weak and hurts where it is strong). P2.5's β anchor is the first lever that addressed this mechanically rather than by choosing a corpus — it pulls the head back toward its pre-training self in proportion to how well it is already accepting — and it is why that arm promoted when the four loss-reweighting arms before it did not.

The measurement protocol is part of the number. τ is quoted as an 8-prompt suite mean at NGEN0 ≥ 200 — past the drafter's 128-token sliding window. F92 measured τ at 1.39 over the first 32 generated tokens, rising to ~3.2 only after ~128, so a short-generation acceptance figure is a transient and is not comparable to anything, including this project's own earlier numbers.

The single most important correction this project has made to its own model of itself: the long-quoted "19.0 tok/s AR roofline" is a normalisation constant, not a target. It assumes every kernel moves bytes at full DRAM bandwidth and that the non-byte part of the step is zero. Neither holds — 22.3 ms of a 71.4 ms step is not bytes at all, and the byte-moving marks average 191 GB/s, not 233. See wiki/measurement-and-traps.md.

The wiki

Reference documents

The model

43 MoE backbone layers + 3 chained DSpark MTP blocks (layers 40/41/42, mtp.0/1/2). Hidden 4096, 64 heads × headdim 512, Q-LoRA/O-LoRA rank 1024, 8 o-groups, 160 routed experts top-6 + 1 shared, `moeintermediate` 2048, hyper-connections ×4 with 20 Sinkhorn iterations, sliding window 128, vocab 129280.

Quantisation as shipped, never re-quantised: MLA/dense FP8 e4m3 with F8E8M0 128×128 block scales; routed experts OCP MXFP4 (E2M1 + E8M0, block 32); norms/embed/lmhead/compressor/indexer BF16. B_tok = 12.26 GB/token.

Hard constraints this repo operates under

• —No additional quantisation. The checkpoint is used as shipped.
• —No invented model constants. Every number traces to config.json, REAP_MANIFEST.json or reap_plan.json. This has been violated once — a head size copied from a paper about a different model — and the correction is recorded in RESEARCH_LOG.md §6(a).
• —Token ids come from the checkpoint's own tokenizer (tools/encode_prompt.py, which gates itself on reproducing the canonical prompt). Inventing ids is the exact mistake the rule exists to stop.
• —Correctness gates before speed gates. One change per measurement. Report bands, not points.
• —DSpark is not DFlash. ~/gemma-cuda-hybrid and ~/laguna-s1-cuda-server are read-only references.

Build and run

bash scripts/build_decode.sh                      # the engine
bash scripts/build_gate.sh                        # the unit gates

# the ONLY sanctioned launcher: enforces single-tenancy and detaches
scripts/run_model.sh <log> ./build/decode <ckpt> "0,671,6102,294,8760,344" 8

The six-id prompt is BOS + "The capital of France is" and the expected first decoded token is 11111. Do not abbreviate it — a truncated list still runs, still prints a tok/s, and silently reports GATE FAIL against a different sequence.

Useful environment flags are catalogued in LEVERS.md §5. The two that matter most: MOE_MMA=1 (tensor-core MoE — right for prefill, wrong for decode) and DSV4_DPROF=1 (multi-level named GPU-phase timing, ~0.4 % overhead).

The server (Phase 6)

bash scripts/build_server.sh    # gates, server, terminal client
bash scripts/serve.sh           # CPU gates as preflight, then listen on :8080
build/dsv4-chat                 # terminal client
                                # web UI at http://localhost:8080/

OpenAI-compatible: /v1/chat/completions (streaming and not), /v1/completions, /v1/models, /health, /metrics. Tool calls, thinking blocks, and a KV prefix cache for agentic turns. One binary, no Python on the request path. See `SERVER.md` for the surface, the gates and the design decisions — in particular why the tokenizer is not gemma's and why the engine is a separate translation unit from src/decode.cu.