thaki-AI/daily-paper-2026-09-25-spec-decoding-acceptance-output-structure

The Draft Law: Measuring How Agentic Output Structure Sets Speculative-Decoding Acceptance and Net Per-Token Cost on Self-Hosted H200

TL;DR — An analytical paper deriving the draft law for speculative decoding on agentic traffic over a self-hosted single H200: expected accepted prefix length is set by the mean structural predictability of the output - schema-constrained tool-call, reasoning, and prose positions - and the net per-token saving is strictly increasing in the schema-constrained fraction, with a closed-form net cost that includes the draft model's own serving overhead and a unique break-even schema fraction below which speculation is net-negative; the decode-side speculation lever composes orthogonally with prefill-side prefix reuse and retry-side validation gating, and a falsifiable four-arm protocol with pre-registered refutation criteria is declared for the H200 (no measurements are reported in this paper).

ThakiCloud AI Research · 2026-09-25 · 📝 Tech blog (KO)

Problem

Speculative-decoding speedups are reported as workload-agnostic gross numbers (e.g., Medusa's >2.2x lossless, 2.3-3.6x co-fine-tuned) that ignore the draft-side serving overhead and never resolve acceptance by output structure. On agentic traffic - where tool calls are schema-constrained, low-entropy, and speculation should pay off most - it is open whether the output structure of a request measurably sets the acceptance rate, and whether the net per-token saving is positive at output quality held identical to a matched BF16 baseline once the draft model's own cost is included.

Approach

An agentic turn is decomposed into position classes - schema-constrained tool calls (S), extended reasoning (R), and free-form prose (F) - with mass fractions wS, wR, wF and reasoning share r. Each class carries a normalized predictability rhoc = exp(-Hc) from its mean next-token entropy Hc; under position-wise acceptance epsilon = kappa(1 - rho) and within-class i.i.d., the expected accepted prefix length tau and the net per-token cost cnet = cT(1 + gamma(delta + d))/(tau + 1) follow in closed form, for a 4B BF16 draft with a 32B BF16 target co-located in vLLM on a single H200, depth gamma in {4, 8, 16}, batches 1 and 8. We characterize the first-order break-even epsilon = 2(1 - delta - d)/(gamma + 1), the unique break-even schema fraction w_S, and the orthogonal composition with prefix-cache reuse and validation-gated resampling, and state a falsifiable four-arm (A-D) protocol with pre-registered refutation criteria R1-R4 on the declared H200 platform.

Key contributions

• —The draft law (science): acceptance is set by mean structural predictability, not by workload identity - tau and the net saving fraction s are strictly increasing in the schema fraction w_S, with a closed-form marginal schema value, a lower bound s >= -gamma(delta+d), and a schema ceiling where the net speedup saturates at (gamma+1)/(1+gamma(delta+d)).
• —A net per-token cost model with an operator rule (system): a closed-form cost that includes the draft model's own serving overhead, a first-order break-even condition epsilon = 2(1 - delta - d)/(gamma + 1), and a unique break-even schema fraction w_S that tells operators exactly when to enable speculation at fixed quality.
• —Orthogonal composition with the token factory (society): the decode-side speculation lever multiplies disjoint cost components with prefill-side prefix reuse and retry-side validation gating under additive quality budgets, specified by a falsifiable four-arm protocol with pre-registered refutation criteria R1-R4 on a declared H200 platform.

Figures

[image] The net saving fraction of speculative decoding is strictly increasing in the schema-constrained fraction of an agentic turn: it is net-negative below the break-even schema fraction, crosses zero at the unique break-even point, and saturates as the schema ceiling is approached. (Analytical model (not measured)) _{Analytical model (not measured)}

[image] Grammar pruning concentrates the next-token distribution at schema positions, so the expected accepted prefix length is predicted to order tool-call, reasoning, and prose positions in that sequence. (Analytical model (not measured)) _{Analytical model (not measured)}

Results (as argued)

Analytical results only; the pre-registered H200 measurement protocol was not executed (experiment infrastructure unavailable), so no measurements are reported in this paper. Under the declared model: the net saving fraction s is strictly increasing in wS, bounded below by -gamma(delta+d), and crosses zero at a unique break-even wS; in the schema ceiling the net speedup saturates at (gamma+1)/(1+gamma(delta+d)); speculation is net-negative whenever the mixture mismatch exceeds epsilon; and the three-lever product model for Ctotal is given. Pre-registered predictions P1-P4 (structure gradient tauS > tauR > tauF with tauS/tauF >= 1.5; break-even identity within 25% relative error on arm D; composition residual within 15% relative error; bit-identical losslessness at temperature 0) await execution of arms A-D.

Limitations

Analytical and positional: no measurements in this paper - the pre-registered four-arm H200 protocol was not run because experiment infrastructure was unavailable. The within-class i.i.d. assumption treats long structured spans as memoryless (conservative if anything); the alignment constant kappa is draft-target-pair- and workload-dependent and is the quantity most in need of direct measurement; precision is held fixed at BF16 (the precision axis is covered by a separate prior paper); the cost model assumes small-batch, memory-bandwidth-bound decode, and mixture-of-experts targets change the draft overhead ratio d; the first-order break-even regime must be checked against fitted parameters on arm A.

Abstract

We are explicit that this is an analytical and positional paper. We study self-hosted agentic serving on a single H200 and show that output structure, not workload identity, is the variable that sets speculative-decoding acceptance. An agentic turn decomposes into position classes: schema-constrained tool-call positions, extended-reasoning positions, and free-form prose positions. Each class carries a normalized predictability, defined as the exponential of the negative mean next-token entropy, and a draft-alignment constant couples draft to target. The mean structural predictability across the position classes determines, in closed form, the expected accepted prefix length, which we call the draft law. We derive the acceptance function as a strictly increasing function of the schema-constrained fraction, together with the net per-token cost that includes the draft model's own serving overhead and the break-even schema fraction below which speculation is net-negative. We prove that the decode-side speculation lever composes orthogonally with the prefill-side prefix-reuse lever and the retry-side validation-gate lever, multiplying on disjoint cost components. We state a falsifiable four-arm protocol with pre-registered refutation criteria, specified to fill the frontier on the declared H200 platform. Along the structure and net-cost axes, the result is an explicit upgrade of Medusa's workload-agnostic gross speedups to the agentic traffic where speculation should pay most.

Files

• —📄 Paper (PDF)
• —LaTeX source
• —References (BibTeX)

Citation

@techreport{thaki_spec_decoding_acceptance_output_structure_2026,
  title  = {The Draft Law: Measuring How Agentic Output Structure Sets Speculative-Decoding Acceptance and Net Per-Token Cost on Self-Hosted H200},
  author = {ThakiCloud AI Research (Hyojung Han)},
  year   = {2026},
  institution = {ThakiCloud}, note = {thaki-AI/daily-paper-2026-09-25-spec-decoding-acceptance-output-structure}
}

Generated by ThakiCloud nightly research pipeline. License: CC BY 4.0.