thaki-AI/daily-paper-2026-09-11-reasoning-verbosity-tax-agentic-cost

The Reasoning Verbosity Tax: Per-Turn Cost-Quality Frontiers of Explicit vs. Compact Chain-of-Thought in Self-Hosted Agentic LLMs on H200

TL;DR — An analytical paper that turns the Qwen3 thinking toggle into a priced dial for self-hosted agentic LLM loops: explicit chain-of-thought persisted in context compounds a per-turn verbosity tax quadratically in the horizon (Theta(T^2)) versus linearly (Theta(T)) under a discard policy, the relative tax widens monotonically toward a structural bound set by the reasoning share of context, and closed-form quality-per-dollar break-even conditions let a cost meter decide thinking on/off per workload class. A falsifiable four-arm protocol (bf16-8B student / NVFP4-27B teacher on H200, BFCL simple/multiple/plus and a 63-case SRA routing bench) is specified to fill the frontier; no measurements are reported.

ThakiCloud AI Research · 2026-09-11 · 📝 Tech blog (KO)

Problem

Agentic LLM deployments re-invoke the model on every turn of a tool-calling loop, and each invocation re-prices the reasoning tokens that persist in context. The reasoning regime — explicit chain-of-thought versus compact thinking-off — is typically pinned once per deployment, and the cost of that pin is invisible in a per-task dollar meter: the extra reasoning tokens are paid again as prompt tokens on every later turn, and the bill compounds quietly. The paper asks how this per-turn reasoning-verbosity tax compounds across multi-turn agentic workloads, and where compact reasoning beats explicit on quality-per-dollar.

Approach

Holding the model family (Qwen3-8B student in bf16, Qwen3.8-27B teacher in NVFP4, both on H200) and the agentic task set (BFCL simple/multiple/plus and a 63-case SRA routing bench) fixed, the paper varies only the reasoning regime through the Qwen3 hybrid-thinking toggle and derives, in closed form: (i) a per-turn tax definition tau1; (ii) a compounding lemma for the absolute explicit-minus-compact bill under persistent versus discard context policies; (iii) a bounded-widening corollary with structural bound U = 1 + vE/(o+w); (iv) quality-per-dollar break-even conditions under two quality models (multiplicative deep chains with gamma = T, and k critical turns with gamma = k); and (v) an alpha-inflation interaction between low-bit serving and the tax. A fully specified four-arm measurement protocol (arms A–D, horizons T in {1,4,8,16}, persistent runs paired with discard counterparts) with refutation criteria R1–R4 is provided to fill the parametric frontier.

Key contributions

• —A formalization of the per-turn reasoning-verbosity tax with a compounding lemma: the absolute explicit-minus-compact bill is Theta(T^2) under persistent context versus Theta(T) under discard, making the context policy a measurable discriminator of the compounding mechanism.
• —Closed-form quality-per-dollar break-even conditions under two quality models, in which the relative tax is bounded by a structural bound U set by the reasoning share of context, and the context policy flips the long-horizon winner whenever the quality premium R lies below U.
• —An alpha-inflation interaction showing that low-bit (NVFP4) serving multiplies the tax through reasoning-token inflation and drives the critical horizon T_crit down toward T = 1, plus a falsifiable four-arm protocol (arms A–D) with refutation criteria R1–R4 specified to fill the frontier on Qwen3 on H200.

Figures

[image] The explicit-minus-compact cost gap grows quadratically in the horizon T under a persistent context and linearly under a discard policy, making the context policy a measurable discriminator. (Conceptual example) _{Conceptual example}

[image] Under the standard context condition the persistent-policy relative tax widens monotonically from the single-turn tax toward the structural bound U, while the discard-policy tax decays toward one. (Analytical model (not measured)) _{Analytical model (not measured)}

[image] Accuracy is non-monotonic in the thinking budget while cost increases monotonically, so the quality-per-dollar curve admits an interior optimum b and a pinned compact regime (b = 0) is acceptable whenever the price of optimality stays within operator tolerance. (Conceptual example)* _{Conceptual example}

Results (as argued)

All results are structural; no measurements are reported in the paper. (1) Under persistent context the absolute gap CT(E) - CT(C) = vE (pout T + pin T(T-1)/2) — a decode term plus a re-sent prefill term — while under discard the prefill term vanishes and the gap is linear in T. (2) Under the standard context condition pin m > pout w, the relative tax tauT increases monotonically from tau1 toward the finite bound U = 1 + vE/(o+w) under persistence and decays to 1 under discard. (3) In routing-heavy workloads (k critical turns) with a quality premium R below U, compact reasoning delivers strictly more quality per dollar beyond a closed-form critical horizon Tcrit, so the long-horizon winner is set by the context policy; deep dependency chains (gamma = T) instead keep the explicit regime justified as the horizon grows. (4) Low-bit serving inflates vE by a factor alpha, raises tau1 turn by turn, and drives Tcrit monotonically down toward T = 1, so a regime that is break-even-neutral in bf16 can be clearly compact-favorable under NVFP4. The four-arm protocol with refutation criteria R1–R4 is specified to measure every one of these predictions.

Limitations

No measurements are reported; the four-arm protocol is specified to fill the parametric frontier, and its refutation criteria are the price of the claims. The Qwen3 hybrid-thinking toggle is template-level (explicit versus no chain-of-thought) and does not cover trained latent recurrent reasoners. Amortized per-token prices assume steady-state utilization: burst workloads and prefix-caching policies change the effective input price of re-sent tokens, and hence tau_1 and U. Per-turn quality is assumed independent, and error propagation across turns can only widen the tax. BFCL and the SRA routing bench cover tool calling and routing; other agentic domains (code, retrieval) may have different critical-turn counts and require re-derivation of the break-even per domain.

Abstract

Agentic LLM deployments re-invoke the model on every turn of a tool-calling loop, and each invocation re-prices the reasoning tokens that persist in context. We study the reasoning regime itself -- explicit chain-of-thought versus compact, thinking-off inference on the Qwen3 hybrid-thinking toggle -- holding model family fixed (Qwen3-8B student in bf16, Qwen3.8-27B teacher in NVFP4, both on H200) and agentic task set fixed (BFCL simple, multiple, and parallel; a 63-case SRA routing bench). We are explicit that this is an analytical and positional paper. We contribute a formalization of the per-turn reasoning-verbosity tax, a compounding lemma (absolute explicit-minus-compact bill \Theta(T^2) under persistent context versus \Theta(T) under discard), a relative tax bounded by a structural bound set by the reasoning share of context, closed-form quality-per-dollar break-even conditions under two quality models in which the context policy flips the long-horizon winner when the quality premium lies below that bound, an \alpha-inflation interaction between low-bit serving and the tax, and a falsifiable four-arm measurement protocol with explicit refutation criteria R1-R4. The work upgrades BDH-CQ -- recurrent latent reasoning that broke the ARC-AGI-1 cost-accuracy Pareto frontier at 150M parameters -- from a small-model in-context setting to agentic multi-turn serving, and is orthogonal to our effort-allocation prior work : we price the format/density axis, not the effort axis. All results are structural; the protocol is specified to fill them.

Files

• —📄 Paper (PDF)
• —LaTeX source
• —References (BibTeX)

Citation

@techreport{thaki_reasoning_verbosity_tax_agentic_cost_2026,
  title  = {The Reasoning Verbosity Tax: Per-Turn Cost-Quality Frontiers of Explicit vs. Compact Chain-of-Thought in Self-Hosted Agentic LLMs on H200},
  author = {ThakiCloud AI Research (Hyojung Han)},
  year   = {2026},
  institution = {ThakiCloud}, note = {thaki-AI/daily-paper-2026-09-11-reasoning-verbosity-tax-agentic-cost}
}

Generated by ThakiCloud nightly research pipeline. License: CC BY 4.0.