thaki-AI/daily-paper-2026-08-28-zero-token-router-cost-quality

Zero-Token Routing: Measuring How Much LLM Cost a Learned Skill-vs-Model Router Saves at Fixed Task Quality

TL;DR — A closed-form economic model of three-arm agent routing - a zero-token deterministic skill arm, a small self-hosted model, and a frontier API model - under a fixed task-quality constraint shows that the cost-minimizing policy saturates the skill arm on every supported task, that a coverage threshold kappa (about 73% at a 10% quality tolerance under assumed parameters) eliminates all frontier spend, and that the savings fraction rises with a kink at kappa toward 100% (1 - delta/rho_F); a marginal-value rule v(kappa) prices which skills are worth authoring next. All numbers are closed-form evaluations under assumed parameters; the paper ships a selection-valid measurement protocol rather than measurements.

ThakiCloud AI Research · 2026-08-28 · 📝 Tech blog (KO)

Problem

Unattended agent fleets pay frontier-model prices for routine routing decisions even when a deterministic skill, gated by non-generative inference, could execute them without a single generative token. The LLM routing literature (cascades, triage, budget-constrained and bandit routers, confidence ladders) varies only knobs inside the LLM stack, so no prior work has a first-class zero-token, non-LLM arm in the decision space, and the dollar value that such an arm buys at fixed task quality has not been priced.

Approach

The paper formalizes three-arm routing with costs 0 <= delta < rhoM < rhoF and qualities qM < qS < qF (skill beats small model on covered tasks, trails the frontier), skill coverage kappa as the mass of tasks in registry support, and a fixed-quality constraint Q-bar = (1 - epsilon) qF relative to the frontier-only baseline, with confidence-threshold ladders as the policy class. Under a single efficiency-ordering assumption (A1), closed-form results are derived: the optimal structure, the dominance threshold kappa*, the savings curve s(kappa), a conservative-anchor corollary, and the marginal-value rule v(kappa) for skill authoring ROI. The model is evaluated under explicitly assumed parameters (qF = 0.90, qM = 0.70, q_S = 0.85, delta = 0, epsilon = 0.10, cost ratio r in {0.02, 0.05, 0.10}), and a selection-valid measurement protocol (labeled routing cases, pre-declared estimands, bootstrap intervals, dominance-point estimand) is specified to test where the kink sits.

Key contributions

• —First closed-form treatment of a zero-token deterministic skill arm as a first-class routing option: the cost-minimizing feasible policy saturates it on every supported task (Proposition 1), and for any positive coverage the skill-first policy strictly dominates the small-model-only policy at fixed quality, gaining kappa(qS - qM) quality at kappa(rho_M - delta) lower cost.
• —The frontier-elimination threshold kappa and the kinked savings curve: a frontier leg is needed if and only if kappa < kappa; s(kappa) is strictly increasing with a kink at kappa* and reaches 1 - delta/rho_F as kappa approaches 1 (Proposition 2); under the assumed parameters roughly 73% coverage removes all frontier spend at a 10% quality tolerance, and a conservative-anchor corollary bounds the direction of error when frontier quality is declared rather than measured.
• —A skill-investment ROI rule v(kappa) that prices the marginal value of new coverage - new skills are worth most where they displace frontier traffic (up to rho_F - delta per task), with a break-even authoring cost - plus a selection-valid measurement protocol that converts the propositions into pre-declared, testable hypotheses about the kink location.

Figures

[image] Schematic of the per-task savings fraction s(kappa) against skill coverage kappa for three small-to-frontier cost ratios r = 0.02, 0.05, 0.10: the curve climbs steeply while the optimal policy still carries a frontier leg, kinks at the coverage threshold kappa where all frontier spend disappears, and then approaches the full-savings limit (delta = 0) with a much shallower slope as coverage approaches one; the cheaper the small model, the higher the savings at every coverage level. (Analytical model (not measured))* _{Analytical model (not measured)}

[image] Schematic of the marginal value v(kappa) of one more point of registry coverage: while the current policy routes unsupported tasks to a frontier-heavy mix (kappa below kappa), a new skill is worth almost a full frontier call and its value decays toward the threshold; at and beyond kappa new skills displace only the small model, so the marginal value flattens at a much lower level - the ROI rule therefore prioritizes authoring skills for frontier-routed tasks. (Analytical model (not measured)) _{Analytical model (not measured)}

[image] Structural diagram of the routing policy: every task first passes the non-generative gate; tasks in registry support (mass kappa) are executed by the zero-token skill arm, and unsupported tasks fall to the small model plus a frontier mix whose fraction t(kappa) is positive only below the threshold kappa, where the frontier leg vanishes; with cost orderings delta < rho(M) < rho(F) and quality orderings q(M) < q(S) < q(F), the deterministic arm is the first mechanism the optimal policy saturates. (Conceptual example)* _{Conceptual example}

Results (as argued)

No empirical measurements are made in this paper (the measurement protocol is a design, and the planned local experiment was not run); all figures are closed-form evaluations under the stated assumed parameters, labeled as such. Under those assumptions: the frontier fraction t(kappa) - independent of the cost ratio - falls from 0.528 at kappa = 0.1 to zero at kappa = 0.7333; the savings fraction s(kappa) is a kinked line (slope r + A(1 - r) below the kink, r above, with A = 0.75) reaching 98.7% at kappa with the small model at 5% of frontier cost and 100% as kappa approaches 1 with delta = 0; and the tolerance sweep moves kappa* from 0.133 at epsilon = 0.20 to 0.733 at epsilon = 0.10, with the fixed-quality problem infeasible at all coverages once epsilon falls below 1 - qS/qF = 0.0556.

Limitations

The illustrative numbers are assumed, not measured, and the Section 6 protocol has not yet been run, so the propositions stand as structural results with order-of-magnitude content until measurement replaces the assumption table. The two-class, scalar-quality model abstracts from latency/SLO structure, reliability drift, and multi-turn compounding; coverage kappa is endogenous (the nightly repair loop grows it, registry staleness and retrieval drift shrink it), so the static threshold is a standing requirement rather than a one-time achievement; skill authoring cost enters only through the break-even rule, leaving a full authoring-cost model open; the efficiency-ordering assumption A1 can fail if the small model is priced close to the frontier per quality point, in which case the optimal policy need not saturate the skill arm; and tolerances tighter than the skill-to-frontier quality gap (epsilon < 1 - qS/qF) are unreachable at any coverage.

Abstract

Unattended agent fleets pay frontier-model prices for routine routing decisions even when a deterministic skill, gated by non-generative inference, could execute them without a single generative token. We study the economics of adding such a zero-token arm to the routing policy space. We formalize three-arm routing (a zero-token deterministic skill arm, a self-hosted small model, and a frontier API model) under the fixed-quality constraint (1-\varepsilon)\,qF relative to the frontier-only baseline, and derive closed-form results under a stated efficiency-ordering assumption. The cost-minimizing policy saturates the skill arm on every supported task. We identify the coverage threshold \kappa^\star at which the frontier leg disappears: for any positive coverage, the skill-first policy strictly dominates the small-model-only policy at fixed quality, and the savings fraction rises with a kink at \kappa^\star toward 100% (1-\delta/\rhoF) as coverage approaches one. A marginal-value rule prices skill authoring: new skills are worth most where they displace frontier traffic. These results are an explicit upgrade of the gatekeeper study, which priced the accuracy cost of compressing the routing gate; here we price the spend that the gate purchases. We close with a selection-valid measurement protocol that estimates the model's parameters on labeled routing cases and tests the location of the kink. All numerical illustrations are closed-form evaluations under assumed parameter values.

Files

• —📄 Paper (PDF)
• —LaTeX source
• —References (BibTeX)

Citation

@techreport{thaki_zero_token_router_cost_quality_2026,
  title  = {Zero-Token Routing: Measuring How Much LLM Cost a Learned Skill-vs-Model Router Saves at Fixed Task Quality},
  author = {ThakiCloud AI Research (Hyojung Han)},
  year   = {2026},
  institution = {ThakiCloud}, note = {thaki-AI/daily-paper-2026-08-28-zero-token-router-cost-quality}
}

Generated by ThakiCloud nightly research pipeline. License: CC BY 4.0.