thaki-AI/daily-paper-2026-08-31-multi-skill-gap-sequence-routing

The Multi-Skill Gap: Measuring the Cost-Quality Frontier of Order-Sensitive Skill-Sequence Routing in a 2,200-Skill Agent Harness

TL;DR — First order-sensitive chain benchmark for skill routing: 48 composite tasks (12 real workflow templates x 4 variants) over a 2,275-skill bilingual production registry, measured across free BM25 top-k retrieval, self-hosted Qwen3-1.7B/14B composers, and a frontier cost anchor. Free top-20 retrieval reaches 75.7% set completeness but 0.0% exact ordered-chain accuracy with 87.0% extra-skill injection; both local composers emit 0 parseable chains on 48/48 tasks (repetition loops) at $0.0007-$0.0011 per task; the pool itself caps full-chain coverage at 56.3%.

ThakiCloud AI Research · 2026-08-31 · 📝 Tech blog (KO)

Problem

Production agent harnesses automate composite work - ordered chains of 2-4 skills in which completeness and order are the deliverable - yet every routing measurement to date, including our own, scores the decision as a selection of at most one skill. A correct set in the wrong order is an unexecutable plan, and missing a middle step breaks the workflow, so single-skill-era metrics certify routers that never produce an executable chain. This measurement blind spot (the multi-skill gap) is unquantified at production registry scale.

Approach

We build a 48-task composite chain bench with ground-truth chains by construction (12 real workflow templates x 4 surface variants, chain lengths 2-4) over a 2,275-entry bilingual (Korean-English) production registry snapshot, and measure four composer arms: (A) free BM25 top-k set as-is, k swept over 1-20; (B) self-hosted Qwen3-1.7B and Qwen3-14B local composers on an exclusive H100 GPU, with 14B as the ceiling proxy for the 27B NVFP4 deployment class; (C) a frontier composer reported as a list-price cost anchor from the measured input load. We score order-sensitive metrics - exact ordered-chain accuracy, set completeness, pair-order fraction, extra-skill injection, first-step recall, pool gold coverage - with per-task token and GPU-second accounting at a $3.00/GPU-hour sensitivity price.

Key contributions

• —Order-sensitive chain benchmark: 48 composite tasks with deterministic labels by construction over a 2,275-skill production registry - the first such measurement in a 2,000+ skill registry.
• —Composer-tier cost-quality frontier: a controlled measurement of free top-k retrieval (k=1-20) against local 1.7B/14B composers and a frontier list-price anchor, with per-task token, wall-time, and dollar accounting.
• —A measured composer-tier decision rule: the local <=14B tier is non-functional (0/48 parseable chains, dominated by phrase-level repetition loops), the substitution threshold k* degenerates to 1 and the 90% first-step anchor k0 is unreachable (max 72.9%), so composite planning should not be gated on a local composer; the binding investment is retrieval-side coverage of the 24.3% of gold skills the top-20 pool misses.

Figures

[image] Set completeness rises monotonically from 0.2153 to 0.7569 while extra-skill injection rises from 0.3125 to 0.8698, and pair-order fraction stays in the 0.57-0.69 band: coverage buys noise, not order. (Measured on GPU pod) _{Measured on GPU pod}

[image] The free top-20 arm recovers 75.7% of gold skills as a set, while both local composer arms (Qwen3-1.7B and Qwen3-14B) score 0.0000 on every chain metric because they emit no parseable chains. (Measured on GPU pod) _{Measured on GPU pod}

[image] Local composers pay $0.0007-$0.0011 per task for zero measured chain quality, while Arm C serves as a list-price cost anchor of $0.0069 per task ($3.00/$15.00 per million tokens on the measured input load; no quality figures). (Measured on GPU pod) _{Measured on GPU pod}

Results (as argued)

Free top-20 retrieval (Arm A): 75.7% set completeness, 0.0% exact ordered-chain accuracy, 62.2% pair-order fraction, 87.0% extra-skill injection; first-step recall peaks at 72.9% so the 90% anchor k0 is unreachable, and pool gold coverage shows only 56.3% of tasks place the entire gold chain inside the top-20 pool - a hard ceiling for any pool-restricted composer. Local composers (Qwen3-1.7B / Qwen3-14B, exclusive H100): 0 parseable chains on 48/48 tasks each, with a 3-word span repeated >=10 times in 87.5% of 1.7B and 58.3% of 14B completions, at $0.0007-$0.0011 per task (0.87-1.32 GPU-seconds per task). Frontier cost anchor (Arm C): $0.0069 per task at list price (6.2x B14, 9.5x B17), no quality figures. The measured cost-quality frontier collapses to the free arm, which strictly dominates both local arms on every chain metric at zero cost.

Limitations

48 tasks from 12 templates of one production harness: a regime characterization, not a population estimate, with external validity to other registries untested. Surface variants overlap gold-skill registry vocabulary by construction, so pool coverage is measured in the most favorable lexical regime. Local composers are measured as configured, with no repetition suppression or re-decoding intervention. Arm A excludes the dense-rerank stage, so hybrid reranking could shift the k-sweep values. Arm C is a cost anchor only - no frontier quality figures are reported. Costs use a $3.00/GPU-hour sensitivity price on exclusive (unbatched) GPU time and public list prices that vary by provider and over time.

Abstract

Production agent harnesses route each incoming task to reusable skills, but every routing measurement to date--including our own--scores the decision as a selection of at most one skill, while the work these harnesses automate is composite: ordered chains of two to four skills in which completeness and order are the deliverable. We close this blind spot with an order-sensitive chain benchmark: 48 composite tasks (12 real workflow templates x 4 surface variants) with ground-truth chains by construction, evaluated over a 2,275-skill production registry whose skill descriptions follow the bilingual (Korean-English) production convention. We measure four composer arms: (A) no composition--BM25 top-k set as-is, free, k\in\1,\dots,20\; (B) self-hosted local composers, Qwen3-1.7B and Qwen3-14B--the largest resident model, a ceiling proxy for the 27B NVFP4 deployment class--served on a single H100 GPU; and (C) a frontier composer, reported as a list-price cost anchor from the measured input load. Three findings. First, the \gap: free top-20 retrieval reaches 75.7% set completeness but 0.0% exact ordered-chain accuracy, 62.2% pair-order fraction, and 87.0% extra-skill injection, and only 56.3% of tasks place the full gold chain inside the pool at any k--a hard ceiling for any pool-restricted composer. Second, the local composer tier is non-functional in this regime: neither 1.7B nor 14B emits a single parseable chain on 48/48 tasks, with completions dominated by phrase-level repetition loops, at \0.0007-\0.0011 per task and 0.87-1.32~s of exclusive GPU time per task. Third, the meas

Files

• —📄 Paper (PDF)
• —LaTeX source
• —References (BibTeX)

Citation

@techreport{thaki_multi_skill_gap_sequence_routing_2026,
  title  = {The Multi-Skill Gap: Measuring the Cost-Quality Frontier of Order-Sensitive Skill-Sequence Routing in a 2,200-Skill Agent Harness},
  author = {ThakiCloud AI Research (Hyojung Han)},
  year   = {2026},
  institution = {ThakiCloud}, note = {thaki-AI/daily-paper-2026-08-31-multi-skill-gap-sequence-routing}
}

Generated by ThakiCloud nightly research pipeline. License: CC BY 4.0.