thaki-AI/daily-paper-2026-09-04-redundancy-effect-skill-registry-routing

The Redundancy Effect: Decomposing Skill-Registry Size from Duplicate Mass in Router Accuracy of a 2,200-Skill Agent Harness

TL;DR — In a ~2,200-skill agent harness, much of the strict top-1 routing-accuracy loss seen as registries grow is a scoring artifact of near-duplicate skills rather than router degradation: a closed-form symmetry result shows that at fixed quality strict top-1 decays as 1/(m+1) in the number of redundant copies while Recall@k (k >= m+1) is unaffected - exactly reproducing the recorded -26.2-point top-1 versus 0.0-point Recall@5 signature - and a composition-dependent model turns duplicate density into an explicit redundancy tax with an effective-size break-point headroom. We ship a one-line cluster-aware metric fix, a break-even deduplication/consolidation policy, and a pre-registered 5-arm controlled injection protocol with falsification criteria; the paper is analytical and the protocol's measurements are specified for replication but not reported here.

ThakiCloud AI Research · 2026-09-04 · 📝 Tech blog (KO)

Problem

Large agent skill registries grow with near-duplicate mass: nightly LLM-curator rewrites land, by embedding distance, on top of skills that already exist (our live registry grew from ~1,600 to 2,234 entries in two weeks). The prior 'Curse of Skills' study measured accuracy decay against registry size at one recorded production point (N = 2,029: top-1 48.9%, Recall@5 86.7%; top-1 fell 26.2 points from N = 512 to N = 2,029 while Recall@5 moved 0.0 points) but held composition fixed and unobserved, so it could not separate how much of the decay is size itself versus redundant mass, and strict-ID scoring silently charges registries for copies instead of quality.

Approach

The paper inverts the experiment analytically: hold registry size fixed and decompose composition into unique functional coverage U = N(1-rho) and near-duplicate mass D = Nrho. (1) A composition-dependent model P_top1(N, rho) = r exp(-pu * (N-1) * (1 + gamma*rho)) extends the prior two-parameter survival model at the recorded operating point. (2) A closed-form symmetry proposition under Gaussian score noise: conditioned on the gold's cluster being top-retrieved, strict top-1 wins with probability 1/(m+1) in m redundant siblings (exact under exchangeability; margin form [Phi(Delta/(sqrt(2)*sigma))]^m under pairwise independence), while Recall@k for k >= m+1 is unaffected. (3) A one-line cluster-aware metric fix scores top-1 correct when the returned ID is the gold or a cluster member. (4) A deduplication/consolidation policy with break-even condition (m/(m+1)) * q * lambda * H > Cc over the three cost channels (near-flat O(log N) index queries, hot-path embedding compute, prompt-cache invalidation). (5) A pre-registered injection protocol: Arm 1 sweeps rho in {0.05, 0.10, 0.20, 0.35, 0.50} with embedding-calibrated synthetic near-duplicate injection at fixed N = 2,234 on the 63-case SRA suite; Arm 2 consolidates real cosine >= tau clusters, with falsification criteria F1-F3.

Key contributions

• —A closed-form symmetry result decomposing the recorded size signature (top-1 -26.2 points versus Recall@5 0.0 points from N = 512 to N = 2,029) into a duplicate-mass component: strict top-1 decays as 1/(m+1) in the number of redundant copies while Recall@k for k >= m+1 is unaffected, so most strict top-1 misses on redundant clusters are functionally correct.
• —A composition-dependent routing model Ptop1(N, rho) = r * exp(-pu (N-1)(1 + gamma rho)) with an explicit redundancy-tax schedule and an effective-size formula Neff = N(1-rho) + N*rho/(mbar+1) that turns duplicate density into break-point headroom against the size limit.
• —A one-line cluster-aware top-1 metric fix, a break-even deduplication/consolidation policy, and a pre-registered 5-arm controlled near-duplicate injection protocol with explicit falsification criteria (F1 scoring-artifact, F2 malignant-collision, F3 cost-channel), turning registry hygiene from folklore into a measurable, replicable discipline.

Figures

[image] At fixed N, top-1 error splits into retrieval failure (A) and ranking collision (B), and collision splits into benign sibling returns (B1) - the redundancy effect - and malignant wrong-but-similar returns (B2), which the one-line cluster-aware scoring fix removes. (Conceptual example) _{Conceptual example}

[image] Conditioned on the gold's cluster being the top-retrieved cluster, strict top-1 wins only when the gold ranks first among the m+1 equal-quality members, so it decays as copies accumulate while Recall@5 is unaffected for m up to 4 where the whole cluster fits inside the top-5. (Analytical model (not measured)) _{Analytical model (not measured)}

[image] At the recorded operating point (N = 2,029, r = 0.867), each unit of duplicate density withdraws strict top-1, and the per-duplicate collision multiplier gamma = 1.0 makes the tax slope twice as steep as gamma = 0.5, dropping from 48.8% to 41.1% and 34.6% at rho = 0.60. (Analytical model (not measured)) _{Analytical model (not measured)}

Results (as argued)

All quantitative claims are recorded values from the prior study or model predictions at the recorded parameters - no new experiments were run in this release (protocol measurements are pre-registered for replication and not reported here). At the recorded point (N = 2,029, r = 0.867, pu = 2.83e-4): the symmetry identity analytically reproduces the -26.2-point top-1 / 0.0-point Recall@5 signature; the redundancy tax schedule predicts strict top-1 falling from 48.8% to 41.1% (gamma = 0.5) and 34.6% (gamma = 1.0) as duplicate density rho rises from 0 to 0.60; at rho = 0.30 with mbar = 3 the model predicts a ~6.7-point redundancy artifact (cluster-aware top-1 ~55.6% versus recorded strict 48.9%); and the effective-size break-point (N_eff* ~ 1,946) is reachable at physical sizes up to ~2,395-3,113 as rho increases from 0.25 to 0.50. Recorded context: the fixed 63-case suite has lost 21 cases to churn (~36-day half-life) and the prior mass decomposition attributes ~38 points to ranking collision versus ~13 to retrieval failure.

Limitations

This is an analytical paper: the injection protocol was pre-registered but not executed (experiment infrastructure unavailable), so no measured dose-response is reported and the break-even condition is not yet calibrated on live arms. The model extends a single-point calibration (N = 2,029) by assumption, not by fit, so the tax schedule inherits that uncertainty. The symmetry result assumes Gaussian score noise and pairwise independence across siblings (the exact 1/(m+1) form needs only exchangeability), and tau = 0.95 is a design constant, not a measured threshold. The benign (B1) versus malignant (B2) collision split is harness-dependent: where near-duplicates differ in subtle ways (parameters, side effects), cluster-aware scoring can over-credit and F2 is the pre-registered test for that case. The 63-case suite's 42% case loss to churn limits statistical power (binomial SE ~6.3 points at p = 0.5), which is why the design relies on a 5-arm dose-response plus case-level bootstraps.

Abstract

We analyze router accuracy in a large agent skill registry at fixed size, separating the effect of duplicate mass from the effect of registry size. Our prior study calibrated a two-parameter survival model, P_\topone(N)=r\,e^-p\,(N-1), at one recorded production point (N=2,029 skills; top-1 48.9%; Recall@5 86.7%) and varied size while holding composition unobserved. This paper inverts the experiment: it holds size fixed and decomposes composition into unique functional coverage and the near-duplicate mass introduced by nightly LLM-curator rewrites. We contribute (i) a composition-dependent model extending the prior one, (ii) a closed-form symmetry result showing strict top-1 decays as 1/(m+1) in the number of redundant copies while Recall@k for k\ge m+1 is unaffected (reproducing the recorded -26.2-point top-1 against 0.0-point Recall@5 signature), (iii) a one-line cluster-aware metric fix, (iv) a deduplication policy with a break-even condition, and (v) a pre-registered controlled injection protocol with explicit falsification criteria. At the recorded parameters the model predicts, for example, a 6.7-point redundancy artifact in strict top-1. Protocol measurements are specified for replication and are not reported here.

Files

• —📄 Paper (PDF)
• —LaTeX source
• —References (BibTeX)

Citation

@techreport{thaki_redundancy_effect_skill_registry_routing_2026,
  title  = {The Redundancy Effect: Decomposing Skill-Registry Size from Duplicate Mass in Router Accuracy of a 2,200-Skill Agent Harness},
  author = {ThakiCloud AI Research (Hyojung Han)},
  year   = {2026},
  institution = {ThakiCloud}, note = {thaki-AI/daily-paper-2026-09-04-redundancy-effect-skill-registry-routing}
}

Generated by ThakiCloud nightly research pipeline. License: CC BY 4.0.