thaki-AI/daily-paper-2026-08-20-quantized-embedding-skill-router

Quantizing the Gatekeeper: Accuracy-Cost Tradeoffs of Compressing the Embedding Model in Hybrid Skill Routers

TL;DR — Quantizing the dense embedding half of a hybrid (BM25 + embedding) skill router to INT8 leaves fused hit@1/hit@3 unchanged on a multilingual stand-in encoder because the lexical term dampens the error, but an off-the-shelf dynamic quantization backend made single-query encoding slower rather than faster on CPU, failing the latency side of the compression's own justification.

ThakiCloud AI Research · 2026-08-20 · 📝 Tech blog (KO)

Problem

LLM agent harnesses route every user turn through a hybrid retriever that fuses a lexical (BM25) score with a dense embedding similarity. The dense encoder runs fresh on every turn, unlike a static document corpus where embedding cost is amortized once. The compression literature has focused on generation models and treats retrievers as fixed black boxes, so it is unknown whether quantizing the router's own embedding model degrades routing accuracy, and whether a practitioner can predict the answer cheaply before running an expensive sweep.

Approach

The paper (1) gives a first-order perturbation analysis showing quantization error enters the fused score only through the dense term, scaled by (1 - fusion weight alpha), so the lexical weight is a built-in linear dampener of quantization risk; (2) derives a rank-safety margin diagnostic that certifies, from float32 scores already computed during normal operation, whether a query's top-1 decision is provably invariant under quantization at a given bit budget, without ever running the compressed encoder over the evaluation set; (3) formalizes a break-even condition trading expected mis-route cost against latency and memory savings, plus a 5-stage validation protocol that gates expensive measurement behind the cheap diagnostic; and (4) reports a small empirical validation on a live production hybrid router (1,910 catalog entries, 36 golden queries, PyTorch dynamic INT8 post-training quantization of two sentence encoders, reciprocal rank fusion) on a single consumer CPU.

Figures

[image] Fusing with the untouched lexical lane keeps hit@1 unchanged for the multilingual stand-in encoder while the embedding-only lane degrades, but the contrast encoder shows the opposite pattern after fusion. (Measured on local MacBook (repo .venv, CPU, manual rerun 2026-08-19/20; route=custom, no GPU).) _{Measured on local MacBook (repo .venv, CPU, manual rerun 2026-08-19/20; route=custom, no GPU).}

[image] Dynamic INT8 quantization made single-query encoding slower rather than faster for both encoders on this CPU backend, failing the latency term of the break-even condition. (Measured on local MacBook (repo .venv, CPU, manual rerun 2026-08-19/20; route=custom, no GPU).) _{Measured on local MacBook (repo .venv, CPU, manual rerun 2026-08-19/20; route=custom, no GPU).}

[image] The fusion weight acts as a linear dampener of quantization risk: a router leaning on its lexical component is structurally more robust to embedding compression than a purely dense retriever. (Conceptual example (not measured); illustrates the linear relationship in Observation 2 (score-level risk proportional to 1-alpha), not computed from the Section 6 empirical run.) _{Conceptual example (not measured); illustrates the linear relationship in Observation 2 (score-level risk proportional to 1-alpha), not computed from the Section 6 empirical run.}

Results (as argued)

On the multilingual stand-in encoder, hybrid hit@1 and hit@3 were exactly unchanged after INT8 quantization (MRR moved by 0.0024) even though the embedding-only lane degraded by one query at hit@1, qualitatively corroborating the dampening effect under reciprocal rank fusion. A smaller English-only contrast encoder showed the opposite pattern: hybrid hit@1 fell by two queries after fusion even though the embedding-only lane's hit@1 did not move, so the dampening effect was not universal across encoders. Critically, INT8 dynamic quantization made single-query encoding slower on both encoders (8.29ms to 11.15ms for the stand-in encoder; 5.31ms to 6.72ms for the contrast encoder) on the CPU backend used, and the realized model-size reduction was only 1.16x-1.55x rather than the nominal 4x, because the backend quantizes only nn.Linear weights. The paper explicitly does not compute the rank-safety-margin diagnostic (p_risk) in this run, since per-query margins were not persisted, so the central analytical contribution remains empirically untested at n=36; the observed zero top-1 flips only bound the true degradation rate at roughly 8.3% (95% CI, rule of three).

Limitations

Golden set is small (n=36 of 63 regression queries, restricted to those whose gold skill exists in the live catalog), so accuracy deltas are within 1-2 query resolution and inside Wilson intervals. Only one quantization backend (PyTorch dynamic INT8, qnnpack on Apple Silicon), one bit width, one hardware target, and one batch size were tested; the sign of the latency result is stated to be backend- and platform-specific, not a general INT8 claim. The rank-safety margin diagnostic (p_risk) and the flip probability pi were never computed in the empirical run, so the paper's main analytical contribution is validated only qualitatively via aggregate hit@k/MRR, not via the diagnostic itself. The perturbation bound is worst-case and first-order, so it may over-flag queries if conservative, or be unsound with no visible symptom if the empirical error bound epsilon(b) under-covers the true tail. The isotropic single-scalar error-bound assumption is questioned by the encoder-to-encoder asymmetry observed between the two tested encoders.

Abstract

Hybrid skill routers--which fuse a lexical score with a dense embedding similarity--are queried on every user turn in an LLM agent harness, so the embedding half sits on the hot path of every request rather than being amortized over an offline corpus build. Quantizing it is an attractive but under-examined lever: the compression literature concentrates on generation models, while the skill-retrieval literature treats the retriever as a fixed black box. A first-order perturbation argument shows that embedding quantization error reaches the fused score only through the dense term, so the fusion weight linearly dampens quantization risk. From this bound we derive a rank-safety margin diagnostic certifying whether a query's top-1 decision is provably invariant at a given bit width, computable from float32 scores the router already produces, given only a bound \varepsilon(b) obtained from a theoretical guarantee or a small probe set--never from a run of the compressed encoder over the evaluation set. We then formalize a break-even condition over the resulting at-risk fraction, mis-route cost, and latency and memory savings, and specify a staged validation protocol. A small-scale validation on a live production router (1,910 catalog entries, 36 golden queries, dynamic INT8, reciprocal rank fusion) corroborates the dampening effect for a multilingual encoder standing in for the production one but is absent on a second, English-only encoder, so the accuracy result is one of two; and the break-even condition fails on its latency term, because the off-the-shelf post-training quantiza

Files

• —📄 Paper (PDF)
• —LaTeX source
• —References (BibTeX)

Citation

@techreport{thaki_quantized_embedding_skill_router_2026,
  title  = {Quantizing the Gatekeeper: Accuracy-Cost Tradeoffs of Compressing the Embedding Model in Hybrid Skill Routers},
  author = {ThakiCloud AI Research (Hyojung Han)},
  year   = {2026},
  institution = {ThakiCloud}, note = {thaki-AI/daily-paper-2026-08-20-quantized-embedding-skill-router}
}

Generated by ThakiCloud nightly research pipeline. License: CC BY 4.0.