thaki-AI/daily-paper-2026-07-28-kv-cache-tiering-pd-disagg-cost

KV Cache Tiering Meets Prefill-Decode Disaggregation: Mapping the Cost-Latency Frontier for MoE LLM Serving on H200

TL;DR — Combining prefill/decode disaggregation with tiered KV-cache offloading is analytically antagonistic: disaggregation raises cache hit value but consumes the TTFT slack the slowest tier needs. Empirical validation failed (vLLM init crash), yielding zero performance data.

ThakiCloud AI Research · 2026-07-28 · 📝 Tech blog (KO)

Problem

LLM serving has two incompatible phases — compute-bound prefill and memory-bandwidth-bound decode — and KV cache is discarded after each request, wasting GPU-hours on redundant recomputation. Whether disaggregation and tiering compose additively or interfere is unresolved.

Approach

Analytical cost-latency model across HBM, DRAM, and NVMe tiers under disaggregation, deriving marginal-capacity breakeven (G*_t) under a power-law reuse distribution. Empirical validation attempted on H200 NVL with vLLM + LMCache (failed: engine init crash).

Key contributions

• —Analytical proof that P/D disaggregation and KV-cache tiering are antagonistic: disaggregation increases cache hit probability (pooling effect) but reduces admissible TTFT slack, disqualifying the disk tier on the critical path — net sign is regime-dependent (hit-rate-limited vs slack-limited).
• —Closed-form marginal-capacity breakeven G*t = (R A beta apre cp / ct)^{1/(1-beta)}: tiering capacity is a workload property first, hardware property second, and the exponent makes provisioning guidance non-transferable between deployments.
• —Null-result finding: vLLM V1 engine core failed to initialize on H200 with LMCache KV connector, twice, producing zero performance samples. The startup exception surface is generic and unactionable without child-process logs — the integration surface between engine memory sizing and third-party KV connectors is a fragile and poorly observable failure mode.

Figures

[image] Time to recover 1 GB of KV cache per storage tier on a log10 (ms) scale, so all four tiers are legible; HBM (~0.21 ms) to NVMe (~333 ms) spans roughly 1600x, which disciplines which tiers can sit on the synchronous TTFT path. (Analytical model (not measured)) _{Analytical model (not measured)}

[image] Breakeven capacity G from Equation 11 grows super-linearly as the reuse tail index beta approaches 1, shown on a log10 (GB) scale; provisioning guidance is therefore highly sensitive to the reuse distribution's tail shape. (Analytical model (not measured))* _{Analytical model (not measured)}

Results (as argued)

No measured performance data (engine initialization failed, zero samples collected). Analytical model is unvalidated but yields two regime-characterizing observations and a breakeven equation. Follow-up measurement protocol specified in 5 steps (harness instrumentation through disaggregated MoE sweep).

Limitations

Analytical model unvalidated; empirical canary was dense 0.5B (not MoE) and single-GPU — could not exercise MoE cost structure or fabric transfer term; public issue reports cited for diagnostic analysis are unreproduced; moving image tag means exact build is unrecoverable.

Abstract

Prefill-decode (P/D) disaggregation and tiered KV cache offloading each improve large language model serving efficiency, but are almost always studied separately. We ask whether combining them shifts the latency-cost Pareto frontier for Mixture-of-Experts (MoE) serving on NVIDIA H200, and where the marginal offload tier stops paying for itself. Analytically, we model expected per-request KV access latency across an HBM, DRAM, and NVMe hierarchy under disaggregation, amortize prefill-pool and decode-pool GPU-hours independently against tiering rent and cross-worker transfer, and derive a closed-form marginal-capacity breakeven under a diminishing-returns reuse distribution. The composition proves structurally antagonistic: disaggregation raises a shared cache pool's value by increasing cross-request hit probability, while consuming the time-to-first-token slack the slowest tier needs to remain admissible. Empirically, our validation attempt on an internal H200 cluster produced no latency, throughput, or cost measurements: the vLLM engine core failed to initialize, twice, and zero performance samples were collected. We report that as a finding, show the observed exception is a generic vLLM startup surface that elides its own root cause, state our unconfirmed hypothesis, and specify the protocol that would validate the model.

Files

• —📄 Paper (PDF)
• —LaTeX source
• —References (BibTeX)

Citation

@techreport{thaki_kv_cache_tiering_pd_disagg_cost_2026,
  title  = {KV Cache Tiering Meets Prefill-Decode Disaggregation: Mapping the Cost-Latency Frontier for MoE LLM Serving on H200},
  author = {ThakiCloud AI Research (Hyojung Han)},
  year   = {2026},
  institution = {ThakiCloud}, note = {thaki-AI/daily-paper-2026-07-28-kv-cache-tiering-pd-disagg-cost}
}

Generated by ThakiCloud nightly research pipeline. License: CC BY 4.0.