mkvn/moe-expert-cache-cliff

The expert-cache cliff

Interactive companion to [Quantization as Cache Amplification](https://huggingface.co/datasets/mkvn/quantization-cache-amplification) — Trillion-Parameter Mixture-of-Experts Inference on a Commodity Laptop (Kavin Kumar, Neural Metrics).

An LRU cache over MoE expert weights hits exactly zero below the k·L expert slots a single token touches. A token routes to k experts in each of L layers and revisits none before the next token — a cyclic reference string, the classical worst case for LRU.

This Space runs the cache simulation client-side, live, on the real published routing traces from OLMoE-1B-7B (int16[16, 49152, 8]), replayed in the true interleaved access order using the same policy logic as code/cache_policy.py in the paper repo.

The window shipped to the browser is 16 contiguous blocks of 256 tokens sampled evenly across all 49,152 traced tokens — contiguous so LRU's temporal locality is intact, spread out so expert popularity stays representative (Pearson r = 0.983 against the full trace). Popularity ranking is the exact per-layer distribution over all 49,152 tokens. Against the paper's own grid the window reproduces 0.00% LRU at 2/5/10% capacity, 26.6% at k·L (paper: 25.3%), and 21.9% popularity-pinned at 10% (paper: 22.9%).

Drag the capacity slider below 128 slots and watch the LRU curve sit flat on the floor. Then compare a popularity-pinned cache, which has no such failure mode.

One honest detail the sweep exposes: the collapse is not a literal mathematical step. In a narrow band just under k·L (~121–127 slots) enough of the previous token's set survives to produce partial hits. Below that band it is exactly zero everywhere. The paper's grid steps from 102 to 128 and so never samples the band; this Space shows it.

Scope: this simulates cache behaviour on measured 7B-scale traces. The paper's trillion-parameter figures are an analytical projection, not a benchmark — no 1T model was executed.