Apppublic

aleada/what-context-fits

sourceHugging Faceapache-2.0updated 18d agoView on Hugging Face

0likes

App README

What context will actually fit?

A model's memory is its weights plus its KV cache, and the cache is the half that decides how long a conversation you can serve. Get it wrong and vLLM does not warn you — it fails at startup, or it starts and dies on a long request.

The general formula is layers × kv_heads × head_dim × 2 × context. It assumes every layer caches, and that every layer caches over the whole context. This page assumes neither: it reads what each model declares about its own attention, and says which rule it applied.

What the models declare

Declared as	What it means for the cache	Effect
`layer_types`	Per-layer attention span. Gemma-4-12B has 40 of 48 layers on a 1024-token window and 8 on the full context; gpt-oss-20b is half and half at 128 tokens.	5.9× · 2.0×
`layers_block_type`	Which layers attend at all. Nemotron-3.5 declares 52 layers of which 6 are attention; the rest keep a fixed-size recurrent state that does not grow with context.	8.7×
`kv_lora_rank`	Latent attention caches one compressed vector per token per layer rather than K and V per head — so its own `num_key_value_heads` describes the attention, not the cache.	24.9×
`sliding_window`	Only a window if the model says to use it. Some configs carry the field beside `use_sliding_window: false`, and applying it would understate the cache — promising a context that will not fit.	—

The effect column is that model's cache at its own declared maximum context, against the general formula. Where a model really does cache every layer over the whole context — most of them — the two agree exactly, and the page says so rather than manufacturing a difference.

Limits, stated

The weight size is an estimate, reconstructed from the Hub's per-dtype parameter counts; it lands within a few percent. The overhead is an allowance for activations and CUDA graphs, adjustable in the form — not a number anybody measured for your workload. Treat the result as a starting --max-model-len, not a guarantee.

Everything is read from config.json and the Hub's tensor index, in the visitor's browser. It never loads the model and never runs it, so it cannot know what a custom attention implementation does at runtime. Where a config does not declare enough to size a cache, the page says so instead of producing a number.

Development

The rules live twice — kv_memory.py (where they were developed, and where two claims were withdrawn after being checked) and kv_memory.js (what ships to the browser, because the free Space tier is static). A parity gate holds them together over whole configs captured verbatim from real models, one per topology, planned against four card sizes including one too small to hold the weights:

bash

./stage-shared.sh         # copy in the modules shared with the other Spaces
python harvest_cases.py   # re-capture configs from the Hub
python gen_golden.py      # record the Python answers
node parity.mjs           # fail if the JavaScript disagrees
./deploy.sh aleada/what-context-fits

deploy.sh runs the gate before it uploads, so a drifted port cannot ship.

Companion tools

—model search — find models by size, quantization, topology and parser
—precision map — how much of a pack is really 4-bit
—pack-integrity-check — does a pack's exclusion list match what is in it?
—reasoning-parser-advisor — which --reasoning-parser, if any

Built from the tooling behind the aleada W4A16 packs.