stevhliu/quantization-picker

Quantization picker

An interactive version of the quantization method table from the Transformers docs. Instead of a 22×13 grid of status emoji, it answers the two questions people actually arrive with: will this fit on my card, and which method should I use.

Screens

The memory model

Figures are computed in lib/data.ts, not hardcoded:

weights = params × bits / 8                      # decimal GB
kv_cache = 2 × layers × kv_heads × head_dim × 2 × context_tokens
total    = weights + kv_cache + activations

Two things this makes visible that a bits-only table cannot:

• —KV cache is a fixed reservation. It does not shrink when you quantize weights, so it is drawn as its own block against the capacity line.
• —Fitting is not the same as being a good idea. 8-bit often clears the line with only a few GB spare, which will not survive a longer context.

ACTIVATIONS_GB is a flat 1.5 GB allowance and context is fixed at 8k. Both are estimates — see Caveats.

Design

Vercel's system (#fafafa canvas, hairline rings, 6px radius, Geist Sans with Geist Mono reserved for labels and metadata) with four accent colours borrowed from Cursor used only to identify precisions:

Fit and overflow are encoded as a tonal value shift — solid obsidian for what fits, graphite for the KV reservation, hairline grey for what spills — rather than red/green, so the chart reads without relying on colour.

Caveats

• —Capability flags mirror the current docs table, including its gaps. ? means genuinely unverified upstream, not "not yet filled in".
• —Config snippets in the comparison table need checking against the installed Transformers version before anyone copies them.
• —Setup minutes are order-of-magnitude estimates for a 70B calibration pass.
• —Quality at a given precision is deliberately not plotted — there is no trustworthy per-method number to plot.

Running it

npm install
npm run dev

Hardware data

lib/hardware.ts is scraped from the Hugging Face hardware table — the Hardware and Memory columns only. 316 entries, one per memory configuration rather than per device, because an RTX 3060 is sold as both 8GB and 12GB and only one of those runs a 7B at 8-bit. Order is the table's own, most owners first.

Two caveats:

• —CPUs are omitted. All 59 CPU rows list — for memory: they draw on system RAM, so there is no fixed pool to size a bar against.
• —`backend` is derived, not scraped. The table has no backend column, so vendor maps to cuda / rocm / metal / intel. That is coarse — a GTX 1060 and a B300 are both cuda — so the picker will offer a method on hardware too old to run it. Method filtering by compute capability, Metal kernel availability, and ROCm version needs a source the table does not have.