AXOlotlvaNschmozelot/quant-fit

Which GGUF quant fits my hardware?

The most common question when running an open-weight model locally: which quantization actually fits my machine?

Type a model's parameter count and your RAM (and GPU VRAM, if any). This little tool shows the approximate GGUF file size for each common quant level — Q8_0, Q6_K, Q5_K_M, Q4_K_M, IQ4_XS, Q3_K_M, Q2_K — and which of them fit, plus a recommended pick.

How it works: each quant has a rough bytes-per-weight, calibrated against real GGUF releases (e.g. an 80B model at Q4_K_M lands around 48–49 GB). Size ≈ params × bytes-per-weight. "Fits" leaves ~15% headroom for a modest context window on top of the weights, plus a couple of GB for the OS/runtime.

Caveats: the numbers are rules-of-thumb — real files vary a little by architecture, and long context windows need memory well beyond the headroom assumed here. And it fits ≠ it's fast: on a weak CPU expect a few tokens/second; a GPU with enough VRAM is far quicker.

Part 2 of The Open-Weights Lifecycle — a small series on owning the models you depend on, from acquire → verify → run → maintain → retire.