PollardWeights/Qwen2.5-7B-Instruct-Pollard

Qwen2.5-7B-Instruct — Pollard

### Pollard shrank this model: 15.20 GB (f16) → 2.04 GB — 87% smaller, 7.5× down. The smallest rung here; larger, higher-fidelity rungs are listed below. | format | this model's size | |---|---:| | f16 | 15.20 GB | | Q80 | ~8.06 GB | | Q6K | ~6.23 GB | | Q4KM | ~4.41 GB | | PollardMix (this repo's IQ1_KT) | 2.04 GB |

Pollard builds of Qwen/Qwen2.5-7B-Instruct made with Pollard Weights — a ladder of measured-allocation quants (bits placed by per-layer sensitivity, not a uniform crush).

Standard GGUF — runs in stock llama.cpp / ik_llama.cpp, Ollama, LM Studio, except where noted. IQ1_KT needs ik_llama.cpp: its allocation puts ik_llama-only atoms on the tensors it protects. The rest run anywhere.

Model details

Which file should I choose?

Every rung is the same weights, sized to a different RAM budget by the measured allocation. Pick the largest one that fits your machine with room for context:

• —~8 GB RAM / VRAM → `Q6_K` (6.25 GB). stock atoms — near-lossless
• —~6 GB RAM / VRAM → `IQ4_XS` (4.46 GB). stock atoms — recommended default
• —~6 GB RAM / VRAM → `IQ3_S` (3.81 GB). stock atoms
• —~4 GB RAM / VRAM → `IQ1_KT` (2.04 GB). (ik_llama.cpp) trellis atoms — ik_llama.cpp only; the flagship mixed 1-bit build

Available files (WikiText-2 raw test, ctx 2048, 145 chunks)

f16 reference PPL 6.5250 ±0.0419.

tok/s measured on an RTX 5070 Ti (16 GB), full GPU offload, Windows/CUDA.

The numbers (WikiText-2 raw, ctx 2048, 145 chunks)

Perplexity with error bars, against the f16 reference. Sizes are the published byte counts:

Q6_K costs 0.3% perplexity at 41% of f16's size. IQ4_XS costs 2.0% at 29%.

What the stock rebuild cost

IQ4_XS and IQ3_S were rebuilt on stock ggml atoms. The files they replaced carried ikllama-only atoms (`IQ4K, IQ5K`) on every `attnv` while wearing stock type names, so they could not open in stock llama.cpp, Ollama or LM Studio — which is what the old card promised they would do. Both versions were measured on the same corpus:

The trade is legible: perplexity is unchanged and the files are a few per cent larger. ikllama's `IQ4K/IQ5K` are genuinely more efficient atoms, and giving them up costs size rather than quality. In exchange these two rungs now load in every runtime their names imply. `IQ1KT` was left on trellis atoms because trellis has no stock equivalent at all — that is where ik_llama's advantage is structural rather than incidental.

Checked with pollard-ggufcheck, which reads ggml type IDs out of the header rather than trusting the filename.

Measured notes

The `IQ1_KT` flagship beats a uniform 1-bit IQ1_KT baseline (PPL 11.86, Mean KLD 0.689, top-1 65.1%) on every metric at the same size class — PPL −14%, Mean KLD −22%, top-1 +4.2 pts. The K-quant ladder is imatrix-guided; on a dense model that is where the bits-per-byte win lives (the measured-KL knapsack is reserved for MoE, and is not claimed here).

Mean KLD is carried for Q6_K and IQ1_KT, whose files are unchanged. It is not restated for the two rebuilt rungs: those numbers were measured against the files that have been replaced, and re-measuring them is outstanding rather than assumed.

Prompt format

ChatML, the Qwen2.5-Instruct template:

<|imstart|>system You are a helpful assistant.<|imend|> <|imstart|>user {prompt}<|imend|> <|im_start|>assistant

The template is embedded in the GGUF metadata, so llama.cpp, Ollama and LM Studio apply it for you.

Download a specific file

pip install -U "huggingface_hub[cli]"
hf download PollardWeights/Qwen2.5-7B-Instruct-Pollard \
  --include "Qwen2.5-7B-Instruct-Pollard-IQ4_XS.gguf" --local-dir ./

How to run

These are standard GGUF and run with llama.cpp:

llama-server -hf PollardWeights/Qwen2.5-7B-Instruct-Pollard:IQ4_XS

or from a local file:

llama-cli    -m Qwen2.5-7B-Instruct-Pollard-IQ4_XS.gguf -ngl 99 -p "Explain why the sky is blue."
llama-server -m Qwen2.5-7B-Instruct-Pollard-IQ4_XS.gguf -ngl 99      # OpenAI-compatible API + web UI at :8080

They also work in anything built on llama.cpp — LM Studio, koboldcpp, Jan, ramalama, Ollama (ollama run hf.co/PollardWeights/Qwen2.5-7B-Instruct-Pollard).

imatrix (calibration)

The importance matrix (qwen7b_calib3.imatrix, included) was computed on Calib-3.0: 13.9 MB of mixed prose, code, reasoning and dialogue, 60 chunks at ctx 512.

ARM / AVX

llama.cpp repacks weights into an interleaved layout at load time for faster inference on ARM and AVX machines — no special file needed, online repacking covers these quants. The old Q4_0_4_4/4_8/8_8 variants are not required.

Errata

• —IQ1_KT carries ikllama-only atoms and needs ikllama.cpp to run; stock llama.cpp rejects any ggml type above 42 outright. Checked with pollard-ggufcheck, from the files' tensor types rather than their names.
• —Measured allocation places bits by per-layer sensitivity under a size budget.
• —Single machine; replication invited.

Credits & license

• —Base model: `Qwen/Qwen2.5-7B-Instruct` (Qwen)
• —Quantization tooling: llama.cpp (ggml-org)
• —Method + tooling: Pollard Weights — measure first, no claim before a number.
• —License: apache-2.0, inherited from the base model.

Built with [Pollard Weights](https://github.com/WestWaters/pollard-weights) — frontier models, small hardware, no compromise.