Modelpublic

KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

2likes855downloads

Model Card

[image]

Qwythos-9B-v2 — full GGUF ladder · measured KLD fidelity + loop-rate test

This is NOT my model. Weights and training are by [empero-ai](https://huggingface.co/empero-ai) (Qwythos-9B-v2, Apache-2.0). This is an independent GGUF repack whose value-add is what the other community GGUFs don't publish: an objective, measured fidelity sweep (KLD / PPL / Top-1 vs the Q8 reference) plus a real loop-rate benchmark on the quantized weights. No weights were modified.

What v2 fixes (per empero-ai): eliminated looping (6.7% → 0% at greedy decoding), restored the MTP head, cleaner identity, FTPO-trained. This repack is built from the official MTP-Q8_0 so the MTP head is preserved.

✅ Recommended files

Use case	File	Size	Top-1 vs Q8
Archival / best	Qwythos-9B-v2-Q8_0.gguf	9.5 GB	100%
Near-lossless	Qwythos-9B-v2-Q6_K.gguf	7.6 GB	98.2%
Balanced	Qwythos-9B-v2-Q5KM.gguf	6.6 GB	97.2%
Best compact (imatrix)	Qwythos-9B-v2-IQ4_XS.gguf	5.3 GB	96.2%
6 GB VRAM (imatrix 3-bit)	Qwythos-9B-v2-IQ3_M.gguf	4.5 GB	93.0%
Smallest (lossy)	Qwythos-9B-v2-Q3KM.gguf	4.7 GB	90.8%

The imatrix wins at every bit level. IQ4_XS dominates Q4_K_M (5.3 vs 5.8 GB, 96.2% vs 94.6% Top-1) and IQ3_M dominates Q3_K_M (4.5 vs 4.7 GB, 93.0% vs 90.8% Top-1, KLD 0.065 vs 0.102). For a 6 GB VRAM card, IQ3_M is the sweet spot — smaller and more faithful than plain Q3KM, with room left for context. Prefer the IQ tiers over the plain Q tiers at 3- and 4-bit.

📊 Measured fidelity — vs the Q8_0 reference

llama-perplexity --kl-divergence over 24×512-token chunks. KLD (Kullback–Leibler divergence) is the gold-standard quant-fidelity metric — it measures how far the quant's full next-token distribution drifts from the reference, catching damage that PPL and Top-1 alone miss. Top-1 match = how often the quant's argmax equals the reference's.

Quant	Size	PPL vs Q8	KLD mean	KLD p50	Top-1 match
Q8_0	9.79 GB	ref	ref	ref	100.0%
Q6_K	7.56 GB	+0.41%	0.0070	0.0008	98.2%
Q5KM	6.64 GB	+1.20%	0.0151	0.0020	97.2%
Q4KM	5.78 GB	+1.60%	0.0377	0.0067	94.6%
IQ4_XS	5.33 GB	+1.10%	0.0243	0.0045	96.2%
IQ3_M	4.52 GB	+5.63%	0.0652	0.0210	93.0%
Q3KM	4.74 GB	+6.90%	0.1017	0.0272	90.8%

[image]

Reference = Q8_0, not BF16. Qwythos-v2 is a hybrid `qwen35` arch (3:1 Gated-DeltaNet linear-attention : full attention + MTP), which llama.cpp does not lower cleanly from safetensors — so the ladder is requantized from empero-ai's official Q8_0 (near-lossless) with --allow-requantize + an imatrix, and fidelity is measured against that Q8_0.

🧪 Real loop-rate benchmark (measured on the quant)

v2's headline is "looping 6.7% → 0% at greedy decoding." We verified it on the quantized weights (Q5KM): greedy decode (temp 0), 10 diverse prompts × 220 tokens, flag any generation that degenerates into a repeated cycle or collapsed vocabulary.

Result: 0.0% loop rate (0/10 prompts) — confirms the quantization preserves v2's loop-fix — greedy (temp 0), 300 tokens each, mean repetition 0.038, every generation coherent (852–1527 chars incl. a naturally-repetitive counting prompt). (Method + raw per-prompt output in `scripts/loop_bench.py` + `reports/loop_bench.json`.)

🚀 Run it

bash

ollama run hf.co/KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q5_K_M
llama-server -m Qwythos-9B-v2-Q5_K_M.gguf -c 32768

Sampling: qwen3.5-family reasoning models want temp 1.0 / top_p 0.95 / top_k 20. For Ollama use RENDERER qwen3.5 + PARSER qwen3.5 (recent build). v2 no longer loops at greedy, but temp 1.0 is still recommended for reasoning quality.

⚠️ Honest scope

Two things are measured here: (1) KLD/PPL/Top-1 fidelity vs Q8, and (2) a loop-rate test. This is not an independent capability benchmark — for agentic/reasoning scores (MMLU, GSM8K, HumanEval, GPQA), see [empero-ai's card](https://huggingface.co/empero-ai/Qwythos-9B-v2) (their numbers, from their internal harness, not re-verified here).

🔁 Provenance

Requantized from the official MTP-Q80 with `llama-quantize --allow-requantize` + imatrix. Scripts in `scripts/` (`pipelinev2.sh, parsemetrics.py`, `loopbench.py), per-quant KLD logs in reports/, SHA-256 in reports/artifact-sha256sums.txt`.

Importance matrix (`qwythos-v2.imatrix`, shipped in this repo): computed fresh on this model's own Q8 (not reused from other weights), over a general-purpose calibration corpus — a ~250 KB mix of general English prose and code (standard for imatrix generation; not weight-specific data). Reuse it directly for your own quants. Note: the imatrix does not cover the MTP/nextn head (block 32) — a normal forward pass doesn't activate it — so IQ3_XS bails ("missing importance matrix … very low-bit"); IQ3_M quantizes cleanly (it tolerates the uncovered head) and is the recommended 3-bit tier.