KikoCis/Qwen3.8-27B-GGUF

Qwen3.8-27B — GGUF quants, with the point where they stop driving tools

Quants of Qwen/Qwen3.8-27B. What this repo adds over the other GGUF builds of the same model is not the files: it is a measured answer to how far down you can quantize before an agent stops working, and the honesty about how noisy that measurement is.

📦 Files

The two top rungs sit above the 3.40-bpw build that scores 40/40, but they were not put through the benchmark themselves — "not measured" here means exactly that, not "assumed fine".

🌐 Running it in a browser

IQ1_M-compact is the build that Elffuss Claw loads on WebGPU — one tab, one machine, no install and no sharding across devices — through Elffuss Engine (Apache-2.0), which reads a standard GGUF over HTTP by range requests and dequantises inside the shader. It is slow and the first load is several gigabytes; it answers, it does not feel instant. Note the row above: at 1.94 bpw this build is for text, not for driving tools.

🧠 Why the context is cheap on this model

Of its 65 blocks: 48 are SSM / linear-attention, 16 are full attention (every 4th), and 1 is an MTP head. Only those 16 layers hold a KV cache that grows with sequence length:

16 layers × 4 kv-heads × (256 + 256) × 2 bytes = 64 KB per token

A dense model with all 65 blocks attending would pay roughly 256 KB/token — four times as much. Full arithmetic and per-context tables in `CONTEXT.md`.

⚠️ The flip side, measured: the same architecture makes prefill slow in llama.cpp today. In multi-turn agent loops, where the conversation is re-processed every turn, this model spends far more time on prefill than a dense model of the same size. Cheap in memory, expensive in time.

The agentic cliff — where this model stops working

Every rung below was measured on the same objective tool-use benchmark (40 tasks, checked by running the produced artifact — no LLM judge), with the sampling profile the model declares in its own metadata (temp 1 / top_p 0.95 / top_k 20).

The threshold sits between 2.66 and 2.23 bits per weight. Above it the model completes multi-step tool tasks essentially perfectly; below it, it solves roughly one task in three — and the dominant failure is not a wrong answer, it is emitting no tool call at all.

• —Want an agent? Take `IQ2_M`. It matches the 3.40-bpw build at 28 % less size.
• —IQ2_XS is worse on a paired test (6 tasks where IQ2_M wins, 0 the other way, p = 0.031) while being only 9 % smaller.
• —The 1-bit rungs are published for completeness, not for agentic use.

Three things worth knowing about how this was measured

Single runs lie. Re-running one unchanged build through the same 40 tasks four times gives 17, 12, 16, 15 — a five-task spread, wider than most differences between builds. Every row above 2.23 bpw is at least two rounds; the 1-bit rows are four.

Fidelity does not predict capability. Against the BF16 original, the 3.40-bpw build disagrees on the next-token argmax for 9.5 % of positions and still scores 40/40. Perplexity and KL-divergence are fidelity data, not capability proxies.

Every number above is tied to a backend, not just to the file. All the rows were measured through Ollama. Re-running IQ1_M-compact through llama.cpp's llama-server — same file, same tasks, same sampling profile — scores 19/40 (47.5 %) instead of the 30.6 % listed. That is seven tasks, outside the five-task noise band above. Treat every figure here as "this build, on this runtime", and never compare it against a number someone else measured on a different one.

On IQ1_M-compact specifically

The smallest build here, 0.27 GB below the community reference. Measured: no pairwise difference is statistically significant (paired sign test, p = 0.473 vs the reference; p = 0.424 vs our larger 1-bit build) — but it ranks last on all three central estimates, and this benchmark lacks the resolution to separate builds at that size. Absence of a detected difference is not evidence of equivalence.

❌ What is NOT verified — read before relying on it

• —The 262,144 context is the model's declared native window, not one we verified. No needle-in-a-haystack run was completed.
• —Text only. The base model is multimodal; this repo ships no `mmproj`, so vision inputs will not work. Other publishers ship one — see below.
• —No fidelity table per tier. If you want a release where fidelity is measured tier by tier against the BF16 reference, see Qwen3.6-27B-GGUF.

🚀 How to run it

llama-cli -m Qwen3.8-27B-IQ2_M.gguf -c 32768 -no-cnv -p "your prompt"

-no-cnv matters: without it, recent llama-cli builds drop into conversation mode and wait on stdin, which looks exactly like a hang when scripted.

Sampling — Qwen documents two profiles, and the choice matters:

The Modelfiles in `configs/` use the thinking profile at 8K / 32K / 128K / 262K.

ollama create qwen3.8-27b-32k -f configs/Modelfile.32k

🔁 Provenance

Two lineages live in this repo. Both are faithful quantizations of the same released weights — neither is a fine-tune:

• —Q3_K_M, Q4_K_M, IQ4_XS — quantized from a local conversion of the original safetensors.
• —IQ2_M, IQ2_XS, IQ1_M-compact — quantized from a BF16 GGUF conversion published by unsloth. Those rungs build on unsloth's conversion work.

Weights are otherwise unmodified. The quantization pipeline itself is not published.

🙏 Credit & alternatives

Model, weights and training are © Qwen — Qwen/Qwen3.8-27B, Apache-2.0. This repo adds only the quantization, the context configs and the measurements.

Other GGUF builds, several more complete than this one — they ship the full ladder and the mmproj for vision: unsloth, bartowski, ggml-org, lmstudio-community.

🗒️ Changelog

• —2026-09-20 — IQ2_M re-uploaded without the inherited unsloth repo/tag fields.
• —2026-09-20 — added the backend caveat: the scores in this card were measured through Ollama, and the same IQ1_M-compact file scores 19/40 through llama-server. Seven tasks apart — the figures describe a build on a runtime, not a file.
• —2026-09-19 — IQ1_M-compact re-uploaded without the unsloth repo/tag fields it inherited from the source conversion — they credited the wrong party for this build.
• —2026-09-19 — provenance corrected: the sub-3-bit rungs come from unsloth's BF16 conversion, the others from a local safetensors conversion. The card claimed one lineage for all of them.
• —2026-09-19 — added IQ2_XS (9.09 GB) — the rung below IQ2_M, already in the cliff table, now downloadable so the threshold claim can be checked.
• —2026-09-16 — larger rungs withdrawn for re-upload with clean metadata; pipeline details removed from the card.
• —2026-09-18 — Q3_K_M, IQ4_XS and Q4_K_M back, with metadata carrying no build-machine paths.
• —2026-09-12 — added IQ2_M: the smallest rung that still drives tools (79/80).
• —2026-08-16 — first release.