Modelpublic

pearsonkyle/gemma4-e4b-coder-gguf

sourceHugging Facegemmaupdated 12d agoView on Hugging Face

0likes287downloads

Model Card

<div style="font-family: 'JetBrains Mono', 'SF Mono', Menlo, Consolas, monospace; border: 1px solid #00ff9c40; border-radius: 14px; overflow: hidden; background: #05080a; box-shadow: 0 0 40px rgba(0,255,156,0.08), inset 0 0 80px rgba(0,255,156,0.02); margin-bottom: 30px;"> <div style="background: linear-gradient(135deg, #04120c 0%, #061a2a 55%, #05080a 100%); padding: 26px; border-bottom: 1px solid #00ff9c33;"> <div style="display: flex; align-items: center; justify-content: space-between; flex-wrap: wrap; gap: 12px;"> <h1 style="margin: 0; font-size: 24px; font-weight: 700; color: #00ff9c; border: none; letter-spacing: -0.5px; text-shadow: 0 0 18px rgba(0,255,156,0.45);">▚ gemma4-e4b-coder · GGUF</h1> <span style="background: #05080a; color: #00e5ff; font-size: 11px; font-weight: 700; padding: 5px 12px; border-radius: 4px; text-transform: uppercase; letter-spacing: 1.5px; border: 1px solid #00e5ff66; box-shadow: 0 0 14px rgba(0,229,255,0.25);">IQ4XS · IQ3M · IQ2M</span> </div> <p style="margin: 12px 0 0 0; color: #6fe8b8; font-size: 12.5px; letter-spacing: 0.3px;">> llama.cpp / Ollama / LM Studio · imatrix collected on its own serving distribution, not wikitext</p> </div> <div style="display: flex; gap: 7px; flex-wrap: wrap; padding: 14px 26px; background: #070d0b; border-bottom: 1px solid #00ff9c22;"> <span style="background: #00ff9c14; color: #00ff9c; font-size: 10.5px; font-weight: 700; padding: 4px 10px; border-radius: 3px; border: 1px solid #00ff9c44;">◈ 3 rungs · imatrix-guided</span> <span style="background: #00e5ff14; color: #00e5ff; font-size: 10.5px; font-weight: 700; padding: 4px 10px; border-radius: 3px; border: 1px solid #00e5ff44;">◈ imatrix ctx 32,768 · 457 chunks · 15 M tok</span> <span style="background: #ffb70014; color: #ffb700; font-size: 10.5px; font-weight: 700; padding: 4px 10px; border-radius: 3px; border: 1px solid #ffb70044;">◈ --parse-special · tool markers intact</span> <span style="background: #ff2e8814; color: #ff2e88; font-size: 10.5px; font-weight: 700; padding: 4px 10px; border-radius: 3px; border: 1px solid #ff2e8844;">◈ 131,072 ctx · 2 GiB of KV</span> <span style="background: #a78bfa14; color: #a78bfa; font-size: 10.5px; font-weight: 700; padding: 4px 10px; border-radius: 3px; border: 1px solid #a78bfa44;">◈ vocab 65,536 · tokenizer baked in</span> </div> <div style="padding: 24px 26px; display: flex; flex-direction: column; gap: 16px;"> <div style="background: #0a1410; border-left: 3px solid #00ff9c; padding: 15px 17px; border-radius: 0 6px 6px 0;"> <h3 style="margin: 0 0 7px 0; font-size: 13px; color: #00ff9c; font-weight: 700; letter-spacing: 1px; text-transform: uppercase;">▶ What this is</h3> <p style="margin: 0; font-size: 13px; color: #b9d9cd; line-height: 1.75;">An <b style="color:#00ff9c;">imatrix-guided GGUF ladder</b> for <a href="https://huggingface.co/pearsonkyle/gemma4-e4b-coder" style="color:#00e5ff;"><b>gemma4-e4b-coder</b></a>. Trained on <b style="color:#00ff9c;">~1.1 B tokens</b> of real agentic-coding sessions, then the importance matrix was collected on <b style="color:#00ff9c;">15 M tokens of that same distribution</b> — 1.3 % of the training corpus — out to <b>32,768</b> context, so the quantization error is placed where the model is actually used. Sibling of the <a href="https://huggingface.co/pearsonkyle/gemma4-e4b-coder-gptq-w4a16" style="color:#00e5ff;"><b>W4A16 build</b></a>, same corpus: <b>that one for vLLM</b>, this one for llama.cpp.</p> </div> <div style="background: #16110a; border-left: 3px solid #ffb700; padding: 15px 17px; border-radius: 0 6px 6px 0;"> <h3 style="margin: 0 0 7px 0; font-size: 13px; color: #ffb700; font-weight: 700; letter-spacing: 1px; text-transform: uppercase;">▶ The ladder stops at 4 bits on purpose</h3> <p style="margin: 0; font-size: 13px; color: #b9d9cd; line-height: 1.75;">The ancestor is the <code style="color:#00e5ff;">-qat-q40-unquantized</code> release, whose weights were conditioned during training onto a <b style="color:#ffb700;">4-bit lattice</b>. Q5/Q6/Q8 builds would spend real bytes storing precision the weights were never trained to carry. Below 4 bits the conditioning stops helping too — IQ3M and IQ2M are coarser and non-uniform, nothing like the grid QAT targeted, so those rungs are <b>pure compression</b> and lean hardest on the imatrix coming from the real distribution.</p> </div> <div style="background: #0a1016; border-left: 3px solid #00e5ff; padding: 15px 17px; border-radius: 0 6px 6px 0;"> <h3 style="margin: 0 0 7px 0; font-size: 13px; color: #00e5ff; font-weight: 700; letter-spacing: 1px; text-transform: uppercase;">▶ The imatrix sees real tool calls, schemas and reasoning</h3> <p style="margin: 0; font-size: 13px; color: #b9d9cd; line-height: 1.75;">The calibration corpus is chat-templated, so tool schemas render as <code style="color:#00e5ff;"><|tool>declaration:…<tool|></code> and tool calls survive intact. <b style="color:#00ff9c;">--parse-special is load-bearing</b>: those markers are single special ids in this vocabulary, and without it llama.cpp would tokenize them as literal BPE text — a distribution the model never sees. Note the markers are <b>asymmetric</b>: <code style="color:#00e5ff;"><|toolcall></code> opens, <code style="color:#00e5ff;"><toolcall|></code> closes.</p> </div> <div style="background: #16070a; border-left: 3px solid #ff2e88; padding: 15px 17px; border-radius: 0 6px 6px 0;"> <h3 style="margin: 0 0 7px 0; font-size: 13px; color: #ff2e88; font-weight: 700; letter-spacing: 1px; text-transform: uppercase;">▶ Pass --jinja or tool calls will not parse</h3> <p style="margin: 0; font-size: 13px; color: #b9d9cd; line-height: 1.75;">Without <code style="color:#00e5ff;">--jinja</code> llama.cpp falls back to a generic template and the model <b style="color:#ff2e88;">never emits a parseable tool call</b>. The bundled template is the one the model was trained against, asymmetric markers and all.</p> </div> </div> </div>

▚ Quick start

bash

llama-server -hf pearsonkyle/gemma4-e4b-coder-gguf:IQ4_XS \
    -c 32768 -ngl 99 --jinja --host 0.0.0.0 --port 8000

bash

ollama run hf.co/pearsonkyle/gemma4-e4b-coder-gguf:IQ4_XS

▚ Which rung

Every file below is in this repo. The 2-bit builds are not — see Quality.

<div style="overflow-x:auto; margin: 18px 0;"><table style="width:100%; border-collapse:collapse; font-family:'JetBrains Mono', 'SF Mono', Menlo, Consolas, monospace; font-size:12.5px; background:#05080a; border:1px solid #00ff9c40; border-radius:10px; overflow:hidden; box-shadow:0 0 24px rgba(0,255,156,0.06);"><thead><tr style="background:linear-gradient(135deg,#04120c 0%,#061a2a 55%,#05080a 100%);"><th style="padding:10px 13px; text-align:left; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">file</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">GiB</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">bpw</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">trunk bpw</th><th style="padding:10px 13px; text-align:left; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">notes</th></tr></thead><tbody><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><code style='color:#00e5ff;'>gemma4-e4b-coder-BF16.gguf</code></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>9.03</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">16.01</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">16.01</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">reference (control arm)</td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><code style='color:#00e5ff;'>gemma4-e4b-coder-IQ4XS.gguf</code></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>2.68</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">4.75</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">4.36</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>recommended</b> - matches bf16</td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><code style='color:#00e5ff;'>gemma4-e4b-coder-IQ3M-awq.gguf</code></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>2.36</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">4.17</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">3.65</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">smallest usable rung; AWQ-scaled</td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><code style='color:#00e5ff;'>gemma4-e4b-coder-IQ3_M.gguf</code></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>2.36</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">4.17</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">3.65</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">baseline for the AWQ comparison</td></tr></tbody></table></div>

▚ Deployment — how many tokens you can actually hold

Only 4 of 42 layers grow their KV cache with context (the rest either reuse shared KV or run a fixed 512-token sliding window), so a sequence costs 16 KiB x ctx + 20 MiB at bf16 — 0.52 GiB at 32 K, 2.02 GiB at the full 131,072. --cache-type-k q8_0 --cache-type-v q8_0 roughly halves it. Add the rung's own file size for the total footprint:

<div style="overflow-x:auto; margin: 18px 0;"><table style="width:100%; border-collapse:collapse; font-family:'JetBrains Mono', 'SF Mono', Menlo, Consolas, monospace; font-size:12.5px; background:#05080a; border:1px solid #00ff9c40; border-radius:10px; overflow:hidden; box-shadow:0 0 24px rgba(0,255,156,0.06);"><thead><tr style="background:linear-gradient(135deg,#04120c 0%,#061a2a 55%,#05080a 100%);"><th style="padding:10px 13px; text-align:left; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">rung</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">weights</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">+ KV @ 32K</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">+ KV @ 131K</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">+ 1 GiB overhead</th></tr></thead><tbody><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ4XS</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">2.68 GiB</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">3.20 GiB</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">4.70 GiB</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">5.70 GiB</td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ3M-awq</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">2.36 GiB</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">2.88 GiB</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">4.38 GiB</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">5.38 GiB</td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ3_M</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">2.36 GiB</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">2.88 GiB</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">4.38 GiB</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">5.38 GiB</td></tr></tbody></table></div>

bash

llama-server -hf pearsonkyle/gemma4-e4b-coder-gguf:IQ4_XS \
    -c 131072 --parallel 8 -ngl 99 --jinja \
    --cache-type-k q8_0 --cache-type-v q8_0

`-c` is the TOTAL KV pool, shared across `--parallel` slots — each slot gets -c / --parallel tokens. That one knob is what decides "max tokens": context length and concurrency trade against a single budget.

Derived from `config.json`, not measured. The arithmetic is in scripts/kv_budget.py in Quant-Tuner. llama.cpp implements its own SWA-aware cache — read its startup line (KV self size = …) and trust that over the table.

▚ Quality

Use IQ4_XS. It is statistically indistinguishable from the bf16 GGUF on this suite. If you need smaller, IQ3_M-awq (2.36 GiB) is the smallest rung that still calls tools reliably; plain IQ3_M ships as its baseline.

The 2-bit rungs are measured here but not uploaded. IQ2_M scores 0.0000 on every metric — 102 of 107 turns emit no tool call at all — and AWQ scaling does not change that. They are reported in full below so the ladder's floor is visible, but shipping 1.84 GiB of a build that cannot call a tool would only invite someone to download it.

Every arm is scored through llama-server on the same 107 held-out turns, and so is a bf16 GGUF — the control. That control is not ceremony: the same bf16 weights score 0.8692 in the transformers harness and 0.9065 here, a 3.7-point gap on identical weights from the stack alone. That is the same size as the effects being measured, so every rung is read against the bf16 row in this table and never against the base model's published numbers.

<div style="overflow-x:auto; margin: 18px 0;"><table style="width:100%; border-collapse:collapse; font-family:'JetBrains Mono', 'SF Mono', Menlo, Consolas, monospace; font-size:12.5px; background:#05080a; border:1px solid #00ff9c40; border-radius:10px; overflow:hidden; box-shadow:0 0 24px rgba(0,255,156,0.06);"><thead><tr style="background:linear-gradient(135deg,#04120c 0%,#061a2a 55%,#05080a 100%);"><th style="padding:10px 13px; text-align:left; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">model</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">schema-valid</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">tool selection</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">param acc</th><th style="padding:10px 13px; text-align:left; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">paired vs BF16</th></tr></thead><tbody><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>BF16</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.9065</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.6636</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.4905</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">control · 107 turns</td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ4XS</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.9065</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.6636</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.4415</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">schema 8 disc. (4/4) p=<b style='color:#00ff9c;'>1.00</b> · selection 18 disc. (9/9) p=<b style='color:#00ff9c;'>1.00</b></td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ3M-awq</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.8224</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.6449</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.4065</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">schema 27 disc. (18/9) p=<b style='color:#00ff9c;'>0.12</b> · selection 28 disc. (15/13) p=<b style='color:#00ff9c;'>0.85</b></td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ3M</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.7009</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.5888</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.3562</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">schema 32 disc. (27/5) p=<b style='color:#00ff9c;'><0.001</b> · selection 24 disc. (16/8) p=<b style='color:#00ff9c;'>0.15</b></td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ2M</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.0000</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.0096</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.0000</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">schema 97 disc. (97/0) p=<b style='color:#00ff9c;'><0.001</b> · selection 72 disc. (71/1) p=<b style='color:#00ff9c;'><0.001</b></td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ2_M-awq</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.0000</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.0000</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.0000</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">schema 97 disc. (97/0) p=<b style='color:#00ff9c;'><0.001</b> · selection 71 disc. (71/0) p=<b style='color:#00ff9c;'><0.001</b></td></tr></tbody></table></div>

Degradation is not gradual — it fails in two stages. The failure reasons say more than the rates do:

That is why IQ3M's *selection* p-value is an unremarkable 0.15 while its *schema* p-value is below 0.001: it has not forgotten which tool to reach for, it has lost the ability to fill the call in. IQ2M's 97 discordant schema turns split 97/0 — there is not one turn where it succeeded and bf16 did not.

IQ2_M also destabilised the server: 3 of 36 sessions ended in API errors (104 of 107 turns scored, against 107 for every other arm). Its row is reported over the turns it completed; treat the rate as generous rather than harsh.

AWQ scaling: a real gain at 3-bit, nothing at 2-bit

Both sub-4-bit rungs were also built with AWQ per-channel scaling folded in before quantization, each re-calibrated with its own imatrix over the same 15 M-token corpus at the same context — so the head-to-head isolates the scaling. Both AWQ builds are byte-identical in size to their plain twins, so nothing here is bought with bits.

At 3-bit AWQ works, and significantly. It recovers 21 turns the plain rung fails while losing 8, p = 0.024. That closes most of the distance to the reference: against the bf16 GGUF, IQ3_M-awq is no longer significantly worse on schema validity (27 disc, 18/9, p = 0.12, versus p < 0.001 for the plain rung) and is at parity on tool selection (15/13, p = 0.85). Parameter accuracy is the one place a real gap survives: −0.084, 95 % CI [−0.160, −0.007], which excludes zero.

Against IQ4_XS it shows no detectable difference on any metric (schema p = 0.12, selection p = 0.86, param CI spans zero) — but that is a statement about this suite's power at n = 107, not proof of equivalence, and schema is directionally behind (18 turns to 9). IQ4XS remains the recommendation because it *matches* bf16 exactly; IQ3M-awq is the choice when 0.32 GiB matters more than that certainty.

At 2-bit AWQ changes nothing — not one of 107 turns differs, and 106 of them emit no tool call at all. This is a trustworthy null: the fold was verified exact to rel = 1.0e-05 before quantization (an earlier attempt folded in bf16 at rel = 0.189, six times over awq.apply's own 0.03 gate, and its results were withdrawn). The reason is structural. AWQ folds 84 groups — 42 layers × (attn, mlp), the entire trunk and nothing else. At IQ2_M the vocabulary tensor is 0.56 GiB, 30.3 % of the file, and AWQ never touches it. A trunk-only intervention cannot repair damage concentrated in an embedding table, and the QAT conditioning that makes 4-bit nearly free offers nothing this far below its lattice. Tellingly, the better-folded build rambles more, not less: turns running to the token cap went 43 (plain) → 59 (bf16 fold) → 67 (fp32 fold).

Neither 2-bit build is shipped. Plain IQ2M and IQ2M-awq both score 0.0000 and are absent from this repo by choice; their per-turn results are in eval/toolcall.csv if you want to verify the floor yourself. Both imatrices are published, so either can be rebuilt locally with llama-quantize --imatrix.

What this does not say. The suite resolves differences of roughly ±0.09 at n = 107, so "IQ4_XS is indistinguishable from bf16" means exactly that and not "lossless" — its parameter accuracy is 0.4415 against 0.4905, a gap this holdout cannot separate from noise but which is not evidence of no gap.

<div style="overflow-x:auto; margin: 18px 0;"><table style="width:100%; border-collapse:collapse; font-family:'JetBrains Mono', 'SF Mono', Menlo, Consolas, monospace; font-size:12.5px; background:#05080a; border:1px solid #00ff9c40; border-radius:10px; overflow:hidden; box-shadow:0 0 24px rgba(0,255,156,0.06);"><thead><tr style="background:linear-gradient(135deg,#04120c 0%,#061a2a 55%,#05080a 100%);"><th style="padding:10px 13px; text-align:left; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">model</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">schema-valid</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">tool selection</th><th style="padding:10px 13px; text-align:right; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">param acc</th><th style="padding:10px 13px; text-align:left; color:#00ff9c; font-weight:700; font-size:10.5px; text-transform:uppercase; letter-spacing:1px; border-bottom:1px solid #00ff9c33; white-space:nowrap;">paired vs BF16</th></tr></thead><tbody><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>BF16</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.9065</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.6636</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.4905</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">control · 107 turns</td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ4XS</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.9065</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.6636</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.4415</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">schema 8 disc. (4/4) p=<b style='color:#00ff9c;'>1.00</b> · selection 18 disc. (9/9) p=<b style='color:#00ff9c;'>1.00</b></td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ3M</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.7009</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.5888</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.3562</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">schema 32 disc. (27/5) p=<b style='color:#00ff9c;'><0.001</b> · selection 24 disc. (16/8) p=<b style='color:#00ff9c;'>0.15</b></td></tr><tr><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;"><b style='color:#00ff9c;'>IQ2_M</b></td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.0000</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.0096</td><td style="padding:9px 13px; text-align:right; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">0.0000</td><td style="padding:9px 13px; text-align:left; color:#b9d9cd; border-bottom:1px solid #00ff9c14; vertical-align:top;">schema 97 disc. (97/0) p=<b style='color:#00ff9c;'><0.001</b> · selection 72 disc. (71/1) p=<b style='color:#00ff9c;'><0.001</b></td></tr></tbody></table></div>

<details> <summary><b>▶ How the quality numbers were measured</b></summary>

Every rung is scored through llama-server, and so is a bf16 GGUF — because the published bf16 numbers for this model came from the in-process transformers harness, and those are different stacks. A reference model can score differently on each for reasons that have nothing to do with quantization, so each rung is read against the bf16 GGUF measured here, never against the number on the base model's card. Cross-stack comparisons are not valid and are not published as if they were.

Three defaults would otherwise have made these incomparable with the base model's table, so they are pinned: --no-stop-on-fail (the default stops a session at its first wrong tool, giving each arm a different denominator — that is how an earlier run scored 32 and 70 turns against a 107-turn table), --ctx 65536 (the default 8192 truncates these sessions, the most likely cause of a spurious zero), and --max-tokens 2048.

Sessions come from the test slice of the seeded split — never used for training or calibration.

</details>

<details> <summary><b>▶ How it was made</b> — imatrix, the converter patch, calibration corpus</summary>

imatrix at ctx 32,768, matching the corpus's pack context. A corpus packed at 32 K and measured at 512 would split nearly every session, and the statistics would then describe fragments — and a middle fragment has no system prompt and no tool schemas, a context that never occurs at inference. The -ctx32k calibration split admits only conversations that fit whole inside 32,768 tokens: 457 chunks over 15,001,898 tokens.

This model has no separate `output.weight`. tie_word_embeddings is true, so the GGUF carries one token_embd.weight doing double duty as input lookup and output projection — verified directly against the bf16 GGUF's tensor list. --process-output therefore has nothing to act on here, and the vocabulary tensor is quantized without imatrix guidance (llama.cpp holds it at higher precision than the trunk for exactly that reason).

What the imatrix does cover is 342 tensors — every quantizable matmul in the trunk. A detail worth reading off it: attn_k and attn_v appear for only 24 of 42 layers, not all 42, because layers 24–41 are num_kv_shared_layers and carry no key/value projection weights at all.

Calibration: the same corpus as the W4A16 build — 11,982 conversations, token-balanced with no source above 6 %, drawn only from the train slice of the seeded split so the evaluation holdouts stay clean. 12,175 tool calls and 2,940 tool-schema declarations survive templating. Corpus SHA-256 is recorded alongside the artifacts.

The converter needed a patch. transformers 5.16.1 writes Gemma 4's attention geometry as per_layer_config: {"05": {"head_dim": 512}, …} on exactly the 7 full_attention layers, where 5.6.2 wrote a single global_head_dim: 512. llama.cpp's gemma.py reads the old key and dies with KeyError: 'global_head_dim'. Patching config.json does not fix it — AutoConfig.from_pretrained().to_dict() normalises the old key back into the new form, so the converter never sees it. The fix derives the value from per_layer_config, verifying the keys are exactly the full_attention layers and that they agree on one head_dim. Patch: patches/llamacpp-gemma4-global-head-dim.patch in Quant-Tuner.

Built with Quant-Tuner against llama.cpp f3e1828.

</details>

<details> <summary><b>▶ The restricted vocabulary, and what every quant must preserve</b></summary>

This family does not use Gemma 4's 262,144-token vocabulary. It is pruned to 65,536 tokens chosen by frequency over the training corpus, which removes 5.24 GB of vocabulary-indexed weights before a single layer is quantized:

Two consequences bind every quantization of it:

1.The tokenizer ships with the model and is not interchangeable. Each GGUF carries its own, so the normal path is safe — but feeding it ids from a stock Gemma 4 tokenizer produces fluent-looking nonsense with no error.
2.The output head stays at high precision. A quantized head over a vocabulary is the classic rare-token failure mode, and a pruned vocabulary concentrates more meaning per row — dropped tokens byte-fall-back into surviving pieces, so those rows carry more of the distribution than they did before.

</details>

<details> <summary><b>▶ Stop tokens</b></summary>

Decoding must halt on <eos>, <turn|> and <|tool_response> — the last is how the model yields after a tool call. Each GGUF carries all three, so the normal path is correct; a client that overrides them with <eos> alone will see the model keep decoding, write the tool's output itself, and call another tool. That reads as a broken quant but is a client config.

</details>

<details> <summary><b>▶ Limits</b></summary>

—Text only. The vision and audio towers were removed when the vocabulary was pruned.
—Not suitable for multilingual use — non-Latin scripts fall back toward per-byte encoding under the pruned vocabulary.
—IQ2_M shows where the floor is, it is not a recommendation. Read its row in the quality table before deploying it.
—Perplexity is not the headline here. llama-perplexity has no --parse-special, so on a chat-templated eval file the control markers tokenize as plain BPE and absolute perplexity is off-distribution. Quant-vs-quant comparisons on the same file stay valid, which is why KLD leads and PPL does not.
—Inherits every limitation of the base model, including that its held-out decision accuracy never improved during training — only format did.

</details>

▚ Family

▚ License

Gemma Terms of Use, inherited from google/gemma-4-E4B-it.