ManniX-ITA/gemma-4-A4B-98e-v7-coderx-it-GGUF

gemma-4-A4B-98e-v7-coderx-it-GGUF

GGUF quantizations of ManniX-ITA/gemma-4-A4B-98e-v7-coderx-it.

All quants made using imatrix with calibration data v5. The imatrix.dat used is included in this repo for reproducibility/audit; mmproj-gemma4.gguf is the shared Gemma 4 SigLIP vision tower (untouched by pruning) for multimodal use.

Quantizations — score, size & answer length

Every published K-quant and CD tier was scored on HumanEval+ (164) and MultiPL-E-100 (llama.cpp, greedy T=0), with per-problem completion length from token_stats. The table replaces a bare file-list — it shows, per tier, the actual code score, the exact size, the true bits-per-weight (bpw = 8 × bytes ÷ 19,877,953,946), and whether the tier stays length-tight or starts to ruminate at low bit. ⭐ marks a recommended pick.

Recommended picks:

• —Q6_K ⭐ (17.81 GB) — max-fidelity tier — 92.07% HE+ / 90.33% MPE-100.
• —Q4_K_S ⭐ (12.21 GB) — best K-quant — 93.29% HE+, the highest of any tier.
• —IQ4_NL ⭐ (11.42 GB) — best compact — 91.33% MPE-100 (highest of any tier) + 91.46% HE+.
• —CD-Q2_K ⭐ (8.82 GB) — smallest tier — 89.02% HE+ at the lowest disk.

Reads: the K-quant and CD tiers hold HE+ in the ~89–93% band and MPE in the 87–91% band with completion length essentially identical to Q6K. The **2-bit `Q2KL` / `IQ2XS** tiers are the cliff — HE+ falls into the 80s/70s and the token p90 blows out (budget saturation / rumination), so the **K-quant CD tier(s)** (CD-Q2_K) are the recommended low-bit path. The **i-quant CD bodies (CD-IQ`) are not offered* — the pruned MoE degenerates on an IQ-family body even with critical-tensor protection (token-salad, score → 0).

Head-to-head by file size — v7-coderx vs Qwen2.5-Coder-14B (iso-disk)

Pairing by tier name is misleading — v7-coderx is a ~20.8B-total MoE and Qwen2.5-Coder-14B is a 14.7B dense model, so the same tier name lands at a different file size. The fair comparison is iso-disk: at a given GB budget, which model scores higher on HumanEval+? Both are measured on the identical rig (single RTX 3090 24 GB, llama-server -ngl 99 --parallel 2, omkeval llama backend, `humanevalplusfull, greedy T=0). Qwen GGUFs are bartowski's [Qwen2.5-Coder-14B-Instruct-GGUF`](https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF); its ladder sits at 83–85% across the whole stack. At every band below the MoE runs lower bpw at the same disk and still scores higher.

Reads:

1. 1.Iso-disk ~10.5 GB. v7-coderx Q3_K_M (10.51 GB / 4.23 bpw / 92.07%) vs Qwen Q5_K_M (10.51 GB / 5.72 bpw / 83.54%): +8.53pp at the exact same file size, −1.49 bpw.
2. 2.Sub-9 GB code-grade. CD-Q2_K (8.82 GB / 3.55 bpw / 89.02%) clears Qwen's whole ladder — its best tier tops out at 85.37% (Q4KM, 8.99 GB), so the MoE holds a code-grade ~89% HE+ ~0.2 GB smaller and ~0.87 bpw lower.
3. 3.Every band wins at lower bpw. Across the ladder the MoE uses 1.5–4 bpw less than the dense Qwen tier at the same disk and still scores higher on HumanEval+ — the point of the iso-disk framing.

CD recipes are open-source — generator at omnimergekit/scripts/generate_cd_maps.py.

How to Use

With llama.cpp:

llama-server -m gemma-4-A4B-98e-v7-coderx-it-Q4_K_M.gguf -c 32768 -ngl 99 \
    --jinja \
    --reasoning-budget 8192 \
    --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.05 \
    --repeat-penalty 1.02 --repeat-last-n 2048

The --reasoning-budget flag is required for Gemma 4 thinking — without it the model emits malformed channel tokens. Keep -c several times larger than the reasoning budget — with -c equal to the budget the thinking phase can fill the whole window and the answer degenerates. See Reasoning budget and thinking stop phrase below for the budget, the wrap-up phrase that stops reasoning leaking into the answer, and why --reasoning-format plays no part in it.

Recommended sampling — --repeat-penalty 1.02 --repeat-last-n 2048 is the field-tested sweet spot for long agentic / tool-calling sessions (validated with opencode driving multi-turn coding work): it suppresses the intermittent repetition loops without side effects. Stronger penalties (1.05–1.1) also stop the loops but cause premature end-of-turn — the model announces a step and then stops mid-task. Narrower windows (64–1024) at 1.02 let long-period loops through; keep the full 2048. The remaining sampler values above are the Gemma 4 vendor defaults baked into the GGUF, stated explicitly so CLI defaults don't override them.

With ollama: ollama pull mannix/gemma4-98e-v7-coderx:Q4_K_M (:latest = Q4KM; :vision-<tier> pairs the tier with the SigLIP vision tower).

Reasoning budget and thinking stop phrase (llama.cpp)

On a hard prompt this model will reason until it has consumed the whole context window and then answer with nothing at all. llama.cpp can bound the thinking block with a sampler, and — the part that actually matters — tell the model why the block is being closed.

Needs llama.cpp b8508 or newer for the flags, b10091 or newer for the per-request overrides.

Serve with a bounded thinking block

llama-server -m gemma-4-A4B-98e-v7-coderx-it-Q4_K_M.gguf -c 32768 -ngl 99 \
    --jinja \
    --reasoning-budget 8192 \
    --reasoning-budget-message $'\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n' \
    --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.05 \
    --repeat-penalty 1.02 --repeat-last-n 2048

Both flags also read from the environment: LLAMA_ARG_THINK_BUDGET and LLAMA_ARG_THINK_BUDGET_MESSAGE.

--reasoning-format is not part of this. It only decides how the thinking is handed back — message.reasoning_content versus left inline in message.content — and never whether the budget is enforced: the delimiters the sampler counts are set by the chat template regardless, so the cap binds under auto, deepseek and none alike. The default auto already extracts reasoning and is behaviourally identical to deepseek (they differ only in name; the sole branch in the parser is != none). Leave it at the default so the model's own tool-call and channel handling stays in play, and pin deepseek only when a harness needs the thinking kept out of content.

--reasoning-budget on its own forces the closing tag the moment the budget runs out, wherever the model happens to be. When that lands mid-thought the model frequently does not register that it was interrupted: it carries on reasoning, now inside the visible answer. The stop phrase is what prevents that — it gives the model a reason to be finishing.

Two wordings that work

# "qwen" — the string Qwen's own service uses, from their docs
--reasoning-budget-message $'\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n'

# "voice" — shorter, in the model's own reasoning voice
--reasoning-budget-message $'\n\nOK, I have enough to answer now.\n'

Wording is model-specific: Qwen note that the ability to act on such a message "is not explicitly trained but emerges naturally", so it is worth trying both on your own workload. Leading and trailing newlines matter — they keep the phrase off whatever half-finished line the cut landed on.

What it measures out to

Measured on the v7-coder IQ4_NL build of this family, served by the same llama.cpp sampler. Three hard questions, temperature 0.6, fixed seed, answer characters with wall time in brackets. Every run answered all three correctly, and thinking length is unchanged by the message in every row:

AIME 2024, all 30 problems, budget 4096, -c 32768, vendor sampling:

The phrase halves wall time and all but removes the runaway answers — single rows go from 82,067 characters of answer to 1,655. The accuracy differences are inside the noise at n = 30 (paired: qwen −4 net, voice −1 net, exact binomial p ≈ 0.22 and ≈ 1.0), and the terse "Final Answer:" suffix from the s1 paper (arXiv:2501.19393) is not reproducing the accuracy collapse reported there at this budget.

Per request, instead of per server

The server accepts both as request fields, overriding the command line:

{
  "messages": [ ... ],
  "thinking_budget_tokens": 8192,
  "reasoning_budget_message": "\n\nOK, I have enough to answer now.\n"
}

On the raw /completion endpoint the delimiters are not inferred, so they have to be supplied with the budget:

{
  "prompt": "...",
  "reasoning_budget_tokens": 8192,
  "reasoning_budget_start_tag": "<|channel>",
  "reasoning_budget_end_tag": "<channel|>",
  "reasoning_budget_message": "\n\nOK, I have enough to answer now.\n"
}

On b10091 the message field must be present on /completion requests even when empty: llama.cpp builds the sequence it forces from message + end_tag inside that field's handler, so omitting it leaves the budget with nothing to force — the sampler logs as though the cap fired while the thinking block stays open.

Rules of thumb

• —Keep -c several times larger than the budget. A budget equal to the context lets the thinking phase fill the window on its own.
• —A quarter of the context is a sensible starting point: 8192 at -c 32768.
• —The budget is per thinking block, not per response — the sampler re-arms when it sees a new opening tag, so a multi-turn agent gets a fresh window each time.

Chat template & end-of-turn tokens (metadata refresh, 2026-07-30)

Every tier in this repo has been re-uploaded with corrected metadata — the rollout completed 2026-07-30 and covers all 14 tiers: Q8_0, Q6_K_L, Q6_K, Q5_K_L, Q5_K_M, Q4_K_L, Q4_K_M, Q4_K_S, IQ4_NL, IQ4_XS, Q3_K_L, Q3_K_M, CD-Q2_K, F16. Every tier was verified after patching (template 19,177 B / md5 8119c2dc…, eos=106, eot=1, bos=2, tool-call macros present, tensor count unchanged) and the published file's sha256 re-checked against the patched file. The matching mannix/gemma4-98e-v7-coderx ollama tags were re-pushed as well — all 13 text tiers and their 13 vision-<tier> counterparts (F16 is HF-only, by design).

The tensor payload is byte-identical — only the GGUF KV header changed — so sizes and scores are unaffected, but you need to re-download to pick the fix up. A local file still reporting eos_token_id = 1 (gguf_dump / llama-server startup log) is a stale copy from before the refresh.

1. Chat template — agentic loop fix, rebased onto Google's current template. The embedded tokenizer.chat_template is now our loop fix applied on top of Google's current Gemma 4 template (upstream revision 2026-07-20, 18,683 B). The result is 19,177 B, md5 8119c2dcd5e62a4a6b79301ab13ac81d, and it is also published at the repo root as `chat_template.fixed.jinja` for anyone serving the bf16 weights.

The bug it fixes: the stock template re-injects earlier assistant turns' `thinking` content back into the prompt on every turn. In long agentic / tool-calling sessions that feeds the model its own reasoning back to itself and drives the repetition loops. Google's current 18,683 B template is still affected — its thinking gate carries an unconditional "index past the last user message" disjunct — so moving to a fresh upstream template does not remove the need for this fix. The rebase leaves Google's newer preserve_thinking flag intact (default false).

`llama.cpp` only uses the embedded template when you pass `--jinja` (added to the command above). Without it llama.cpp falls back to its own built-in Gemma 4 formatter and the fix does not apply. The recommended sampler above is still recommended — the template fix and --repeat-penalty 1.02 address different halves of the loop problem.

2. End-of-turn tokens. eos_token_id is now 106 (<turn|>) and eot_token_id 1 (<eos>), which gives llama.cpp the end-of-generation set {106, 1, 50} — matching generation_config.json ([1, 106, 50]) in the source repo. Previously every tier shipped eos_token_id = 1 with no eot, so stopping depended on stop strings; the model can now end its turn on its own token.

ollama caveat. ollama ≥ 0.32 formats Gemma 4 with a compiled-in `gemma4` renderer and parser and never reads the GGUF's jinja template, so the mannix/gemma4-98e-v7-coderx tags receive the EOG fix only. That is expected, not a defect: ollama show --modelfile printing TEMPLATE {{ .Prompt }} next to RENDERER gemma4 / PARSER gemma4 is the correct state. For the template fix, serve the GGUF with llama.cpp --jinja.

Gemma 4 A4B 98-Expert v7-coderx — code-maximal prune (~20.8B)

Eval complete (Q6_K / llama.cpp, greedy, same host). Every cell in the scoreboard is read from summary.json under the cohort-pinned greedy recipe (temperature 0.0, top_p 1.0, top_k 0). The 128e, v6-coder and v7-coder columns are the matching same-host Q6K runs. The GGUF and NVFP4A16 formats are deployment targets and are **not** separately benchmarked (cohort policy) — the Q6K column is representative. Headline — the cohort's code specialist. v7-coderx spends its whole prune budget on code and short-form reasoning. On the all-hard LiveCodeBench-77 set — the most demanding and most discriminating LCB slice — it scores 85.71%, the highest in the cohort (128e 79.22%, v7-coder 84.42%), and it leads on HumanEval+ 93.29%, HumanEval 96.95%, MATH-500 95.0% and AIME 76.67%. On the easier LCB-medium-v4 slices it sits a little below the generalists (LCB-55 92.73 / LCB-100 91.0 vs 128e's 96.36 / 97.0) — those slices no longer separate the strongest coders. This is also the loop-fixed build: it force-keeps the agentic loop-protection experts and replaces the earlier looping `fs2440` prune. The trade is graduate science: GPQA-diamond sits at 51.01% (this recipe carries no targeted_gpqa term). For the broader LCB-medium lead and HumanEval, see the sibling **v7-coder** (LCB-55-v4 98.18%, HE 98.17%, LCB-hard-77 84.42%; GPQA ≈ 51 like this model).

A research checkpoint that prunes the unpruned Gemma 4 26B-A4B-it (128 experts/layer, top-8 + shared, 30 layers) down to 98 experts per layer. The drop map concentrates protection on generic-code and LiveCodeBench-medium with no science or multilingual targeting — the code-maximal member of the v7-coder cohort — and force-keeps the agentic loop-protection experts so the served model does not loop. Same 98e shape, same router, same attention, same norms as the rest of the cohort, plus the mandatory shared-FFN α=1.2 upweight all coder variants carry.

Benchmarks

Q6_K · llama.cpp · greedy (temperature 0.0, top_p 1.0, top_k 0), all four models scored on the same host from summary.json. Row-max in bold. This repo = v7-coderx.

_{Metrics: GPQA & GSM8K = exact_match flexible-extract · MATH500 = math_verify · ARC & AIME = exact_match · IFEval = prompt_level_strict_acc · HumanEval/+ = pass@1 chat-extract · LCB-55/100 & MultiPL-E = pass@1. 128e uses the lcb_medium_55/100 templates; the prunes use lcb_medium_*_v4 (corrected harness, equivalent task). The all-hard LCB-77 cross-model comparison is in the table below.}

v7-coderx leads the cohort on the hardest code slice (LCB-hard-77, below) and on HE+ / MATH-500 / AIME; the budget is paid on graduate science (GPQA) and on the easier instruction / ARC axes, which carry no protection term in this recipe.

Coder-field comparison — v7-coderx vs Qwen2.5-Coder-14B / 7B + Qwen3.5-9B (Q6_K, llama.cpp, greedy)

The 9 canonical benches + MultiPL-E-100, all on the identical llama.cpp Q6_K / greedy recipe (reasoning models served with --reasoning-format deepseek --reasoning-budget 12288 --parallel 2). Architectures differ — this is a same-harness comparison, not a same-class one:

• —v7-coderx — Gemma-4 26B-A4B MoE pruned to 98 experts (~20.8B total, ~A4B active), reasoning.
• —Qwen2.5-Coder-14B / 7B-Instruct — dense, non-reasoning code specialists (bartowski Q6_K).
• —Qwen3.5-9B — dense reasoning model (bartowski Q6_K).

† Qwen2.5-Coder-14B HumanEval+ / LCB-medium-55 are the same-stack GGUF HE+ sweep numbers (not re-run in this chain). All Qwen cells are the same-host reference runs used on the v6-coder card — Qwen is a fixed reference, so the columns are identical across the cohort; only the Gemma column changes.

Note on Qwen3.5-9B. Qwen3.5-9B is a verbose, slow thinking model: it emits long <think> reasoning chains (often ≥1900 tokens even on a trivial GSM8K question), so it runs several× slower per question than the non-reasoning Qwen2.5-Coder models — well beyond what its 9B size would suggest. Its GSM8K / MATH-500 / GPQA cells were re-run after a harness fix (under batched, reasoning-parsed serving the verbose thinking intermittently left the final answer inside the reasoning block, mis-scored as empty content).

LiveCodeBench across problem sets

v7-coderx's code score depends on the LiveCodeBench slice. All cells are the same greedy Q6_K / imat-Q6 llama.cpp stack (build provenance verified per run); v4-55/100 mirror the 9-bench above. The all-hard 77q set is the most demanding and the most discriminating across the cohort.

† LCB-v6-55 is a small, noisier 55-problem slice (no greedy 128e baseline was run); it is included for completeness, but all-hard 77q is the reference for cross-model comparison.

Answer-length analysis (anti-rumination)

The pruned reasoning model thinks with a bounded thinking_token_budget=12288; the question is whether that length is productive (long thinking that PASSes) or rumination (long thinking that fails). Per-problem completion length is measured from omkeval `tokenstats (characters from the raw completion; tokens via the 128e tokenizer) on the real-n` benches, against 128e and v6-coder on the same problems, same greedy Q6_K / llama.cpp stack.

Per-problem completion length — characters (p50 / p90 / max):

Per-problem completion length — tokens (p50 / p90 / max):

Budget-saturation incidence — share of problems whose completion reached ≥12k tokens (at/near the thinking_token_budget=12288 cap). Saturation by itself is not rumination — a saturated output that PASSes is productive use of the budget; the pruned reasoning model saturates on nearly every LCB problem, 128e almost never does.

Rumination — long thinking that fails to PASS. The right metric is not median length (128e looks short only because it answers easy problems fast). It is the share of the model's budget-saturated outputs that still fail — tokens burned without a correct answer:

Key findings:

• —128e only thinks long when it is lost. Every 128e output that reaches the budget cap is a failure (1/1 on LCB-55, 2/2 on LCB-100), and its failed problems run several× longer than its passed ones (mean 13782 vs 1392 tok on LCB-100).
• —v7-coderx's long thinking is overwhelmingly productive. It saturates on ~97% of LCB-100 problems but only 9/97 of those saturated outputs fail (9.3%); its PASS and FAIL completions are nearly the same length (mean 12623 vs 15143 tok), so failures are not driven by extra rumination. On LCB-55 it is 4/54 saturated-and-failed.
• —Comparable to v6-coder's rumination rate. v6-coder ran 4/54 (LCB-55) and 6/98 (LCB-100) saturated-and-failed; v7-coderx is at or below on LCB-55 (4/54) and near on LCB-100 (9/97). The saturated-fail share tracks the model's LCB pass-rate — these are the genuinely hard problems, not extra rumination (PASS and FAIL completions are near-equal length).
• —Non-LCB benches stay tight. On the short-answer benches (GSM8K / MATH-500 / HE / HE+ / MultiPL-E) p50/p90 length tracks 128e and v6-coder within a few tokens — the targeted prune did not trade length for accuracy on the everyday benches.

Methodology. Per-problem lengths come from omkeval `tokenstats over each bench's samples*.jsonl` / `lcbresult.samples.jsonl; saturation/PASS-FAIL is computed per problem from completion_tokens + passed. **MultiPL-E measures code length, not reasoning** (its samples store only the final code block, no <think>` trace), so it is a code-conciseness reference rather than a thinking-length signal.

Recipe (summary)

98e prune from 128e on the v7 competence map with the generic-code and LiveCodeBench-medium classes up-weighted (no science or multilingual targeting), then the mandatory shared-FFN α=1.2 upweight, plus an agentic loop-protection force-keep that pins the experts emitting end-of-turn / answer-channel tokens. This is the published, loop-fixed build — it replaces the earlier looping fs2440 prune. v7-coderx is the code-maximal sibling of v7-coder — same map, minus the targeted_gpqa science term. Full recipe and calibration-class table are on the bf16 card.

Intended use & limitations

Compact (~12 GB at Q4_K_M, single 12–16 GB GPU) Gemma 4 checkpoint for maximal coding throughput and instruction-following. Serve with the reasoning parser enabled. A research prune, not an official Google release; generic_multilingual is de-weighted (0×) and graduate science (GPQA) is the explicit budget axis — at 51.01% it is well below the unpruned 128e (67.17%). Choose v7-coder if science matters. Prefer Q4KM or higher.

Lineage

128e → (v4 → v5 → v6-coder code line) → v7 competence-map rebuild → code floor + agentic loop-protection force-keep = v7-coderx. Built and evaluated on the omnimergekit toolchain.