ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF

Qwen3.6-27B-A3B-Coder

A code-specialist expert prune of Qwen3.6-35B-A3B: the MoE is reduced from 256 experts to 184 (72 dropped per layer, ~35B→27B, still A3B active) using a code-targeted competence map (LiveCodeBench + MultiPL-E competence classes). Same router, attention, norms, MTP head and vision tower as the base — only the expert keep-set changes.

Served at top-10 (num_experts_per_tok = 10, baked as the default). This is a routing-recovery lever: after pruning to 184 experts, activating the top-10 (vs the base top-8) recovers instruction-following at no cost to code (see below). No fine-tuning, no distillation — pure expert selection + a routing-width dial.

Recipe

1. 1.Competence map: the 256e teacher is profiled per-expert on a balanced corpus + targeted LiveCodeBench and MultiPL-E (Rust/Java/JS) PASS-response classes.
2. 2.Drop map: wmax aggregation with the LCB + MPE classes up-weighted (1.5) → 72/256 experts dropped per layer, protecting the code-competent experts.
3. 3.Top-10 routing (num_experts_per_tok = 10) baked into the config → the shipped default. Pass --override-kv qwen35moe.expert_used_count=int:8 to any llama.cpp tool to A/B back to native top-8.

Evaluation (Q6_K, llama.cpp, temp 0.6 / top-p 0.95 / top-k 20)

Highlights: best code profile of any prune — MultiPL-E 0.840 (above the teacher; +17pp over the LCB-only coder that this model supersedes), LiveCodeBench 0.688 (tied best), HumanEval 0.970. Average 0.808 sits at the LCB-coder level and within 0.03 of the full teacher.

Verbosity / rumination (length breakdown per eval)

Aggressive expert pruning makes the model verbose on open-ended reasoning — it over-thinks before answering. This is largely inherited from the base (the 256e teacher does the same on GPQA/AIME) and is bounded by the generation cap; it does not affect the code benches, which have a natural termination anchor.

Response length in characters (content + reasoning), this model vs the 256e teacher; runaway = responses > 20k chars (of 100, or 30/198 for GPQA, 30 for AIME):

Reading it: GPQA/AIME verbosity is essentially the base model (58 vs 58, 25 vs 29). Only IFEval shows prune-added rumination (30 vs 11) — the trade for the code-targeted drop map. Code and math-with-boxing tasks terminate cleanly. If you want tighter output, a repetition/length penalty at serve time (or top-8 via the override above) reduces the tail.

Reasoning budget and thinking stop phrase (llama.cpp)

Qwen 3.6 reasons at length by design, and on a hard prompt it can consume the whole context window before it answers. llama.cpp can bound the thinking block with a sampler, and — the part that actually matters — tell the model why the block is being closed.

Needs llama.cpp b8508 or newer for the flags, b10091 or newer for the per-request overrides.

Serve with a bounded thinking block

llama-server -m Qwen3.6-27B-A3B-Coder-Q4_K_M.gguf -c 32768 -ngl 99 \
    --jinja \
    --reasoning-budget 8192 \
    --reasoning-budget-message $'\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n' \
    --temp 0.6 --top-k 20 --top-p 0.95

Both flags also read from the environment: LLAMA_ARG_THINK_BUDGET and LLAMA_ARG_THINK_BUDGET_MESSAGE.

--reasoning-format is not part of this. It only decides how the thinking is handed back — message.reasoning_content versus left inline in message.content — and never whether the budget is enforced: the delimiters the sampler counts are set by the chat template regardless, so the cap binds under auto, deepseek and none alike. The default auto already extracts reasoning and is behaviourally identical to deepseek (they differ only in name; the sole branch in the parser is != none). Leave it at the default so the model's own tool-call and channel handling stays in play, and pin deepseek only when a harness needs the thinking kept out of content.

--reasoning-budget on its own forces the closing tag the moment the budget runs out, wherever the model happens to be. When that lands mid-thought the model frequently does not register that it was interrupted: it carries on reasoning, now inside the visible answer. The stop phrase is what prevents that — it gives the model a reason to be finishing.

Two wordings that work

# "qwen" — the string Qwen's own service uses, from their docs
--reasoning-budget-message $'\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n'

# "voice" — shorter, in the model's own reasoning voice
--reasoning-budget-message $'\n\nOK, I have enough to answer now.\n'

Wording is model-specific: Qwen note that the ability to act on such a message "is not explicitly trained but emerges naturally", so it is worth trying both on your own workload. Leading and trailing newlines matter — they keep the phrase off whatever half-finished line the cut landed on.

What it measures out to

Measured on the Qwen3.6-35B-A3B base this model is pruned from. Three hard questions, temperature 0.6, fixed seed, answer characters with wall time in brackets. Every run answered all three correctly, and thinking length is unchanged by the message in every row:

The 4096 row is the failure this exists for: the cap lands mid-thought and the reasoning simply continues in the answer, ten times longer and 2.2x the wall time, for the same three correct answers. Both phrases remove it.

Per request, instead of per server

The server accepts both as request fields, overriding the command line:

{
  "messages": [ ... ],
  "thinking_budget_tokens": 8192,
  "reasoning_budget_message": "\n\nOK, I have enough to answer now.\n"
}

On the raw /completion endpoint the delimiters are not inferred, so they have to be supplied with the budget:

{
  "prompt": "...",
  "reasoning_budget_tokens": 8192,
  "reasoning_budget_start_tag": "<think>",
  "reasoning_budget_end_tag": "</think>",
  "reasoning_budget_message": "\n\nOK, I have enough to answer now.\n"
}

On b10091 the message field must be present on /completion requests even when empty: llama.cpp builds the sequence it forces from message + end_tag inside that field's handler, so omitting it leaves the budget with nothing to force — the sampler logs as though the cap fired while the thinking block stays open.

Rules of thumb

• —Keep -c several times larger than the budget. A budget equal to the context lets the thinking phase fill the window on its own.
• —A quarter of the context is a sensible starting point: 8192 at -c 32768.
• —Qwen recommend keeping a thinking budget above 1024 tokens; below that the cap tends to land before the model has committed to an approach.
• —The budget is per thinking block, not per response — the sampler re-arms when it sees a new opening tag, so a multi-turn agent gets a fresh window each time.

Formats

• —GGUF (this repo family): full imatrix quant sweep (Q8_0 → IQ2, plus ContribDynamic CD- per-layer quants) in [`Qwen3.6-27B-A3B-Coder-MTP-GGUF`](https://huggingface.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF). Includes the native MTP head (speculative decoding) and a `-vision` mmproj* for multimodal use. imatrix.dat archived in-repo.
• —Ollama: `mannix/qwen3.6-27b-a3b-coder` (text) and …-vision tags (with mmproj).

Quantization quality

Every K-quant (Q4KS → Q6K and the `L` variants) is built with imatrix. On this model the imatrix is load-bearing at 4-bit — the opposite of Gemma-4, where imatrix degrades K-quants. Measured on a deterministic greedy MultiPL-E code probe, the entire imatrix K-family and the ContribDynamic CD-\ tiers sit at full-precision parity (within measurement noise of the F16 anchor). The only outlier was a plain*, imatrix-free Q4KM, which fell ~12pp below parity — which is why imatrix is now the default for every tier in this repo.

• —Recommended tier: CD-IQ4_K_M (~16 GB) — full-precision-parity code quality at the smallest at-parity size.
• —Q4_0 / Q4_1 are not shipped — superseded by the imatrix K-quants (legacy round-to-nearest tiers offered no quality at their size).

⚠️ Hardware note — low i-quants (IQ2M, IQ3M) on Blackwell (sm_120) GPUs

The low i-quant tiers `IQ2_M` and `IQ3_M` produce incoherent output ("token salad") on NVIDIA Blackwell GPUs (sm_120, e.g. RTX PRO 6000) — under both stock llama.cpp and opencoti-llamafile, which share the same ggml CUDA IQ2_S/IQ3_S kernel. The weights are fine: the identical GGUF is fully coherent and produces correct code on CPU and on Ampere/Ada GPUs (verified on an RTX 3090). This is a llama.cpp/ggml CUDA-kernel issue on the sm_120 `IQ2_S`/`IQ3_S` path, not a defect in these files.

• —On a Blackwell GPU, instead use: the K-quants (Q2_K_L, Q3_K_S/Q3_K_M/Q3_K_L), or IQ2_XS / IQ4_XS / the CD-* tiers — all coherent on Blackwell in the same size band (Q2_K_L/IQ2_XS ≈ the IQ2_M band; Q3_K_M/Q3_K_L ≈ the IQ3_M band).
• —IQ2_M / IQ3_M are kept in the repo because they are correct on CPU and Ampere/Ada GPUs.

Notes

• —Top-10 is baked as the default; the model was selected and evaluated at top-10.
• —Same tokenizer, chat template, MTP head and vision tower as the base.
• —Research checkpoint. Verbosity on open-ended prompts is a known, base-inherited trait.

Tool-calling benchmark — tool-eval-bench hardmode (88 scenarios, 176 pts)

Benchmarked file: `Qwen3.6-27B-A3B-Coder-Q4_K_M.gguf` (this repo), served on llama.cpp with MTP speculative decoding enabled. The score below belongs to THIS quant — other tiers in this repo were not run.

A3B-Coder scores 123.2 ±2.3 of 176 — last of the ten models in this cohort. That is the measured result and it is reported here unmodified, but two things must be read with it.

First, this model is the one most damaged by a scorer defect in the harness version used. tool-eval-bench v2.6.0 crashes scoring TC-62 and records FAIL/0 while keeping the scenario in the denominator (details below). A3B-Coder hits that crash on all five seeds — more than any other model in the cohort — and later harness commits credit the behaviour the crash discards. A corrected re-run would raise this figure; by how much is not known, and no adjusted number is published here because it has not been measured.

Second, this is a coding-specialised prune, evaluated on general agentic tool use. The profile is consistent with that: Parameter Precision 6/6 (100%), Creative Composition 6/6, Toolset Scale 7.4/8 (92.5%), Structured Output 10.8/12 (90%) — the mechanics of calling tools correctly are intact. What falls away is the long-horizon agentic layer: Autonomous Planning 2.4/6 (40.0%) and Hard Mode 16.8/38 (44.2%), both last in the cohort.

If you want this lineage with agentic tool use intact, use A3B-CoderX instead: same family, +14.2 pts (137.4), with Hard Mode 24.6/38 vs 16.8/38.

15 safety-critical failures across five seeds (TC-31, TC-34, TC-60 on every seed).

[image]

Full cohort

* one seed (s42) is graded on 174 pts, not 176 — see that model's card.

<details> <summary><b>Basis — read before comparing these numbers to anything</b></summary>

• —Scorer: `tool-eval-bench` v2.6.0 (the pip/uv-installed package, verified via tool_eval_bench.__file__, not a git checkout). An earlier note in the runner claimed cf54b4b (v2.6.0-45); that is wrong and has been corrected — no cell ever ran it. All 50 cells ran the same v2.6.0, so the cohort is internally consistent.
• —v2.6.0 carries a known scorer crash on TC-62. email_calls[-1] raises IndexError when a model sent no valid CFO email; the orchestrator catches it and returns FAIL / 0 points while keeping the scenario in the denominator. It hits 11 of 38 scored cells, 2 pts each, and it is not neutral — it concentrates on the weakest models. Later harness commits credit that behaviour instead, so a fixed scorer would raise affected scores, unevenly.
• —5 paired seeds [42–46], 64k context, context-pressure 0.25, max 8 turns, 120 s timeout, thinking enabled, sampler temp 0.6 / top-p 0.95 / top-k 20 (not greedy).
• —Served on llama.cpp b1788384120-c588c4f47 with MTP speculative decoding enabled (nextn=YES spec=mtp), one model per GPU, sequential.
• —Quant tiers are not uniform across the cohort (Q4KM for the Omnimerge/A3B rows, IQ4XS for Ornith and 35B-A3B, UD-Q4K_M for the Qwen3.8 base). Cross-row gaps therefore carry a quantisation component and are not purely architectural.
• —Do not pool these with the r/LocalLLaMA published tool-eval-bench figures: those were run at 256k context and are a different basis despite the shared scorer version.

</details>