utautako/Qwen3.6-35B-A3B-JA-GGUF

Qwen3.6-35B-A3B-JA-GGUF

GGUF quantizations of `Qwen/Qwen3.6-35B-A3B`, using an imatrix calibrated on mixed English and Japanese text. Two builds are provided: Q5_K_M and Q4_K_M. Both preserve the MTP head for draft-mtp speculative decoding and were benchmarked fully GPU-resident on an RTX 5090 with llama-bench.

On the evaluated held-out Japanese corpus, both builds produced lower KL-divergence to the BF16 reference than the compared unsloth / bartowski GGUFs (see Japanese fidelity).

Highlights

• —Lower Japanese KLD than the compared public GGUFs — on the held-out Japanese corpus, Q5_K_M reaches Mean KLD 0.00554 (unsloth UD-Q4_K_XL: 0.01024) and Q4_K_M 0.01189 (bartowski Q4_K_M: 0.01616). English PPL was also lower in both matched comparisons on the evaluated corpus.
• —Mixed English + Japanese imatrix — calibrated on ~803 K tokens; the calibration corpus was selected using held-out Japanese KLD.
• —MTP head preserved (both builds) — the blk.40 next-token-prediction head is kept at Q8_0, so both quants run --spec-type draft-mtp speculative decoding.
• —Two builds — Q5_K_M for closer fidelity to the BF16 reference, Q4_K_M for a ~4.4 GiB smaller footprint.

Which build should I pick?

Both files use the same mixed imatrix and carry the MTP head. Generation speed differs by ~3 % on a large-VRAM GPU, so the practical trade-off is fidelity vs. size / context headroom.

Files

Usage (llama.cpp)

# Full-GPU, with MTP speculative decoding
llama-server \
  -m Qwen3.6-35B-A3B-JA-Q5_K_M.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -ngl 999 \
  -c 131072 \
  --jinja

• —`--spec-draft-n-max 2` is a stable default across both builds. n=3 can be faster on high-acceptance content (code, and the Q4_K_M build) but can regress on Japanese with Q5_K_M (see Benchmarks).
• —`-ngl 999` offloads every layer. The Q5_K_M build (25.3 GiB) leaves less KV headroom on a 32 GB card; drop -c if you hit the VRAM cliff. The Q4_K_M build fits a larger context.
• —This is a thinking model. Reasoning is emitted inside <think>…</think>. To disable thinking for a turn, pass "chat_template_kwargs": {"enable_thinking": false} in the request body (the /no_think soft switch does not affect the chat template).

Benchmarks

RTX 5090 (32 GB), llama.cpp b10098 (cf8fda35c), fully GPU-resident (-ngl 999), f16 KV.

Throughput (llama-bench, forward pass, r=3)

MTP speculative decoding (llama-server, real chat, temp 0, 256 tokens/prompt)

tg (tok/s) with draft-acceptance rate. Baseline is non-speculative on the same server.

On a 32 GB GPU (not memory-bandwidth-bound), MTP n=2 adds roughly +5 – 15 % generation throughput. n=3 is not consistently faster: it improves high-acceptance content (code, English, and the Q4_K_M build) but regresses on Japanese prose with Q5_K_M (216.9 tok/s at 43 % acceptance). n=2 is the safer default. A larger relative benefit is expected on bandwidth-bound CPU decoding, for which these builds were originally tuned, although CPU results are not included here. The MTP head is kept at Q8_0 to avoid reducing draft acceptance.

Japanese fidelity (KL-divergence)

Measured with llama-perplexity --kl-divergence against a BF16 reference, on held-out Japanese text that is not in the imatrix corpus (so the numbers are not calibration overfit). Lower is better. KLD is a property of the quantization and is independent of the runtime hardware.

• —Q5_K_M has lower Japanese KLD and lower Japanese / English PPL than unsloth UD-Q4_K_XL, at a similar size class (only Japanese KLD was measured; the table's other columns are PPL).
• —Q4_K_M has 26 % lower Japanese KLD than the compared bartowski Q4_K_M.
• —English PPL was also lower than the corresponding compared GGUF on the evaluated corpus (Q4_K_M 5.965 < bartowski 6.003). The mixed calibration changes how quantization error is distributed; it does not modify the model weights through training.

Task accuracy: on the easier Japanese multiple-choice benchmark JCommonsenseQA (1119 items), all of these quants score ~94 % with no statistically significant difference (χ² p=0.93) — 35B is at ceiling there. The KLD gap is a distribution-fidelity difference; it shows up on distribution metrics, not on saturated multiple-choice accuracy.

Quantization / conversion notes

Converted from the BF16 source with convert_hf_to_gguf.py --outtype bf16 (MTP head included), then llama-quantize with the mixed English + Japanese imatrix and per-tensor overrides. Each per-token (always-active) tensor is kept at higher precision; the size reduction is spent on the routed experts (8 of 256 are read per token).

`Q5_K_M` — base Q5_K_M:

`Q4_K_M` — base Q4_K_M:

The blk.40 MTP head is set to Q8_0 (it is not covered by the imatrix, and a lower-bit MTP head would reduce draft acceptance); being read only during speculation, it does not affect plain-decode speed.

imatrix corpus: ~803 K tokens / 658 documents of mixed English and Japanese, with a small chat/code fraction. Two calibration corpora were built and quantized; the one with lower held-out Japanese KLD (0.01189 vs. 0.01241) was used for the released builds.

Tensor composition

Model details

License and attribution

Released under Apache-2.0, following the upstream model `Qwen/Qwen3.6-35B-A3B`. This repository redistributes an unofficial imatrix-quantized GGUF conversion; it is not affiliated with or endorsed by the Qwen team. Please also follow the upstream model card's terms and usage guidance.

Acknowledgements

Thanks to the Qwen team for the base model and to the llama.cpp project for GGUF, imatrix, MTP, and the qwen35moe runtime.