fraQtl/Qwen3.6-35B-A3B-Hi-Fi-GGUF

Qwen 3.6 35B-A3B (Q4KM)

by fraQtl · calibration-aware, MoE-aware

Same size as a standard Q4_K_M. Measurably closer to the Q8 teacher across every measured lane (code/math, chat, tool calling, long-form text).

[image]

A drop-in Q4KM for Qwen 3.6 35B-A3B. Identical file size, identical kernel path, identical loader. ~23% lower output-distribution divergence from the Q8 teacher on code/math, ~8% lower on general (chat + tool calling + long-form text) vs a public Q4KM baseline — and ~42% / ~29% lower vs a public IQ4_XS baseline — measured on the same held-out slices, same prompts, same temperature.

No retraining. No custom runtime. Standard llama.cpp Q4KM kernel. The win is in the calibration and per-tensor bit allocation.

💻 This is the local / consumer ship. Runs on Apple Silicon (M-series) and consumer GPUs with stock llama.cpp — no patched runtime, no special flags. A separate MTP runtime variant targets datacenter speculative decoding (its 1.49× decode speedup is A100-80GB only and shows no speedup on consumer hardware), so for local use, this is the build you want.

Quickstart

huggingface-cli download fraQtl/Qwen3.6-35B-A3B-Hi-Fi-GGUF \
  Qwen3.6-35B-A3B-fraQtl-Q4_K_M.gguf --local-dir .

./llama-cli -m Qwen3.6-35B-A3B-fraQtl-Q4_K_M.gguf \
  -p "Write a Python function that returns the nth Fibonacci number." \
  -n 256 --temp 0.2

Or via llama-server for an OpenAI-compatible local API:

llama-server -hf fraQtl/Qwen3.6-35B-A3B-Hi-Fi-GGUF:Q4_K_M

Apple Silicon (M-series)

Verified on Apple M4 / 24 GB unified memory — CPU mode (`-ngl 0`):

llama-server -m Qwen3.6-35B-A3B-fraQtl-Q4_K_M.gguf -ngl 0 -c 2048

Observed on this hardware:

• —Cold load: ~128 s
• —Decode: ~4.9 tok/s
• —Output: coherent (correct Fibonacci function)

The ~20 GB file fits in 24 GB RAM in CPU mode, and because this is an A3B MoE (~3 B active params per token), CPU decode is usable.

Full Metal offload is not verified for this card. On Apple M4 / 24 GB, both -ngl 99 and -ngl 28 fail with a Metal out-of-memory error (kIOGPUCommandBufferCallbackErrorOutOfMemory) because the ~20 GB GGUF exceeds the practical ~16.8 GB Metal allocation ceiling on a 24 GB machine. Use CPU mode (-ngl 0) on 24 GB Apple Silicon.

32 GB+ Apple Silicon: full Metal offload (-ngl 99) is expected to be more viable, but is not yet receipt-backed by us. Treat as experimental until we publish a hardware receipt.

Provenance note: this receipt was produced on the Hi-Fi MTP-runtime GGUF, which shares the same main-model quantization as this file. A strict receipt for the non-MTP Hi-Fi file itself is pending.

Works in any standard llama.cpp consumer:

• —llama.cpp • LM Studio • Ollama • llama-cpp-python • koboldcpp • Jan • Text Generation Web UI

Quality — measured (not claimed)

Same Q4KM file size, same llama.cpp kernel path, measured against two leading public Q4-class quants of the same base model. All five metrics run on the same eval harness against the same baselines.

• —KLD = symmetric top-20 vs the Q8 teacher, restricted to the Q8 support
• —Wikitext-2 PPL on the standard test split
• —GSM8K on 200 test questions (random sample, seed=0), greedy decoding, 0-shot instructed chain-of-thought
• —MATH-500 on the standard 500-question slice, greedy decoding, 0-shot instructed chain-of-thought

Wins or ties every metric. Loses nothing with statistical significance. - KLD code/math: −23% vs Public Q4KM, −42% vs Public IQ4XS - KLD general: **−8%** vs Public Q4KM, **−29%** vs Public IQ4XS - Wikitext-2 PPL: lowest of the three - GSM8K and MATH-500: at these sample sizes the 95% confidence interval is ±4.2 pp, so the −1.5 pp (GSM8K, n=200) and −3.4 pp (MATH-500, n=500) differences vs Public Q4KM are statistical ties, not losses. On MATH-500 we win +5.2 pp vs Public IQ4_XS (just outside the CI).

Reproducibility: three independent eval runs reproduced KLD to five decimal places (drift 0.00000). Build + eval pipeline is deterministic.

What's different

A higher-fidelity Q4KM of Qwen 3.6 35B-A3B (MoE, 256 routed experts), built with two changes vs a stock Q4KM:

1. 1.Per-tensor protection policy. Architecturally critical tensors (router, attention input projections, shared FFN) are quantized at higher precision; routed experts stay at the Q4_K floor. Same total size, smarter bit allocation.
2. 2.Calibration tuned to a measured optimum. Imatrix budget set to the empirically best point on this packet (see Calibration section).

The .gguf is a standard Q4KM; any llama.cpp build that runs Q4KM runs this. No patched runtime, no special flags.

Prompt format

Qwen 3.6 chat template, with optional <think> pre-fill for chain-of-thought:

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>

Calibration

• —Packet: ~414K tokens of curated code + math (worked solutions, multiple languages, mostly Python).
• —Imatrix budget: 256K tokens — the measured optimum on this packet.
• —Decontamination: packet decontaminated against GSM8K test, MATH-500, and Hendrycks MATH-train. Eval slices are disjoint from calibration content.

Why 256K and not "use the whole packet": a 384K budget produced a measurably worse artifact on the same eval slice (+7.20% relative KLD vs the 256K build, byte-identical substrate). The calibration-budget curve is non-monotonic; more is not always better.

Calibration budget is a real, measurable lever — but the lever has a measured peak on this packet (256K tokens), not a monotone curve.

The included imatrix.dat makes the calibration step independently reproducible.

Per-tensor protection policy (summary)

Same total bit budget, smarter spend. The recipe protects the architecturally critical tensors and quantizes the bulk routed experts at the Q4_K floor:

Limitations and scope

• —What this card claims (measured): KLD vs Q8, Wikitext-2 perplexity, GSM8K accuracy, MATH-500 accuracy — all on the same eval harness as the comparison baselines.
• —Still unmeasured for this artifact: MMLU, BBH, HumanEval, tool-calling end-to-end. KLD ≠ benchmark accuracy across all tasks; do not over-generalize from the metrics shown.
• —No speed claim. Decode / prefill throughput is unmeasured. Standard Q4KM kernel performance.
• —No long-context claim. Evaluation ran at 4096-token context. Behavior beyond 4K is unmeasured.
• —Comparator scope: measured against public Q4KM and IQ4_XS baselines on these two slices. Not claimed as universally best across all Q4-class quants or all evaluation slices.
• —Hardware: measurements ran on H100 (Modal) with llama-cpp-python. Reproducibility on other CUDA archs is expected (Q4KM is a stable kernel path) but not separately verified.

Files

License

Apache 2.0 — inherits the base model's license.

Citation

@misc{fraqtl-qwen36-35b-a3b-q4km,
  author = {fraQtl},
  title  = {Qwen 3.6 35B-A3B (Q4_K_M) — fraQtl calibration},
  year   = {2026},
  url    = {https://huggingface.co/fraQtl/Qwen3.6-35B-A3B-Hi-Fi-GGUF}
}

<details> <summary><b>Provenance & reproducibility</b> (for verifiers)</summary>

</details>

By [fraQtl](https://fraqtl.ai). Built on the open-source work of the [Qwen](https://huggingface.co/Qwen) team and the [`llama.cpp`](https://github.com/ggml-org/llama.cpp) community.

The fraQtl ladder

Same discipline at every tier: pinned provenance, measured numbers, losses disclosed.

More from fraQtl

The serving lane — KV-cache compression sidecars for vLLM — just fit nine concurrent ~128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14): fraQtl/qwen3-4b-instruct-2507-kv-sidecars. On-device lane: Gemma-4-E2B Hi-Fi. Same standard per artifact: pinned provenance, measured numbers, results reported in both directions. Org page: huggingface.co/fraQtl.