protoLabsAI/ThinkingCap-Qwen3.6-27B-MTP-GGUF

ThinkingCap-Qwen3.6-27B — MTP-GGUF (Blackwell / speculative-decoding edition)

GGUF quantizations of **BottleCapAI/ThinkingCap-Qwen3.6-27B** — their "brevity finetune" of Qwen3.6-27B that keeps full accuracy with ~46% fewer thinking tokens — repackaged for the two things stock GGUFs don't give you:

• —NVFP4 — native FP4 for Blackwell (RTX 50-series / RTX PRO 6000) tensor cores.
• —MTP baked into every quant — the Multi-Token-Prediction draft head travels inside each file, so you get speculative decoding for free (--spec-type draft-mtp), no second model to wire up.
• —A full low-bit ladder — IQ2→Q8_0 + a lossless bf16 master, so it fits everything from a 24GB card to a workstation.

Same weights as upstream. Strictly more ways to run them, faster.

Files

quant size notes NVFP4-MTP 18.2 GB ← Blackwell FP4 tensor cores + MTP. The one to grab on RTX 50xx / PRO 6000. bf16-MTP 54.7 GB lossless master (exact bf16, not a lossy f16 re-map) + MTP Q80-MTP 29.0 GB near-lossless reference + MTP Q6K-MTP ~22 GB + MTP Q5KM-MTP ~19 GB + MTP Q4KM-MTP ~17 GB + MTP (the common daily-driver size — and here it carries the draft head) IQ4XS-MTP ~14 GB imatrix low-bit + MTP [fast-follow] Q3KM-MTP 13.5 GB imatrix low-bit + MTP — integrity-verified, NVFP4 parity (see below) IQ3M-MTP 12.8 GB imatrix low-bit + MTP — smallest full-quality rung, fits a 16 GB card IQ2_M-MTP ~10 GB imatrix low-bit, fits a 12GB card + MTP [fast-follow] mtp-head/… ~2.4 GB standalone draft head, for pairing with a base GGUF via --model-draft

NVFP4 mini-ladder (for tighter VRAM)

NVFP4-Q4_K_M-MTP (15.7 GB) lowers the non-FP4 base tensors to Q4 — aimed at 24 GB dual-Blackwell (2×12 GB) where the 18.2 GB flagship + 128k KV + MTP won't fit. Honest finding: the savings are marginal because the NVFP4 GEMMs dominate the file and are fixed — the base type only touches ~3 GB of embeddings/norms/GDN. The full curve we measured (Q8→Q6→Q5→Q4 base): 18.2 / 16.8 / 16.3 / 15.7 GB. Quality holds (quant-sensitivity + coherence verified); only the bottom rung meaningfully helps a 24 GB budget, so that's the one shipped. GGUF GPU-speed numbers are pending (validated on CPU here; llama.cpp CUDA testing is a follow-up).

The MTP head is embedded in each bundled quant — it rides the trunk, nothing extra to download. The standalone head (mtp-head/mtp-ThinkingCap-Qwen3.6-27B-head-Q8_0.gguf) is only for pairing with a separate base GGUF; loading it alone crashes. It lives in a subdirectory on purpose — keeping it out of the repo root so HF's "Use this model" / llama.cpp -hf never resolves to it by mistake.

Run it (with speculative decoding)

llama-server --model ThinkingCap-Qwen3.6-27B-Q4_K_M-MTP.gguf \
  --n-gpu-layers 99 --ctx-size 8192 --flash-attn on --jinja \
  --spec-type draft-mtp --spec-draft-n-max 3

--spec-draft-n-max 2 maximizes acceptance; 3 maximizes throughput. On Blackwell, grab the NVFP4 file — MTP verification is nearly free on FP4 tensor cores.

One critical setting: don't decode greedy. This is a thinking model; at temperature 0 it can loop and never close </think>. Use the model's intended sampling (temp 0.6, top_p 0.95, top_k 20). Greedy is the #1 cause of "it rambled and gave no answer" — not the quant.

Blackwell / NVFP4 notes (what we found forging these)

• —The NVFP4 checkpoint serves correctly on sm120 (verified real, coherent output; the brevity behavior survives quantization — a hard problem still answers with a near-empty <think> and a direct solution).
• —vLLM serving of the NVFP4 build needs --linear-backend marlin on this hybrid (GDN/Mamba) arch: the FlashInfer FP4 kernel silently hangs in CUDA-graph capture, while marlin surfaces the real cause — a Mamba-cache-block limit fixed with --max-num-seqs 256 --gpu-memory-utilization 0.85.
• —llama.cpp: the mixed NVFP4 file keeps FP4 GEMMs + Q8_0 for the rest — best size/quality on Blackwell.

Compatibility

MTP-baked GGUFs need a recent runtime that understands nextn blocks:

• —llama.cpp — recent build (any with --spec-type draft-mtp).
• —Ollama — ~0.31+ (older fails with "layer N missing attn_qkv"). Non-MTP runtimes still load the trunk; you just don't get the free speculative decoding.

Benchmarks (measured on the NVFP4 build)

The brevity survives quantization — that's the whole point, and here's the number:

thinking tokens (15-prompt reasoning set, temp 0.6) base Qwen3.6-27B mean 1401 tok ThinkingCap-NVFP4 mean 675 tok → ~40% fewer, per-prompt mean (range 3–82%)

Quant integrity (our eval harness, judge-free where possible):

quantsensitivity (arithmetic/exact-recall/format) 93% near-lossless functioncall 94% quant clean on tool-use coding (hard_v2, adversarial) 50% suite is deliberately hard for a 27B; quant-uniform damage ruled out by the two above MTP draft acceptance (--spec-type draft-mtp) 62–91% (prompt-dependent)

The low-bit rungs hold. We gated Q3_K_M and IQ3_M on the same integrity suites before shipping them (imatrix computed over a standard calibration set) — they come out at parity with the NVFP4 build, so the quant curve isn't leaking quality down to ~12 GB:

suite Q3KM IQ3M NVFP4 (ref) quantsensitivity 14/14 14/14 13/14 function_call 51/54 51/54 51/54 (same 3 misses across all builds = a model/harness ceiling, not quant damage)

Method note: token saving is measured token-count; accuracy-preservation is evidenced by the quant-sensitivity + FC suites above, not graded on the 15 brevity prompts. Coherence-at-depth and 3-hardware speed follow as a card update.

Provenance & credit

• —Base / all the intelligence: BottleCapAI/ThinkingCap-Qwen3.6-27B (a finetune of Qwen/Qwen3.6-27B). Go star their work.
• —This repo: quantization + MTP packaging only, by protoLabsAI.
• —MTP head: the upstream model's own bundled draft head. We measured ~62% draft acceptance (mean accepted length 2.83) on the NVFP4 build via --spec-type draft-mtp — a solid free speed-up straight out of the box, confirming the authors' "MTP works well as-is."

Request a size

Need a quant level that isn't here? Open a discussion — we turn most requests around in ~48h.