vmarcelo/ThinkingCap-Qwen3.8-27B-MIX_GGUF

### ⚠️ Superseded — use Swift 1.5 instead This quant has been superseded by [`ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF`](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF), the official GGUF release of Swift 1.5. I recommend using that release instead — this page remains available as an archive. ---

ThinkingCap-Qwen3.8-27B: IQ4_XS-MIX

A custom mixed-tensor IQ4 quantization of `bottlecapai/ThinkingCap-Qwen3.8-27B`, the ThinkingCap fine-tune of Qwen3.8-27B that keeps the base model's answer quality while reasoning in far fewer tokens.

It comes in at 13.40 GB and 3.924 BPW, which is 2.08 GB smaller than the published `IQ4_XS` (15.48 GB), and it keeps the MTP head embedded so speculative decoding needs no second file.

Built with llama.cpp `58367713a` from BottleCap's own f16 GGUF, using a purpose-built importance matrix and per-tensor overrides. general.file_type is 30 (MOSTLY_IQ4_XS), matching the name.

Files

Vision needs an mmproj file, so use BottleCap's `mmproj-ThinkingCap-Qwen3.8-27B-f16.gguf`.

Recipe

The file averages fewer bits than a uniform `IQ4_XS` (3.924 vs 4.25 BPW), because it spends more on the tensors that matter and less on the feed-forward. The feed-forward is 62.6% of all parameters, which makes it the only place with enough mass to fund a smaller file.

FFN allocation by depth:

The following tensors are protected, because they are cheap and because the sensitivity research flags them as fragile in hybrid GatedDeltaNet models: ssm_alpha/ssm_beta at Q8_0; linear-attention attn_gate at Q5_K with ssm_out/attn_qkv at IQ4_XS; full-attention attn_k/attn_v at Q8_0, attn_output at Q5_K and attn_q at IQ4_XS; output.weight at Q5_K; token_embd at Q3_K; and the MTP block at Q6_K.

Why there is no front-end lift: early blocks are among the cheapest in this architecture, so they take the deepest cuts rather than extra bits. Only the last twelve blocks are lifted.

Usage

llama-server -m ThinkingCap-Qwen3.8-27B-IQ4_XS-MIX-3.92bpw.gguf \
  --mmproj mmproj-ThinkingCap-Qwen3.8-27B-f16.gguf \
  -ngl 999 -c 65536 -b 2048 -ub 512 -t 12 -np 1 \
  --cont-batching --jinja --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --host 0.0.0.0 --port 8080

The MTP head is embedded, so --spec-type draft-mtp alone enables speculative decoding and there is no --model-draft file to pass. Dropping that flag frees about 1.2 GiB of VRAM and roughly doubles the usable context, at the cost of around 27% of generation speed (see below).

Use BottleCap's recommended sampling settings from the main model card. Note that greedy decoding can loop, so keep temperature at the recommended value.

Context guidance (16 GB VRAM)

Only 16 of 65 blocks are full attention, so the KV cache is small: 64 KiB/token at F16, 34 KiB at `Q8_0`, 21 KiB at `Q5_0`/`Q4_1`. Every figure below is measured on an RX 9070 XT (RADV/Vulkan) with this exact file and a ~22.5K-token prompt.

MTP is the dominant variable here, not the KV type. The speculative-decoding draft carries its own KV cache and verification buffers, costing about 1.2 GiB, which is roughly 27K worth of Q8_0 context. Turning MTP off buys around 50% more context for about 27% of generation speed, and it slightly raises prompt throughput.

Reading the spill column. GTT is system memory the driver fell back to when VRAM ran out. A ~1 GB spill costs 6–9% of prompt throughput and 22% of generation, so it is slower but still a legitimate operating point if you want the extra context. Past ~1.6 GB it collapses, losing 35–46%. Spill is silent, in that it still exits cleanly, so check GTT rather than assuming a configuration fits.

Recommended picks:

• —Fastest, fully resident: Q5_0/Q4_1 + MTP at 80–96K, which gives the best generation (47.9 t/s).
• —Long context with lossless KV: Q8_0/Q8_0 + MTP off at 80–96K, which gives the best prompt throughput (1017.6 t/s) with uncompressed attention.
• —Accepting a mild spill: Q5_0/Q4_1 + MTP at 128K, a third more context than the spill-free ceiling in exchange for 22% of generation speed.
• —Maximum context: Q5_0/Q4_1 + MTP off at 160K, still spill-free.

Prefer Q8_0 KV where the context allows. It is consistently about 10% faster on prompt processing than Q5_0/Q4_1 in every configuration measured here, with generation within noise, so compressed KV is a capacity lever rather than a speed one.

Why Q5_0/Q4_1 instead of Q4_0/Q4_0?

If the cache has to be compressed, Q5_0/Q4_1 is the better trade. Measured against a bf16 cache on a 27B model of this exact shape:

Q4_0/Q4_0 is 16.7% smaller (18 vs 21 KiB/token), but its worst 0.1% of positions diverge 32% more, which costs about 2.8 points of tail precision. Perplexity cannot see this difference. It shows up as broken JSON keys and failed tool calls, which is the one failure an agentic session cannot absorb. Three rules fall out of the same data:

• —Crossing V below q5 costs a full point of tail precision, while steps within q5 cost about half a point. That is why Q5_0/Q4_1 lands within 0.05 points of Q5_0/Q5_0 at less size.
• —Spend bits on K first. At the same footprint, Q5_0/Q4_0 beats symmetric Q4_1.
• —Symmetric Q8_0/Q8_0 is a validation mode rather than a practical default, because dropping V to Q5_1 costs only about 0.4 points of tail precision while shrinking the cache by 7.8% (34 down to 28 KiB/token here, saving 576 MiB at 96K).

Source: KV Cache Quantization Benchmarks for Long Context (Anbeeld; 99.9% KLD against a bf16 cache, Qwen 3.6 27B on an RTX 3090).

How it was built

Quantized from BottleCap's f16 GGUF directly, with no re-conversion, so the MTP head, tokenizer and chat template are exactly as published. The importance matrix was computed on a 13.4 GB IQ4 copy of the model rather than the 54.7 GB f16, which keeps it fully resident in VRAM (583 chunks at ctx 512, ~3.2 s/pass). Running it on the f16 would spill out of page cache and re-read tens of GB per chunk.

Credits

• —Model: `bottlecapai/ThinkingCap-Qwen3.8-27B` by BottleCap AI, the weights this was built from.
• —Base model: `Qwen/Qwen3.8-27B` by the Qwen team.
• —Calibration corpus: `bartowski/Qwen3.8-27B-GGUF` (calibration-v6, chat-template rendered).
• —Toolchain: llama.cpp at 58367713a.
• —Quantized by vmarcelo.

License

The base model is Apache-2.0. The ThinkingCap contribution is released under the PolyForm Small Business License 1.0.0, and these GGUF conversions inherit those terms. See the source repository for the full licence text.