Modelpublic

bottlecapai/ThinkingCap-Qwen3.8-27B-MLX-4bit-DWQ

sourceHugging Faceotherupdated 13h agoView on Hugging Face

6likes156downloads

Model Card

ThinkingCap-Qwen3.8-27B-MLX-4bit-DWQ

MLX build for Apple Silicon, in a mixed 4/8-bit affine layout (group 64) whose quantization parameters are fitted to the bf16 model's own outputs by distillation (DWQ) rather than set by rounding alone. 22.5 GB on disk (21.0 GiB), against ≈56 GB for bf16 — it runs on a 32 GB Mac.

Built from bottlecapai/ThinkingCap-Qwen3.8-27B (bf16). The MLP projections of all but the last 8 layers are 4-bit; self-attention, the wide Gated-DeltaNet projections, the last 8 layers' MLPs, lm_head and the embeddings are 8-bit. The vision tower, the MTP head's fc, the narrow Gated-DeltaNet input gates and all norms stay bf16. Vision input and MTP self-speculative decoding both work from this repo — the drafter is in mtp-drafter/ and is also embedded in the main shards.

Decode on Apple Silicon is bound by memory bandwidth, so bits per weight set the speed; the 8-bit share is placed where a flat 4-bit layout costs the most accuracy.

Serving

Use mlx-vlm or oMLX. mlx-lm drops the MTP head and the vision tower when it loads a checkpoint, so it will serve this one as text-only without speculative decoding.

bash

pip install mlx-vlm

python -m mlx_vlm generate \
  --model bottlecapai/ThinkingCap-Qwen3.8-27B-MLX-4bit-DWQ \
  --prompt "Explain the Cauchy–Schwarz inequality." \
  --enable-thinking --max-tokens 4096

MTP self-speculative decoding

The model's own next-token head drafts for it; there is no draft model to download.

bash

python -m mlx_vlm generate \
  --model bottlecapai/ThinkingCap-Qwen3.8-27B-MLX-4bit-DWQ \
  --draft-model bottlecapai/ThinkingCap-Qwen3.8-27B-MLX-4bit-DWQ/mtp-drafter \
  --draft-kind mtp --draft-block-size 3 \
  --prompt "..." --enable-thinking --max-tokens 4096

Acceptance is 2.41 tokens per round, against 2.43 for the bf16 weights on the same prompts.

Faster prefill on M5

M5 has INT8 matrix units, so you can turn on oMLX's INT8-activation prefill with these weights — no separate build needed, the 4-bit projections are already in the affine group-64 format those kernels take. Turn it on in oMLX under Model Settings → Experimental Features; it can't be combined with ANE prefill. oMLX measured 34% faster prefill at 32K on an M5 Max, 615 → 827 tok/s, with generation slightly slower. It's off by default and can change outputs; the numbers below were measured with it off.

Sampling

Thinking mode, as the base model: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0.

Expected performance

Accuracy and completion length against the bf16 weights, on the same questions at the same sampling seeds — GPQA-Diamond 198, MMLU-Pro 1,500, RealWorldQA 765 (images), IFBench 300, AA-LCR 100 (long-context prompts) — thinking at the chat template's default reasoning effort (xhigh), with sampled decoding (temperature 1.0, topp 0.95, topk 20, min_p 0.0). The MMLU-Pro set is a uniform sample of the 12,032-question test split; AA-LCR answers are graded by Gemma-4-26B-A4B-it with thinking off, the same judge the comparison's bf16 row used. median tokens / mean tokens = completion length (reasoning plus answer) over the questions.

Every benchmark lands within about two points of bf16, in both directions, and completion length is unchanged. Compared question by question, wins and losses are evenly split on all five. Evaluated, not proven lossless.

GPQA-Diamond (graduate-level science) — 198 questions × 4 seeds | config | acc | median tokens | mean tokens | |---|---|---|---| | ThinkingCap-Qwen3.8-27B bf16 | 0.880 | 1033 | 7115 | | MLX-4bit-DWQ | 0.862 | 1142 | 6916 |

MMLU-Pro (knowledge) — 1,500 questions × 1 seed | config | acc | median tokens | mean tokens | |---|---|---|---| | ThinkingCap-Qwen3.8-27B bf16 | 0.841 | 166 | 1436 | | MLX-4bit-DWQ | 0.849 | 167 | 1546 |

RealWorldQA (vision) — 765 questions × 2 seeds | config | acc | median tokens | mean tokens | |---|---|---|---| | ThinkingCap-Qwen3.8-27B bf16 | 0.831 | 112 | 488 | | MLX-4bit-DWQ | 0.818 | 114 | 484 |

IFBench (instruction following) — 300 questions × 2 seeds | config | acc | median tokens | mean tokens | |---|---|---|---| | ThinkingCap-Qwen3.8-27B bf16 | 0.797 | 1819 | 4531 | | MLX-4bit-DWQ | 0.785 | 1911 | 4529 |

AA-LCR (long-context reasoning) — 100 questions × 1 seed | config | acc | median tokens | mean tokens | |---|---|---|---| | ThinkingCap-Qwen3.8-27B bf16 | 0.810 | 844 | 1718 | | MLX-4bit-DWQ | 0.830 | 899 | 1843 |

Where to find us

Need even more efficiency? The open release is production-ready. Our enterprise versions go further — fewer thinking tokens still, tuned to your workload, at matched accuracy on your own tasks. Built for AI labs, inference providers and enterprises running models at scale. Deployed on your infrastructure, or in the cloud and region you choose. Talk to our team

License

ThinkingCap: PolyForm Small Business 1.0.0 + BottleCap personal-use grant (see LICENSE).

Upstream Qwen materials: Apache-2.0 (see NOTICE).

Commercial license: contact BottleCap AI.

Citation

If you use this model, please cite:

bibtex

@misc{ThinkingCap-Qwen3.8-27B,
  title     = {bottlecapai/ThinkingCap-Qwen3.8-27B},
  author    = {Osusky, Adam and Lindauer, Jan and Jirkovsky, Adam and Mihal, Filip and Platek, Ondrej and Herel, David and Ihnatchenko, Luka and Bartek, Vojtech and Jirak, Jiri and Kubista, Daniel and Krus, Frantisek and Mikolov, Tomas},
  year      = {2026},
}