p4ik/Qwen3.8-27B-MLX-8bit

Qwen3.8-27B-MLX-8bit

A uniform 8-bit MLX quantization (group size 64) of Qwen/Qwen3.8-27B for Apple Silicon — as a full package: bf16 vision tower, MTP speculative-decoding head, processor configs, hardened chat template. Every layer at 8 bits, no mixed precision.

Note: maximum fidelity, zero quantization risk — the anchor every build in the comparison below is measured against.

Highlights

• —🖼️ Image input on all three stacks: `optiq serve`, `mlx-vlm`, `vllm-mlx`. Ships the base model's processor configs, which quantization pipelines commonly drop — without them, images are silently ignored.
• —⚡ MTP speculative decoding, engine-agnostic. Head at mtp/weights.safetensors — the default path optiq serve and vllm-mlx both search. This repo's head is prequantized at 8 bit, matching the weights.
• —🔧 Hardened chat template, adopted from [unsloth](https://huggingface.co/unsloth/Qwen3.8-27B). Accepts developer, merges system messages, guards tool-call arguments; renders byte-identically to the original (verified).

How it compares

¹ Bits per weight, file-based: shard bytes × 8 / parameters, same formula for every column.

² Our uniform reference builds — full packages (bf16 vision, MTP head, hardened template), deliberately without measured allocation or KV config.

³ ✓ = works on all three stacks (optiq serve, mlx-vlm, vllm-mlx). Vision needs the base model's processor configs, which quantization pipelines commonly drop; the MTP head needs the engine-neutral path mtp/weights.safetensors. "OptiQ only": runs solely under optiq serve — that repo lacks the processor configs, and its MTP head sits on optiq's internal path that other engines do not search.

⁴ Paired next-token NLL over a 196k-token corpus (agentic transcripts with tool calls and thinking, German prose, WikiText) against the uniform 8-bit anchor; corpus and method are ours.

⁵ All German-prose deltas lie beyond 2 SE; every other ΔNLL row is within noise.

⁶ Tokens the 8-bit anchor is near-certain about (NLL < 0.05) that jump above NLL 0.5 — the failure mode that breaks tool-call syntax. Lower is better.

⁷ Free-running greedy decoding, 32 tokens from 168 held-out prompt windows of the NLL corpus: mean position of the first token that departs from the anchor's trajectory (higher is better). Share of trajectories still identical after 8 tokens: 46 / 48 / 46 / 35 %.

Use

Image input, uniform 8-bit KV cache, MTP speculation — one line:

pip install mlx-optiq
optiq serve --model p4ik/Qwen3.8-27B-MLX-8bit --mtp --kv-bits 8

--kv-bits 8 (group size 64) keeps the cache on the same no-compromise tier as the weights. Text-only use works with plain mlx-lm; image input also runs under mlx-vlm and vllm-mlx.

Files

No measured KV config and no sensitivity table — those are products of the measured OptiQ builds.

Sampling

From the base model card, unchanged: temperature 1.0 / top_p 0.95 (thinking), 0.7 / 0.8 (instruct).