ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve

Qwen3.6-27B 4-bit + MTP — up to 2× faster decode with mlx-serve

Qwen 3.6 27B that drafts its own future. This is the standard mlx-community/Qwen3.6-27B-4bit conversion (trunk shards and config byte-identical) plus the model's multi-token-prediction head (mtp/weights.safetensors, ~500 MB) — the one-layer draft head Qwen trains and ships in Qwen/Qwen3.6-27B but that gets dropped by most conversions. The head here is extracted directly from Qwen's original bf16 weights, INT8-quantized (near-lossless for a draft head).

With a runtime that knows how to use it, the model speculates on itself: the MTP head drafts the next tokens, the trunk verifies them in one batched pass with exact rejection sampling — the same output distribution as regular decoding, at temperature 0 or your real sampling settings, just faster. No second model eating your RAM, no greedy shortcut.

Quick start

**mlx-serve** is a native (Zig, no Python) inference server for Apple Silicon with OpenAI- and Anthropic-compatible APIs, plus MLX Core, a macOS menu-bar app with chat, agent mode, and MCP tool calling. It detects the MTP sidecar in this repo and enables speculative decoding automatically — zero configuration:

# Install mlx-serve
brew tap ddalcu/mlx-serve https://github.com/ddalcu/mlx-serve
brew install mlx-serve          # CLI server
brew install --cask mlx-core    # optional: GUI menu-bar app

# Get the model
hf download ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve --local-dir Qwen3.6-27B-4bit-MTP

# Serve it
mlx-serve --model ./Qwen3.6-27B-4bit-MTP --serve --port 11234

curl http://127.0.0.1:11234/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"default","messages":[{"role":"user","content":"hi"}],"stream":true}'

Works with Claude Code out of the box (ANTHROPIC_BASE_URL=http://localhost:11234 claude), plus the OpenAI SDK, Continue, Cursor, Open WebUI — /v1/chat/completions, /v1/messages, /v1/completions (FIM), and /v1/responses, streaming and non-streaming. Speculation engages on every endpoint; opt out per request with "enable_mtp": false or at launch with --no-mtp.

Project: github.com/ddalcu/mlx-serve · Site: ddalcu.github.io/mlx-serve · App + CLI downloads: latest release

Up to 2× faster — measured

Speculative decoding pays off most where the next tokens are predictable: agent loops, file edits, refactors, code generation. Measured with mlx-serve on an M4 Max (greedy, identical output to non-speculative decoding):

Newer chips with more compute headroom (M5 generation) push the speculative win further — batched verification gets cheaper relative to memory-bound autoregressive decoding, which is how this technique reaches the 2× range. mlx-serve also watches the per-request acceptance rate and adapts draft depth on the fly, so there is no tuning pass and no regression on hard-to-predict content — --mtp-depth is there if you want to experiment.

These numbers stack on top of mlx-serve's baseline lead: on identical 4-bit MLX weights it decodes ~35% faster (geomean) than LM Studio on the same hardware — see the project README for the full benchmark suite.

Also works as a plain Qwen 3.6

The trunk shards and config.json are unmodified from the mlx-community 4-bit (group 64) conversion, so this repo loads anywhere that model does — mlx-lm, LM Studio, etc. — which simply ignore the mtp/ folder and run standard autoregressive decoding.

What's in the box

Attribution

See NOTICE for the full derivation chain.