CoolFace
Modelpublic

tokenfires/Qwen3.8-27B-MLX-6bit-MTPLX

sourceHugging Faceapache-2.0updated 28d agoView on Hugging Face
1likes626downloads
Model Card

Qwen3.8-27B — MLX 6-bit + MTPLX MTP adapter

A 6-bit MLX quantization of Qwen/Qwen3.8-27B with a trained multi-token-prediction (MTP) sidecar for MTPLX — native speculative decoding on Apple Silicon with no external draft model.

Built with MTPLX Forge v2.9.2, which converts, calibrates, and then verifies the speedup on real hardware before accepting the adapter. On the build machine (Apple Silicon M5 Max, macOS 26.6.2), Forge measured 2.87× faster decode than autoregressive baseline at MTP depth 3.

Contents

FilePurpose
model-0000*-of-00005.safetensorsLanguage model trunk, 6-bit affine quantization (group size 64)
mtp.safetensorsMTP draft heads, kept BF16 (depth up to 3)
model-vision.safetensorsVision tower (333 tensors), included from the base model
mtplx_runtime.jsonForge provenance, MTP contract, and verified speed evidence

Total footprint: ~22 GB on disk, ~25.4 GB resident during inference.

Quantization recipe

  • —Trunk: 6-bit, affine mode, group size 64 (MLX)
  • —MTP sidecar: BF16 (unquantized — draft heads must track the trunk's distribution closely; quantizing them costs acceptance rate)
  • —Source: Qwen/Qwen3.8-27B @ `1d4bf0f`, BF16 native

Measured performance

Forge verification (greedy continuation, single prompt, M5 Max — recorded in mtplx_runtime.json):

MTP depthAcceptance by positiontok/svs. baseline
0 (autoregressive)—22.61.00×
197%42.31.87×
298% / 94%57.42.53×
399% / 95% / 89%65.12.87×

Verdict: mtp_depth_wins — depth 3 fastest on this hardware, quality checks passed at every depth, no configurations rejected.

[image]

Interactive chat (sampled at temperature 0.6, best of 3 runs, same machine):

MetricValue
Prefill60 tok/s
Decode50 tok/s
Draft acceptance75%
Memory25.4 GB
Cold-start first prompt36.9 tok/s

Sampled decoding accepts fewer drafts than the greedy verification pass, so real-chat throughput sits below the verify numbers. Both are honest measurements of the same artifact.

[image]

Usage

Requires MTPLX ≥ 2.9 on Apple Silicon:

bash
mtplx pull tokenfires/Qwen3.8-27B-MLX-6bit-MTPLX
mtplx run tokenfires/Qwen3.8-27B-MLX-6bit-MTPLX

Recommended sampler settings (shipped in mtplx_runtime.json): temperature 0.6, top-k 20, top-p 0.95. Recommended profile: sustained.

The trunk is standard MLX 6-bit and also loads in mlx-lm without the MTP speedup; the mtp.safetensors sidecar is only used by MTPLX.

Provenance

  • —Forged locally 2026-08-25 with MTPLX Forge v2.9.2 (arch_id: qwen3-next-mtp)
  • —Full recipe, MTP contract, acceptance data, and artifact fingerprint in mtplx_runtime.json
  • —Base model license: Apache-2.0 (included as LICENSE)

This is a community quantization, not affiliated with Qwen or the MTPLX project.