barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit

Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit

MTPLX 3-bit conversion of [barozp/Qwen3.8-27B-Opus-Distill-v2](https://huggingface.co/barozp/Qwen3.8-27B-Opus-Distill-v2) — the bugfix release of Qwen3.8-27B-Opus-Distill ("v1") — for speculative decoding on Apple Silicon via mtplx.

Sibling formats: the bf16 safetensors source above, the GGUF collection for llama.cpp/Ollama/LM Studio, and the 4-bit MTPLX repo.

Converted straight from the bf16 safetensors weights (not from any GGUF quant), so there is no dequantize-requantize drift in the chain.

Why this release exists

A community member (zxbc2023) reported a clean, deterministic bug on v1: certain stacked output-format constraints (e.g. "no prose" + "no markdown") sent the model into a non-converging self-verification loop, burning the entire token budget with zero visible output. Root cause traced to part of v1's training data being sourced from a dataset with reconstructed (not verbatim) reasoning traces. This release retrains on a rebuilt dataset — every row traced to a verified genuine Opus source, every row scanned (30-way parallel LLM-judge pass over the full set) and dropped if it showed the same pathological pattern.

MTPLX 3-bit keeps that fix intact and adds self-speculative decoding: the bf16 MTP head proposes draft tokens, the quantized trunk verifies them, all in a single MLX process. At ~13.6 GB on disk it fits comfortably on 24 GB unified-memory Macs while preserving vision.

Bug fix validation

Re-ran the reporter's exact repro (temp=0, seed=1234, fib(n) with stacked constraints):

All three converge quickly, </think> closes normally — no loop.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (--limit 500). Same protocol as v1, so the delta column is the meaningful signal.

General knowledge (MMLU) and language modeling (wikitext) are essentially unaffected — in the same noise band as v1. Reasoning (ARC, GPQA) shows a large, real gain over base, not comparable to Qwen's own published GPQA number (thinking-mode-on, different harness) — this is a same-protocol base-vs-distill delta.

Available MTPLX conversions

K/MT comparison for context: MLX-4bit (14.97 GB, no MTP) and MLX-8bit (27.49 GB) exist for non-speculative MLX workflows; GGUF covers llama.cpp.

Which one to pick:

• —Best speculative speedup + quality headroom → 4-bit MTPLX (2.39x at D3 in our tests)
• —Tightest fit / 24 GB Mac → this 3-bit MTPLX (1.63x at D2, still strong)
• —No speculation needed → MLX-4bit; near-lossless → MLX-8bit / GGUF Q6K / Q80

Conversion notes

• —Source: barozp/Qwen3.8-27B-Opus-Distill-v2 (bf16, barozp--Qwen3.8-27B-Opus-Distill-v2 local cache), forge-local via mtplx==2.9.1
• —Recipe: body_bits=3, body_group_size=64, body_mode=affine, mtp_policy=keep_bf16, quantized trunk + bf16 MTP sidecar
• —MTP contract: base_hidden_variant=post_norm, hidden_variant=post_norm, concat_order=embedding_hidden, mtp_position_mode=local, mtp_quant_group_size=64, mtp_quant_mode=affine
• —Output: 3 safetensors shards + model-vision.safetensors (879 MB, 333 tensors, bf16) + mtp.safetensors (810 MB, bf16 sidecar) + model.safetensors.index.json; architecture qwen3_next (dense 27B, 64 layers, hybrid Gated-DeltaNet / full-attention)
• —Forged at: 2026-08-24T10:32:21+03:00 on Apple M5 Pro (18 CPU / 20 GPU, 24 GB unified memory) — tuned for 24 GB Macs · macOS 27.0 arm64, mtplx_runtime.json ships in-repo as provenance

Vision

Unlike the GGUF builds (where the tower ships separately as mmproj), the vision encoder is inside these weights — image input works out of the box, no extra file to load. The tower is stored separately as model-vision.safetensors but is auto-loaded by mtplx; you just pass --image:

mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit \
  --prompt "Describe this image." --image photo.jpg --depth 2

Text-only usage needs nothing extra. High-resolution photos expand into many visual tokens (~12k observed); cap resolution when needed (MTPLX inherits the max-pixels handling from the base processor).

MTPLX usage

# install
pip install -U mtplx

# text-only chat (verified optimum is --depth 2 for this 3-bit)
mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit \
  --prompt "Explain the Monty Hall problem briefly." \
  --depth 2 --max-tokens 512

# multimodal
mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit \
  --prompt "Describe this image." --image photo.jpg --depth 2

# OpenAI-compatible local server
mtplx serve --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit \
  --depth 2 --port 8080
# then: curl http://localhost:8080/v1/chat/completions ...

Defaults verified for this checkpoint: temperature=0.6, top_p=0.95, top_k=20, recommended_profile=sustained. Depth is the number of draft tokens proposed per step.

Verification results

Verified locally with mtplx forge verify (MLX backend, 512-token budget, quality gate enabled). Depth 0 is plain AR.

• —Recommended depth: 2 (mtp_depth_wins at D2; D3 passes quality but loses throughput)
• —Verdict: mtp_depth_wins; quality_rejected=[], failure_reasons=[]
• —Hardware: Apple M5 Pro (18 CPU / 20 GPU, 24 GB) · macOS 27.0 arm64 (Apple Silicon, MLX), mtplx 2.9.1, artifact sha256:2d5b4a0c...

All depths passed the quality gate; throughput is the tiebreaker. For a faster but slightly larger alternative, the 4-bit sibling sustains D3 at 2.39x on the same hardware.

Training details (source safetensors model)

• —Base: Qwen/Qwen3.8-27B — dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
• —Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
• —Data: `barozp/opus-reasoning-distill-v2` (11,716 examples, 100% verified-genuine Opus traces)
• —Run: 1 epoch (696 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~4h15m on A100 80GB
• —Final validation loss: 0.4334
• —Vision + MTP: carried over byte-for-byte from the base checkpoint — never trained

Source chain

Qwen/Qwen3.8-27B (base) → barozp/Qwen3.8-27B-Opus-Distill-v2 (LoRA finetune, safetensors) → this repo (MTPLX 3-bit conversion)

Thanks to zxbc2023 for the bug report that made this release possible.