barozp/Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit

Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit

MTPLX 3-bit conversion of [barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP](https://huggingface.co/barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP) — the REAP-pruned (205/256 experts) Qwen3.6-35B-A3B MoE merged with an Opus-reasoning-distilled LoRA, plus its native MTP layer — for speculative decoding on Apple Silicon via mtplx.

Sibling formats: the bf16 safetensors source above and the GGUF collection for llama.cpp/Ollama/LM Studio.

Converted straight from the bf16 safetensors weights (not from any GGUF quant), so there is no dequantize-requantize drift in the chain.

Why this release exists

MTPLX grafts multi-token prediction (MTP) onto a standard checkpoint and exposes it through MLX on Apple Silicon: the draft head proposes 1–3 tokens per step, the base model verifies them, and accepted tokens are emitted in a single forward pass. For this MoE (29B total, ~3B active per token) the 3-bit trunk cuts the on-disk footprint to ~13.8 GB and keeps the MTP sidecar in bf16, unlocking speculative speedups without a separate draft model.

This repo is the 3-bit affine point of that family. If you can afford the VRAM/unified-memory budget, the sibling 4-bit MTPLX build (when published) will be closer to lossless; for tight budgets this 3-bit file is the entry point.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness (HF backend, bfloat16, chat template disabled) on the underlying safetensors checkpoint. Same base chain as the GGUF card: unpruned Qwen3.6-35B-A3B → REAP 205/256 (no LoRA, RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2) → this checkpoint.

Key finding: the reasoning LoRA pushes ARC-Challenge above the unpruned 256-expert base (0.616 vs 0.532) — a genuine skill transfer from the Opus chain-of-thought data. See the GGUF card and the safetensors card for full methodology.

Available MTPLX conversions

No separate mmproj/vision file — this MoE is text-only.

Which one to pick:

• —Best quality → bf16 source
• —Apple Silicon with ≤24 GB unified memory → this 3-bit MTPLX
• —llama.cpp / Ollama / LM Studio workflows → use the GGUF repo instead (IQ/MTP-GGUF quants)

Conversion notes

• —Source: barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP (commit 9fa24b5, 16 shards + model-mtp.safetensors), forge-local via mtplx==2.9.1
• —Recipe: body_bits=3, body_group_size=64, body_mode=affine, mtp_policy=keep_bf16, contract_calibration_depth=1 / limit=8 / windows=4, concat_orders=embedding_hidden, MoE-aware (qwen3-next-mtp, mtp_sidecar=bf16-qwen-moe)
• —MTP contract: base_hidden_variant=pre_norm, hidden_variant=post_norm, concat_order=embedding_hidden, mtp_position_mode=local, mtp_quant_group_size=64, mtp_quant_mode=affine
• —Output: 3 safetensors shards + mtp.safetensors (1.3 GB, bf16 sidecar) + model.safetensors.index.json; architecture Qwen3_5MoeForCausalLM (40 layers, hidden 2048, 205/256 experts, hybrid Gated-DeltaNet / full-attention)
• —Forged at: 2026-08-23T18:26:09+03:00 on Apple M5 Pro (18 CPU / 20 GPU, 24 GB unified memory) — tuned for 24 GB Macs · macOS 27.0 arm64, mtplx_runtime.json ships in-repo as provenance

Vision

Not applicable — text-only MoE. No model-vision.safetensors, no mmproj.

MTPLX usage

# install
pip install -U mtplx

# single-turn generation (auto-selects the verified depth if you omit --depth)
mtplx run --model barozp/Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit \
  --prompt "Explain the Monty Hall problem briefly." \
  --depth 1 --max-tokens 512

# OpenAI-compatible local server
mtplx serve --model barozp/Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit \
  --depth 1 --port 8080
# then: curl http://localhost:8080/v1/chat/completions ...

# text-only chat (temperature 0.6, top_p 0.95, top_k 20 are the verified defaults)
mtplx run --model barozp/Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit \
  --prompt "Write a short story about a robot learning to paint." \
  --depth 1

Depth is the number of draft tokens proposed per step. Higher depth is not always faster — use the verified optimum below.

Verification results

Verified locally with mtplx forge verify (MLX backend, greedy + quality gate, 512-token budget). Depth 0 is plain autoregressive (no MTP); depth N proposes N draft tokens.

• —Recommended depth: 1 (mtp_depth_wins at D1; D2/D3 lose throughput on this MoE+3-bit combo despite passing quality)
• —Verdict: mtp_depth_wins; quality_rejected=[], failure_reasons=[]
• —Hardware: Apple M5 Pro (18 CPU / 20 GPU, 24 GB) · macOS 27.0 arm64 (Apple Silicon, MLX), mtplx 2.9.1, artifact sha256:6af5c44e...
• —All depths passed the quality gate; the gate is strict (rejects on drift), so any quality_passed=true row is safe to serve. Throughput is the tiebreaker.

Note: This MoE's head is unusually sensitive to quantization at 3-bit (single-position acceptance 58% vs 83–98% on the Qwen3.8 dense family). Expect the best speedup at D1 only; the dense VLM siblings in this collection sustain D2/D3 at >2x.

Training details (source safetensors model)

• —Base: Qwen/Qwen3.6-35B-A3B — MoE, REAP-pruned to 205/256 experts
• —Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
• —Data: `barozp/opus-reasoning-distill-train` (Opus reasoning traces)
• —Vision + MTP: vision N/A; MTP layer carried over from base and kept in bf16 for MTPLX

Source chain

Qwen/Qwen3.6-35B-A3B (base) → RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2 (REAP 20% pruning) → barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill (LoRA finetune) → barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP (MTP head) → this repo (MTPLX 3-bit conversion)