Litwein/Qwen3.6-35B-A3B-Claude-4.7-Opus-oQ4-MTP-MLX

<!-- V2BANNER --> [](https://huggingface.co/Litwein/Qwen3.6-35B-A3B-Claude-4.7-Opus-oQ4e-DWQ-MC-MTP-MLX)

# ⚠️ Superseded — use v2 instead ### 👉 `Qwen3.6-35B-A3B-Claude-4.7-Opus-oQ4e-DWQ-MC-MTP-MLX` v2 is a two-stage imatrix → DWQ build (reasoning, then math/code) of the same base. It is materially better on code — HumanEval 90.2 % (this v1 line: 84.8 %) · LiveCodeBench 39.9 % (v1: 32 %) · MBPP 75.4 % — at the same size class, same MTP packaging. This v1 repo is kept for reproducibility only. New users should download v2.

Qwen3.6-35B-A3B-Claude-4.7-Opus-oQ4-MTP-MLX

A 4-bit MLX quantization (oQ) of `lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled` — text-only, with MTP speculative decoding.

This is a text build of the reasoning-distilled Qwen3.6-35B-A3B, packaged for fast local inference on Apple Silicon with an embedded Multi-Token-Prediction (MTP) head for speculative decoding.

⚠️ These are quantized weights. All capability comes from the base model `lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled` — please star/cite it first. This repo's contribution is the quantization + MTP packaging and its fidelity/speed validation.

Model lineage

Qwen/Qwen3.6-35B-A3B                      (Apache-2.0 base, 35B MoE · 256 experts · ~3B active · 262k ctx)
  └─ lordx64/…-Claude-4.7-Opus-Reasoning-Distilled   (SFT distillation of Claude Opus 4.7 reasoning)
       └─ THIS REPO: oQ 4-bit MLX + MTP

• —Architecture: Qwen3.6-35B-A3B — Mixture-of-Experts, 256 experts (8 routed + 1 shared), ~3B active parameters/token, 40 layers (linear + full attention), up to 262,144-token context (the base card recommends ≤64k for best quality).
• —Reasoning distillation (by [lordx64](https://huggingface.co/lordx64)): SFT on ~7,800 chain-of-thought traces from Claude Opus 4.7 (Anthropic), teaching explicit <think>…</think> reasoning. Attention-only LoRA (r=16) via Unsloth + TRL.
• —This repo: 4-bit MLX quantization (oQ) with MTP preserved.

Quantization: oQ4 (oMLX sensitivity-mixed), 4-bit

This is an oQ4 quant produced by oMLX's built-in oq quantizer — sensitivity-based dynamic mixed precision. Each layer is measured for its impact on the output and assigned 4, 5, 6 or 8 bits accordingly; the most sensitive projections keep more bits, the rest go to 4-bit.

Mixed precision spends extra bits on sensitive layers; the sibling DWQ4 repos reach the same task quality at a smaller, uniform 4-bit footprint (see Evaluation).

Evaluation

Two kinds of numbers below: quant-fidelity metrics measured on this quant, and capability benchmarks inherited from the bf16 base (a 4-bit quant preserves — does not create — capability; the fidelity metrics show how little was lost).

Quant fidelity (measured on this model, served via oMLX)

¹ The single miss is a grader formatting artifact on a prime-factorization prompt (7×13 vs 7 and 13), not a reasoning error.

Note on long-form math. This is a reasoning model that routinely emits 8,000+ tokens of <think> on olympiad problems. Fixed-budget pass/fail on such problems is token-budget-limited, not quantization-limited — every 4-bit and 8-bit level behaves the same. Give it a large max_tokens (32k) for hard problems.

Inherited base-model benchmarks (bf16, from the base card)

Reported by the base model lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled via lm-evaluation-harness; listed here as a capability reference for the family, not re-measured on this quant.

Sibling repos (same base, pick your trade-off)

TL;DR: the DWQ builds are smaller, a touch faster, and match the oQ builds on task quality — distillation lets uniform 4-bit keep up with mixed precision. oQ builds spend extra bits on sensitive layers (a more conservative, classic dynamic quant).

How to run

These are MLX weights (Apple Silicon). The tested serving path is [oMLX](https://omlx.app), which supports this model's native MTP speculative decoding out of the box.

Text-only (no vision tower) — leanest footprint for pure text + MTP.

1. Place the folder in your oMLX models directory:

# after downloading this repo:
mv Qwen3.6-35B-A3B-Claude-4.7-Opus-oQ4-MTP-MLX ~/.omlx/models/

2. Enable MTP for this model (one-time), then call the OpenAI-compatible API:

curl -X PUT http://127.0.0.1:8003/admin/api/models/qwen3.6-35b-a3b-claude-4.7-opus-oq4-mtp-mlx/settings \
  -H "Authorization: Bearer sk-local" -H "Content-Type: application/json" \
  -d '{"mtp_enabled": true}'

curl -X POST http://127.0.0.1:8003/v1/chat/completions \
  -H "Authorization: Bearer sk-local" -H "Content-Type: application/json" \
  -d '{"model": "qwen3.6-35b-a3b-claude-4.7-opus-oq4-mtp-mlx",
       "messages": [{"role": "user", "content": "Prove there are infinitely many primes."}],
       "max_tokens": 8000, "temperature": 0.6}'

The oq mixed-precision (4–8-bit) layout is built for oMLX; stock mlx-lm does not reliably reconstruct the 5-bit layers, so run this build through oMLX. MTP speculative decoding needs an MTP-aware runtime — oMLX (tested), or mlx-lm with PR #990.

Recommended sampling

temperature 0.6, top_p 0.95, top_k 20. For hard reasoning set max_tokens ≥ 32000 — the model thinks in explicit <think>…</think> blocks and can be long.

Intended use & limitations

• —Built for hard reasoning: competition math, STEM, code walk-throughs, multi-step logic. It thinks in long <think> blocks — budget tokens accordingly.
• —Reasoning ≠ knowledge. Quantization (and the base distillation) transfer how to reason, not new facts.
• —Quantization loss: 4-bit is lossy vs bf16. The fidelity metrics above quantify it (small); for maximum quality use the base bf16 or an 8-bit quant.
• —Distillation provenance: the base's training traces were generated with Anthropic's Claude Opus 4.7. Downstream users should confirm compliance with Anthropic's usage policy.

Datasets

• —Base distillation (training): `lordx64/reasoning-distill-opus-4-7-max-sft` — ~7,800 Claude Opus 4.7 reasoning conversations.
• —Quant calibration (oQ): oMLX sensitivity proxy over the base weights (no external dataset).

Acknowledgements

• —[lordx64](https://huggingface.co/lordx64) — the reasoning-distilled base model this repo quantizes. All capability is theirs.
• —Qwen team — Qwen3.6-35B-A3B, released under Apache-2.0.
• —Anthropic — Claude Opus 4.7, the reasoning teacher for the base distillation.
• —Apple MLX — mlx / mlx-vlm — the MLX runtime that oMLX's oq quantizer is built on.
• —oMLX — the oq quantizer, MTP serving runtime, and OpenAI-compatible API.
• —Unsloth & EleutherAI lm-evaluation-harness — used by the base model (training / eval).

License

Apache-2.0, inherited from the base model and Qwen3.6-35B-A3B.

Citation

@misc{qwen36_a3b_2026, title={Qwen3.6-35B-A3B}, author={Qwen Team}, year={2026},
  howpublished={\url{https://huggingface.co/Qwen/Qwen3.6-35B-A3B}} }
@misc{lordx64_qwen36_distill_2026,
  title={Qwen3.6-35B-A3B distilled from Claude Opus 4.7 reasoning}, author={lordx64}, year={2026},
  howpublished={\url{https://huggingface.co/lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled}} }