NovaeonStudio/Qwen3.8-35B-A3B-Distill-oQ8-fp16-mtp

[image]

Qwen3.8-35B-A3B-Distill · oQ8-fp16-mtp — a Novaeon.Studio build

A fast, agentic, reasoning local worker for Apple Silicon — with a working MTP head. This is an oMLX oQ8 (near‑uniform 8‑bit, router fp16) quantization of `empero-ai/Qwen3.8-35B-A3B-Distill` — a distillation of the Qwen3.8 frontier teachers (including Qwen3.8 Flash Next) into the sparse Qwen3.6‑35B‑A3B MoE architecture — repacked for the Mac in oMLX‑native format. Built, tuned, and benchmarked on an Apple M5 Max (128 GB) by [Novaeon.Studio](https://novaeon.studio).

Why this build exists: we run local agents on Apple Silicon and wanted frontier‑distilled reasoning at 8‑bit fidelity, in oMLX format, with a documented optimal serving profile. Unlike other 35B‑A3B builds we ship, this one's native MTP head actually accelerates decode under oQ8 — so it's tuned MTP‑on. Everything below is measured on our hardware, not copied.

Best for & why it's here

Best for: the default local agent workhorse — tool-using agents, multi-step reasoning, coding, long-context (256k+), and vision, all on Apple Silicon. Fast enough to sit behind live agents and crons.

Why we published it: this is our own fleet seat — the model every Novaeon agent runs on. We wanted frontier-distilled reasoning at 8-bit fidelity in oMLX format, with a native MTP head that actually accelerates decode and a documented optimal serving profile. 6/6 on our agentic probe, effectively refusal-free, ~80–93 tok/s.

Highlights

• —Frontier‑distilled reasoning. SFT on Qwen3.8 teacher chain‑of‑thought; every answer opens with a <think> block (served as reasoning_content). Set enable_thinking: false for plain answers.
• —Working MTP → fast. The native multi‑token‑prediction head is not a decorative graft — under oQ8 it sustains real speculative acceptance (draft‑6 optimal), keeping decode at frontier‑seat speed.
• —Super‑agentic. 6/6 on our controlled tool‑use probe suite (tool selection, parallel calls, sequential chains, argument fidelity, correct abstention, no hallucinated tools).
• —Vision intact. Multimodal (text + image → text); the vision tower is preserved from the base.

Measured benchmarks (Apple M5 Max 128 GB · oMLX)

All numbers below were measured on our hardware, on these exact weights, with the seat isolated (no other load) using the optimal profile (thinking off).

Decode throughput vs context

[image]

_{Cold load ≈ 1 s (warm registry). Peak short‑context decode reaches ~110 tok/s with MTP draft‑6 in ideal runs.}

Agentic & tool‑use

[image]

Our probe on this build — 6/6 (temp 0): tool selection, parallel tool calls, sequential tool chains, argument fidelity, abstention when no tool is needed, and no hallucinated tools. Standard probes (JSON, code, reasoning, thinking, vision) all pass.

• —IFEval (ours, this build): 84.86 avg — measured locally against these quantized weights (thinking off), scored with the official IFEval verifier (541 prompts, 834 instructions).

Optimal oMLX settings (figured out empirically)

We swept the tuning levers on this build. Because the native MTP head genuinely accelerates decode under oQ8, the optimal profile is MTP‑on with ANE‑prefill off — the inverse of a build whose MTP is inert. (With a working MTP, ANE‑prefill's dispatch overhead is a net loss.)

<details><summary>Drop‑in <code>~/.omlx/model_settings.json</code> entry</summary>

{
  "max_context_window": 262144,
  "mtp_enabled": true,
  "mtp_num_draft_tokens": 6,
  "qwen35_ane_prefill_enabled": false,
  "turboquant_kv_enabled": true,
  "turboquant_kv_bits": 8.0,
  "turboquant_skip_last": true,
  "qwen35_oq_a8_enabled": false,
  "dflash_enabled": false,
  "moe_expert_offload_enabled": false
}

</details>

On the MTP head: empero ships the multi‑token‑prediction head natively in the checkpoint (the model.safetensors.index.json maps the mtp.* tensors directly). oMLX recognizes it (mtp_compatible: true) and it accelerates decode under oQ8 — our A/B shows MTP‑on ≈ +40% short‑context tok/s over MTP‑off, i.e. real draft acceptance. This is unusual for an 8‑bit MoE build and is why we serve with mtp_enabled: true.

Long context: the base uses mRoPE with a high 10M theta, so context extrapolates past the 262,144 native window without YaRN — verified coherent (needle‑in‑haystack recall) at 320k tokens. The hybrid attention (only ~10 of 40 layers carry a growing KV cache) keeps memory tiny even at very long context.

Base model benchmarks

Reported by the base‑model authors (empero-ai/Qwen3.8-35B-A3B-Distill), zero‑shot, vs the Qwen3.6-35B-A3B base it distills onto:

_{Reproduced from the base model card; the distillation lifts reasoning (ARC) while holding MMLU. Full teacher/trace details are on the base card.}

Quickstart

# oMLX (recommended, Apple Silicon)
omlx serve NovaeonStudio/Qwen3.8-35B-A3B-Distill-oQ8-fp16-mtp --port 8000
# then hit the OpenAI-compatible endpoint at http://127.0.0.1:8000/v1

# mlx-vlm
pip install -U mlx-vlm
python -m mlx_vlm.generate --model NovaeonStudio/Qwen3.8-35B-A3B-Distill-oQ8-fp16-mtp \
  --max-tokens 4096 --temperature 0.6 --top-p 0.95 \
  --prompt "A snail climbs 3m by day, slips 2m by night, in a 10m well. How many days to escape?"

Recommended inference parameters

temperature 0.6 · top_p 0.95 · top_k 20 · reasoning model (emits <think>; set enable_thinking: false for plain answers) · parse and strip <think>…</think> for end users.

Attribution & license

• —Base model: `empero-ai/Qwen3.8-35B-A3B-Distill` — Apache‑2.0, by Empero. A distillation of Qwen3.8 frontier teachers into the Qwen3.6‑35B‑A3B architecture.
• —This quantization & serving profile: Novaeon.Studio, 2026. Released under Apache‑2.0, same as the base.

This is an independent community quantization. It is not endorsed by Empero. All credit for the model's capabilities belongs to the base‑model and Qwen authors.

@misc{novaeon2026qwen38distilloq8,
  title  = {Qwen3.8-35B-A3B-Distill oQ8-fp16-mtp: an oMLX build for Apple Silicon},
  author = {Novaeon.Studio},
  year   = {2026},
  note   = {Quantization of empero-ai/Qwen3.8-35B-A3B-Distill},
  url    = {https://huggingface.co/NovaeonStudio/Qwen3.8-35B-A3B-Distill-oQ8-fp16-mtp}
}

_{novæon — digital business architecture + AI · <a href="https://novaeon.studio">novaeon.studio</a>}