LookUpMark/Ornith-1.5-35B-A3B-oQ6e-mtp

Ornith-1.5-35B-A3B · MLX 6-bit (oQ6e) + MTP

MLX quantization of `ornith-ai/Ornith-1.5-35B-A3B` — a Mixture-of-Experts model with ~3B active parameters — in 6-bit (oQ6e) with Multi-Token Prediction (MTP, speculative depth 3), built to run on Apple Silicon via a local oMLX server.

Quantization details

• —Base model: `ornith-ai/Ornith-1.5-35B-A3B` (qwen3_5_moe architecture, MoE with ~3B active parameters per token)
• —Quantizer: oQ — oMLX v0.6.4, mixed-precision
• —Vision encoder preserved — runs as a full VLM
• —Report: oq_imatrix_report.json in this repo · 244 per-tensor sensitivity overrides

Benchmarks on Apple Silicon (M1 Max 64 GB)

Measured on 2026-09-05 with the oMLX server on identical seed-fixed prompts (same-day runs): TurboQuant KV cache at 4 bits, DFlash2 speculative decoding (community draft `jzinno/Ornith-1.5-35B-A3B-DFlash2`), greedy decoding. Decode throughput, medians of two runs.

Decode speed (tokens/s) vs context length:

• —Speculative decoding acceptance: 87–92% on generation tasks (~+70% tokens/s vs plain autoregressive decoding)
• —Quality probes: needle-in-a-haystack recall 4/4 with zero false positives on all three variants (same fixed codes, ~23k context); factual, math and Italian-language checks all correct
• —Agentic checks (headless runs of the pi agent, verified programmatically): 6/6 completed — bug-fix (3 planted bugs, unittest suite passes: 46/42/58 s for oQ4e/oQ6e/oQ8e) and multi-file analysis (5 CSVs into a verified summary.json: 33/42/52 s)

Takeaway: from 4-bit to 8-bit the throughput stays in the same band and retrieval recall is identical. On 64 GB machines the 6-bit variant is the sweet spot.

Sibling variants

• —Ornith-1.5-35B-A3B-oQ4e-mtp — smallest footprint
• —Ornith-1.5-35B-A3B-oQ6e-mtp — recommended on 64 GB machines
• —Ornith-1.5-35B-A3B-oQ8e-mtp — highest fidelity

Credits

All credit for the base model goes to the Ornith team ([ornith-ai](https://huggingface.co/ornith-ai)):

• —Base model: ornith-ai/Ornith-1.5-35B-A3B (MIT — see LICENSE), which extends Ornith-1.0 (built on Qwen3.5 and Gemma4) via an end-to-end self-improvement loop
• —Blog: deep-reinforce.com/ornith.html · ornith.ai/ornith_1_5.html
• —Quantization tooling: oMLX
• —Speculative decoding draft: `jzinno/Ornith-1.5-35B-A3B-DFlash2`

This repo is only a quantization for local Apple Silicon inference. Same MIT terms apply.

Intended use

Local inference on Apple Silicon Macs (oMLX / mlx-lm). For base-model benchmarks and details, see the upstream card.