TokenAI-zer/Ternary-Bonsai-2-27B-MLX-oQ4-mtp

Ternary-Bonsai-2-27B-MLX-oQ4-mtp — oQ4 MLX quant of Bonsai 2 27B, with a grafted MTP head

Mixed-precision oQ4 (4.70 bits per weight) build of prism-ml/Ternary-Bonsai-2-27B, produced with oMLX 0.6.4 on an Apple M5 Max / 128 GB, with the multi-token-prediction head from Qwen/Qwen3.8-27B grafted in so oMLX can run self-speculative decoding.

I am not affiliated with Prism ML, pipenetwork or Alibaba Cloud. Created using Bonsai by Prism ML. See Provenance and License.

Format note: MLX safetensors. Not GGUF — these will not load in llama.cpp, Ollama or LM Studio. Use oMLX, or stock mlx-vlm ≥ 0.7.

What is different about this build

The Bonsai weights are ternary — every language weight is one of three values. The upstream GGUF packs them at a true 1.72 bpw with custom llama.cpp kernels; the official MLX release needs prism-ml's own runtime and ships without vision and without MTP.

These builds instead use stock MLX quantization, so they run in unmodified oMLX and mlx-vlm, and they keep both:

• —the vision tower, 333 tensors, untouched in BF16
• —an MTP head (15 tensors, 0.42 B parameters), grafted from Qwen/Qwen3.8-27B

Pick a variant — measured, not guessed

Each build's logits were compared against the bf16 source on the same five prompts (last-token logits, greedy): KL divergence, maximum relative error, agreement on the top-1 token, and overlap of the top-5.

The loss is monotonic and far from flat, which was not obvious beforehand: a ternary model still benefits from more bits. Affine quantization builds its grid from the group minimum and maximum, so with 4 levels over a symmetric {−a, 0, +a} group the levels land at −a, −a/3, +a/3, +a — zero itself is not representable, and zero is the most common value in a ternary tensor. More bits make the grid finer and approximate it better.

oQ2 is the only build that flips a top-1 prediction in this sample. Treat it as the "fits in less memory" option, not as a free lunch.

Caveat on the measurement: the reference is the bf16 conversion, not the original GGUF. An error inside that conversion would be inherited by every build here and would not show up in this table.

MTP: does the grafted head actually work?

The head comes from stock Qwen3.8-27B; the backbone is ternarized by Prism ML. Whether the two speak the same language is an empirical question, so here are the numbers, measured in oMLX on the oQ6 build:

End-to-end on the same model and machine: 13.1–13.4 tok/s without MTP, 15.9–23.9 tok/s with it; the longest run reached 23.9 tok/s against 13.1 for a comparable run without, about 1.8×.

For reference, a model of this family with its native MTP head accepts around 74% on the same setup. So the graft works, at reduced efficiency — expected, since the head was trained against full-precision hidden states and now reads ternarized ones.

Requirements

• —Apple Silicon, macOS 15+ (built and tested on M5 Max, 128 GB)
• —oMLX ≥ 0.6.4 for MTP speculative decoding, or stock mlx-vlm ≥ 0.7 for plain inference
• —about 15.85 GiB of free unified memory for a short prompt, more for long context

Usage

oMLX (what the MTP head is for)

git clone https://huggingface.co/TokenAI-zer/Ternary-Bonsai-2-27B-MLX-oQ4-mtp ~/.omlx/models/Ternary-Bonsai-2-27B-MLX-oQ4-mtp
omlx serve --model-dir ~/.omlx/models --port 8000

In the model settings enable `mtp_enabled`; without it the head is loaded but the speculative path never runs and you simply pay for the extra weights.

Stock mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("TokenAI-zer/Ternary-Bonsai-2-27B-MLX-oQ4-mtp")
prompt = apply_chat_template(processor, model.config, "Explain speculative decoding.", num_images=0)
print(generate(model, processor, prompt, max_tokens=512))

Provenance

Four steps, each of which can be checked independently:

1. 1.[Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) — Apache-2.0, the original 27B hybrid-attention VLM. The 15 mtp.* tensors come from here, unmodified.
2. 2.[prism-ml/Ternary-Bonsai-2-27B](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) — ternarization of that model to 1.72 bpw, with a blockwise Hadamard rotation applied online by their own kernels.
3. 3.[pipenetwork/Ternary-Bonsai-2-27B-MLX-bf16](https://huggingface.co/pipenetwork/Ternary-Bonsai-2-27B-MLX-bf16) — the Hadamard rotation unfolded back into the standard weight basis, so the model loads with no custom runtime. This is the source these builds quantize.
4. 4.This repository — MTP head grafted in (renamed to the language_model.mtp.* namespace mlx-vlm expects, no transform applied, since the weights are already in standard basis), mtp_num_hidden_layers set to 1, then oQ4 mixed-precision quantization with preserve_mtp.

The graft follows the approach documented by decent-jawfish/bonsai-2-27b-mtp for the GGUF format: the MTP tensors are stock Qwen weights and must not receive the Hadamard transform. No code was copied from it.

Known caveats

• —Quantization is lossy, and the table above says exactly how much. Compare against a higher build before reporting a regression.
• —Task-level benchmarks have not been run. The divergence numbers measure fidelity to the bf16 source, not intelligence.
• —MTP acceptance was measured on oQ6 only; the other builds are expected to behave similarly but this was not verified.
• —Vision is carried through in BF16 but VQA accuracy was not evaluated.
• —The grafted head is not a substitute for one trained on these weights.

License

Apache License 2.0, inherited through the whole chain. LICENSE and NOTICE.txt from Prism ML ship in this repository unchanged, as Apache-2.0 §4(d) requires.

As the upstream NOTICE asks: Created using Bonsai by Prism ML. The base model is Qwen3.8-27B, Copyright 2026 Alibaba Cloud, Apache 2.0.

Acknowledgements

• —Prism ML for Bonsai and the ternarization work.
• —pipenetwork for unfolding the Hadamard rotation into a stock-runtime MLX build.
• —Qwen team / Alibaba Cloud for Qwen3.8-27B and its MTP head.
• —oMLX for the oQ quantizer and the speculative decoding runtime.