vcruz305/Bonsai-27B-GGUF

Bonsai-27B — Standard GGUF Ladder (Q3KM → Q8_0)

Imatrix GGUF quants of prism-ml's Bonsai-27B (Qwen3.6-27B backbone, 262K context, hybrid attention, vision), covering the quality tiers between the official QAT releases and F16.

Which repo should you use?

If you have ≤8GB: use prism-ml's official QAT quants, not these.

• —Q1_0 (3.8GB) — 89.5% of F16 quality at 1.125 bits
• —Ternary (7.2GB) — 95% of F16

Those are quantization-aware-trained with custom kernels — at their sizes, they beat anything post-training quantization can produce, including anything in this repo.

This repo covers the gap above them: the standard ladder for 12–32GB setups where you want maximum quality per GB, quantized from prism-ml's own F16 GGUF with an importance matrix (63KB coding/reasoning calibration corpus).

Quants

All tiers individually smoke-tested (coherent code generation, chat template engages via --jinja). Q8_0 quantized without imatrix (unneeded at 8-bit); all others use the included Bonsai-27B.imatrix. Decode rates scale with your memory bandwidth — a 936GB/s GPU (RTX 3090-class) will run ~3× these numbers.

Both official mmproj files are included for vision (--mmproj Bonsai-27B-mmproj-Q8_0.gguf).

Speculative decoding with the official DSpark drafter (+31% measured)

prism-ml ships a DSpark drafter (1.8GB) trained against this exact target model — and it works with this repo's quants, verified:

Output at temperature 0 is byte-identical with and without the drafter (speculative decoding is verify-always — the drafter can only affect speed, never quality).

Requires [PrismML-Eng's llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp) (branch prism) — the dspark drafter architecture is not in mainline llama.cpp (yet; their pr/* branches suggest upstreaming is underway). Build and run:

git clone --depth 1 --branch prism https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server

./build/bin/llama-server \
  -m Bonsai-27B-Q4_K_M.gguf \
  -md Bonsai-27B-dspark-Q4_1.gguf \
  --spec-type draft-dspark --spec-draft-n-max 4 \
  -ngl 999 -ngld 999 -fa on -c 16384 -np 1 --jinja

Gotchas (from prism-ml's SPECULATIVE.md + our testing):

• —--spec-draft-n-max must be exactly 4 (the drafter's block size) — other values crash at the first draft round
• —Speculative mode disables cross-request prompt caching and concurrency (-np 1) — best for single-user/agent use
• —Use -c 16384+ and a generous max_tokens — the model regularly thinks 1.5–2K tokens before the visible answer
• —CUDA only for now (their Metal spec path is still being optimized)
• —Expect bigger gains on higher-bandwidth GPUs (prism-ml measured ~1.9× on datacenter CUDA; our +31% is on a bandwidth-bound GB10)

Speed hunt: fastest Bonsai-27B config on a DGX Spark (GB10)

We benchmarked every acceleration path available for this model on GB10 (2026-07-14). Full matrix — 3-pass medians, coherence-gated, temp 0:

Finding — 1-bit QAT makes speculative decoding counterproductive: against the Q10 target, every drafting scheme failed to beat base speed on this hardware: the official DSpark drafter *halved* throughput (21.6 tok/s at 78% acceptance), a Q2K requant of it was no better, Bonsai-8B-Q1_0 as a draft-simple drafter was an exact wash (43.9), and ngram speculation got 6% acceptance. At 3.5 GiB of weights, decode is no longer bandwidth-bound — there's nothing left for a drafter to amortize. KV-cache quantization and batch tuning were also flat. prism-ml's experimental megakernel/rmsnorm-qmv-fuse branch measured parity (42.8).

Practical guidance: on unified-memory hardware (GB10, Apple Silicon, Strix Halo), run the QAT Q10/Q20 plain — save the drafter for the PTQ tiers in this repo, where it genuinely pays (+31% on Q4KM). On high-bandwidth discrete GPUs the tradeoff may differ — measure via timings.draft_n_accepted.

Provenance

• —Source: prism-ml/Bonsai-27B-gguf F16 (53.8GB) — quantized directly from their official F16 GGUF, no re-conversion
• —llama.cpp: mainline, commit cecbf5fb0 (quantization + baseline numbers); PrismML fork 62061f91 branch prism (drafter numbers)
• —imatrix: 63KB ChatML coding/debugging/reasoning corpus, 4096 ctx
• —LICENSE/NOTICE carried over from the upstream repo (Apache-2.0)
• —Quantized on NVIDIA DGX Spark (GB10, aarch64) — day-zero release, ~2.5h after the upstream drop

Report issues in the community tab — smoke-test failures, incoherence, or numbers that don't reproduce. Community benchmark reports welcome (include hardware, backend, and full launch command).