Modelpublic

ayshinko/Bonsai-2-27B-PTQ1_0-MTP-Uncensored-Ready-GGUF

sourceHugging Faceapache-2.0updated 6d agoView on Hugging Face

0likes465downloads

Model Card

Bonsai 2 27B PTQ1_0 + MTP | 12GB GPU | Uncensored-Ready (Optional OrcaRouter LoRA)

27B-Class Local AI. Built for 12GB GPUs. Bonsai PTQ1_0 + ProCreations MTP. Up to 91.178 tokens/s in our measured short-context RTX 4070 Super benchmark with the optimized backend.

A memory-efficient, 27B-class local-inference GGUF built for a single consumer GPU with roughly 12 GB VRAM. It combines PrismML's PTQ1_0 ternary-quantized Bonsai 2 backbone with ProCreations' Bonsai-adapted MTP (multi-token prediction) speculative-decoding head, exported with a Q80 prediction head and verified on the optimized Prism llama.cpp fork with the **PR #218 PTQ10 CUDA kernels. It is Uncensored-Ready: an optional, separately-downloaded OrcaRouter rank-1 LoRA can alter runtime behavior, but the shipped GGUF itself is unchanged and uncensored behavior is off by default**.

Independent community derivative. This repository is an independent, community-made derivative of work by PrismML and ProCreations. It is not affiliated with, endorsed by, or maintained by PrismML. The MTP prediction head was trained by ProCreations, not by the author of this merge. The PTQ1_0 / PR #218 CUDA-optimized inference path comes from PrismML's llama.cpp fork including PR #218 (developed by sudoingsx).

At a Glance — Verified Facts

Metric	Value	Notes
Model class	27B parameters	Bonsai 2 architecture (`qwen35`)
GGUF file	5.96 GiB (6,397,969,728 bytes)	single file
GPU tested	NVIDIA RTX 4070 SUPER 12 GB	Ada, sm_89, driver 610.57.04
Short-context MTP 2 speed	91.178 tok/s	56-token prompt → 500 tokens
Long-context MTP 2 speed	69.140 tok/s	17,531-token prompt → 500 tokens
MTP 2 VRAM (short-ctx)	10,663 MiB	single-process, idle GPU, ctx 40960
Max tested context	40,960	configured in benchmark runs
Backend	Prism llama.cpp fork (PR #218)	build 10718, commit `3443ddece`

These figures describe different measurements and must not be conflated: - The 91.178 tok/s short-context MTP 2 number is a single specific benchmark on a specific GPU/backend and does not apply to every context length or GPU. - The GGUF file size (≈5.96 GiB) is not the same as total VRAM usage (MTP 2 ≈ 10.7 GB at the tested context); VRAM depends on context, KV cache, MTP settings, runtime, and other GPU memory consumers. - No claim of universally superior speed or efficiency is made without a directly comparable benchmark.

About the parameter count shown in Hugging Face search: This repository contains two GGUF files — the 27.32B-parameter base model (6.4 GB) and a ~2.42M-parameter optional LoRA adapter (9.6 MB). Hugging Face's model indexer currently picks the adapter file when determining the listed parameter count. The actual base model has 27,320,697,856 parameters (~27.32B). The LoRA is a separate runtime adapter that is never merged into the base weights.

📦 Download Prism Model Manager

Jump to Prism Model Manager integration ↓

Manage, load, and benchmark this model (and any GGUF) with the independent open-source terminal manager for Omarchy / Arch Linux:

—GitHub: <https://github.com/Ayshinko/prism-model-manager>
—Release v3.0.1 (integrated): <https://github.com/Ayshinko/prism-model-manager/releases/tag/v3.0.1>

Independent community project — not affiliated with, endorsed by, or maintained by PrismML.

1. What it is and who it's for

Targets local-AI users on a single consumer GPU with roughly 12 GB VRAM who want a 27B-class model with:

—Memory efficiency from PTQ1_0 ternary quantization (backbone fits in a ~6.0 GB single file),
—Higher decode throughput from an embedded MTP speculative-decoding head,
—Optional runtime behavior changes (Uncensored-Ready) via a separate OrcaRouter LoRA.

It requires a Prism llama.cpp fork (not stock upstream llama.cpp) to load PTQ1_0 tensors. See Required backend.

2. Model file, size, and checksum

Field	Value
File	`Bonsai-2-27B-PTQ1_0-MTP-Q8_0.gguf`
Size	6,397,969,728 bytes (≈ 5.96 GiB)
SHA256	`4c21bfcfa7e643d32179db409751941c5d1f7fcf188901c286e37d3913809cd1`
Architecture	`qwen35` (Hybrid-Attention + SSM, Bonsai 2)
Blocks	65 (64 backbone + 1 MTP)
Context (metadata)	262,144
Head quant	Q8_0 (matrix) / F32-norm multipliers
`general.file_type`	141

The on-disk GGUF is not an "abliterated" model. Its behavior is the original Bonsai backbone + ProCreations MTP head. The optional OrcaRouter LoRA is a separate download and is not merged into this file.

3. Benchmark charts (verified data)

All charts are diagrams computed from the verified benchmark record — they are not GPU screenshots and are not fabrications.

4. Verified short-context benchmark table

Backend PR #218, context 40960, KV q80/q80, FA on, -ngl 99, batch 2048, ubatch 512, Batch-Invariant OFF; 1 warm-up + 3 measured runs, 56-token input → 500 tokens/run. Median over 3 measured runs.

MTP	n-max	Median gen tok/s	Acceptance	VRAM after (MiB)
OFF	—	58.469	N/A	8729
`draft-mtp`	1	80.741	76.6%	10065
`draft-mtp`	2	91.178	65.1%	10663
`draft-mtp`	3	83.534	56.8%	11169
`draft-mtp`	4	69.314	46.5%	11771

All responses: 500 predicted tokens, stop limit, 0 CUDA errors, 0 CPU fallback, no early stop.

5. Verified long-context benchmark table

Same backend/settings, 17,531-token input (actual context ≈ 18,031 within a 40,960 window), 500 tokens/run. Median over 3 measured runs.

MTP	n-max	Median gen tok/s	Acceptance	VRAM after (MiB)
OFF	—	45.889	N/A	8791
`draft-mtp`	1	60.875	61.7%	10058
`draft-mtp`	2	69.140	69.0%	10578
`draft-mtp`	3	67.038	56.3%	11177
`draft-mtp`	4	56.192	42.5%	11773

Do not treat the short-context number as the long-context number: with ~17.5K tokens resident, every config is ~15–24% slower than its short-context equivalent.

6. Why MTP 2 is the fastest tested draft length

n-max=2 is the fastest in both contexts (91.2 short / 69.1 long tok/s). Going to n-max=3/4 reduces acceptance (and thus effective speedup) while adding VRAM. In the table above, MTP1 ≈ +38% short / +33% long over MTP-off; MTP2 ≈ +56% short / +51% long over MTP-off. For most ~12 GB users `draft-mtp` n-max=1 remains the recommended default because it gives the best balance of speed, acceptance, and VRAM headroom.

7. Hardware requirements and VRAM caveats

GPU	Status
RTX 4070 SUPER 12 GB	TESTED — use the exact measured settings and results above
RTX 4070 12 GB	NOT TESTED — similar capacity; performance and fit must be validated separately
RTX 4060 Ti 16 GB	NOT TESTED — potentially relevant due to VRAM capacity; no performance/compatibility claim
RTX 4060 8 GB	NOT TESTED — cannot claim full-GPU operation under the verified 10.7 GB MTP 2 config

For an 8 GB GPU: reduced context, different KV quantization, and/or CPU offloading (-ngl < 99) would be necessary and have not been validated by the published benchmark.

No FPS, tok/s, VRAM, or compatibility numbers are invented for the RTX 4060, 4060 Ti, or RTX 4070. Actual VRAM depends on context, KV cache, MTP settings, runtime, and other GPU memory consumers; always re-validate for your hardware.

Verified VRAM (single-process, idle GPU)

Measured absolute nvidia-smi at context 40960, KV q80/q80, FA on, -ngl 99. VRAM is essentially flat during generation, so peak ≈ load.

Mode	VRAM	Fits 12 GB?
MTP off	~8.7 GB	Yes, generous headroom
`draft-mtp` n-max=1	~10.0 GB	Yes, ~2 GB headroom
`draft-mtp` n-max=2	~10.7 GB	Yes, ~1.3 GB headroom
`draft-mtp` n-max=3	~11.2 GB	Yes, ~1.1 GB headroom
`draft-mtp` n-max=4	~11.8 GB	Marginal — near the 12 GB limit

These are absolute samples in a single-process idle GPU. Other GPUs of the same size, other drivers/builds, and concurrent workloads will differ.

8. Prism Model Manager 3.0 integration

[Prism Model Manager](https://github.com/Ayshinko/prism-model-manager) is an independent, community-developed terminal UI for managing GGUF models with a Prism backend, built for Omarchy / Arch Linux. It is not an official PrismML application. It is a general-purpose GGUF model manager (not a Bonsai-only launcher).

Verified capabilities (from the released v3.0.0 source/README):

—Discovering and loading/managing local GGUF models with per-model profiles.
—Selecting a compatible inference backend and validating required flags from its --help.
—Configuring context size, GPU layers, KV cache quantization, Flash Attention, and sampling.
—Controlling MTP speculative decoding (MTP, MTP_MODE, MTP_DRAFT_FLAG, MTP_DRAFT_MAX) — v3.0.0 defaults MTP_MODE=draft-mtp mapped to --spec-type draft-mtp.
—Managing local llama-server workflows (start/stop, live logs, status, VRAM/RAM monitoring, quick chat tests, raw benchmarks, Web UI).
—Supporting the specialized Prism backend (PTQ10/PQ20 detection via nvidia-smi).
—Native LoRA support with a per-model "Uncensored LoRA" control (LORA_ENABLED / LORA_SCALE / LORA_PATH), wired to the backend's --lora-scaled flag, plus a "LoRA Scale Intelligence A/B" benchmark that scores behavior with and without the adapter.

<details> <summary>Prism Manager native LoRA path (verified in v3.0.0 source)</summary>

In the model-settings TUI you can set Uncensored LoRA = on, point LoRA file at the OrcaRouter adapter you downloaded, and set LoRA scale (start at 2). The manager then passes --lora-scaled <adapter>:<scale> to llama-server automatically, after checking that the backend advertises --lora-scaled. For full control you can also use the manual backend flags in section 11.

Uncensored LoRA   [on]
LoRA scale        [2]
LoRA file         [~/Models/bonsai-abliterate-lora.gguf]

</details>

Install and recommended profile

The [integrated PMM 3.0.1 release](https://github.com/Ayshinko/prism-model-manager/releases/tag/v3.0.1) bundles a verified PR218-compatible llama-server — users don't need to download or build the backend separately. See the manual launch example for exact llama-server arguments, and --dry-run / --check in Prism Manager to preview and validate a command before launching.

bash

# Option 1: integrated release (recommended)
wget https://github.com/Ayshinko/prism-model-manager/releases/download/v3.0.1/prism-model-manager-3.0-linux-x86_64-cuda.tar.gz
tar xzf prism-model-manager-3.0-linux-x86_64-cuda.tar.gz
cd prism-model-manager-3.0-linux-x86_64-cuda
./install.sh
export PATH="$HOME/.local/bin:$PATH"

# Option 2: git clone (source only)
git clone https://github.com/Ayshinko/prism-model-manager.git
cd prism-model-manager && ./install.sh
export PATH="$HOME/.local/bin:$PATH"

First launch (point at your model directory; the backend is pre-configured with the integrated installer or set PMM_SERVER_BIN manually for git installations):

bash

PMM_MODEL_ROOT="$HOME/Models" \
PMM_SERVER_BIN="$HOME/path/to/prism/llama-server" \
prism-model-manager

Then set these recommended MTP settings for a 12 GB card (validated in v3.0.0 docs and consistent with the benchmark parameters):

bash

# config.env / TUI values for a 12 GB card
CTX=40960
CTK=q8_0
CTV=q8_0
NGL=99
BATCH=2048
UBATCH=512
FLASH=on
MTP=on
MTP_MODE=draft-mtp       # must be advertised by the backend's --spec-type
MTP_DRAFT_FLAG=--spec-draft-n-max
MTP_DRAFT_MAX=1          # n-max=2 is faster but uses ~0.6 GB more and lower acceptance

The failed to fit params ... n_gpu_layers already set by user to 99, abort startup warning on MTP configs is the auto-tuner aborting because -ngl 99 is user-forced; it does not move layers to CPU.

9. Required compatible inference backend

This model requires a Prism llama.cpp fork with the custom PTQ10 path and the MTP (`--spec-type draft-mtp`) support the model metadata depends on. **The stock upstream llama.cpp release will not load this file's PTQ10 tensors. The exact build used for the benchmarks above is the Prism fork at PR #218** (3443ddece) with the Hadamard-inverse MTP embedding fix applied.

—The benchmarked PR #218 PTQ1_0 CUDA optimization is not in every public Prism build; check the exact revision.
—PR #218 was developed by sudoingsx; it is an upstream open-source contribution within the Prism llama.cpp fork, not the author's or this model's work.

10. MTP configuration and runtime requirements

Exact recommended MTP 2 command shape

bash

# llama-server arguments (short-context-verified config, MTP n-max=2)
-c 40960 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 \
--spec-type draft-mtp --spec-draft-n-max 2 \
--jinja --reasoning-effort medium

Recommended default for most 12 GB users: `--spec-draft-n-max 1` (best speed/VRAM balance). Use n-max=2 when you want peak short-context speed and have ~1.3 GB headroom.

MTP acceptance varies with prompt/content; speedups are workload-dependent and may not appear on every hardware/backend revision. Use context ≈ 40960 on a 12 GB card; do not raise it toward the 262,144 metadata limit (the full-context KV would not fit).

11. Optional OrcaRouter Uncensored LoRA

The main GGUF is not inherently uncensored — it is the original Bonsai backbone + ProCreations MTP head. Uncensored-related runtime behavior is OPTIONAL and requires the separate OrcaRouter adapter to be enabled. OrcaRouter is a separate project (not the author of this repo), and neither the refusal direction nor the adapter was created, trained, or developed here. We credit it clearly and do not call the default GGUF an "abliterated" model.

—Adapter source: <https://github.com/Continuum-AI-Corp/OrcaBonsai-27B-Uncensored>
—Artifact: gguf/bonsai-abliterate-lora.gguf (rank-1 LoRA, F32, 129 sites)
—SHA256: f1669534803d340a496015f5c45125f3437b4d13ec764f40e34488ce83967f42
—License: Apache-2.0 (upstream)
—In this repo: adapters/bonsai-abliterate-lora.gguf — redistributed with attribution under Apache-2.0

The adapter is provided as a convenience download. The main GGUF weights remain unchanged; the LoRA is applied at runtime as two extra matmuls and is never merged in.

Compatibility status — VERIFIED independently

We independently tested the adapter against this combined PQ1 + MTP GGUF on the PR #218 backend (see ORCAROUTER-COMPAT-REPORT.md):

1.Loads cleanly on this merged GGUF (0 CUDA errors).
2.Genuinely applied in the graph: scale-0 output is byte-identical to the no-LoRA baseline, and scale-2 output visibly differs — the documented OrcaRouter discriminator.
3.Coherent output.
4.Works together with MTP: draft-mtp n-max=2 + LoRA generated cleanly at ~82 tok/s.
5.GPU memory and generation speed were measured for each configuration.

Use the native Prism Manager path for the adapter: in Prism Model Manager v3.0.0 you can enable the per-model "Uncensored LoRA" control (LoRA file → your downloaded bonsai-abliterate-lora.gguf, scale → 2), and it will pass --lora-scaled automatically (see section 8). The direct backend-flags manual commands below remain valid for any setup.

Reproducible startup

bash

# A. main GGUF without LoRA (default, unchanged)
llama-server -m Bonsai-2-27B-PTQ1_0-MTP-Q8_0.gguf -c 40960 -ngl 99 -fa on \
  -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 --jinja --host 127.0.0.1 --port 8080

# B. main GGUF with OrcaRouter LoRA at full strength (scale 1)
llama-server -m Bonsai-2-27B-PTQ1_0-MTP-Q8_0.gguf -c 40960 -ngl 99 -fa on \
  -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 --lora adapters/bonsai-abliterate-lora.gguf --jinja

# C. main GGUF with OrcaRouter LoRA + MTP (draft-mtp, n-max=2)
llama-server -m Bonsai-2-27B-PTQ1_0-MTP-Q8_0.gguf -c 40960 -ngl 99 -fa on \
  -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 --lora adapters/bonsai-abliterate-lora.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 --jinja

For higher ablation strength OrcaRouter documents --lora-scaled adapter.gguf:2 (a sample of one prompt tells you little; their own eval still saw ~6% refusals at full strength).

When using the Prism Model Manager integrated installer: the bundled PR218 backend is installed to $PREFIX/lib/prism-llama/. Download adapters/bonsai-abliterate-lora.gguf and place it in an accessible directory, then set Uncensored LoRA = on and LoRA file to that path in the PMM settings TUI. The manager passes --lora-scaled <path>:<scale> automatically.

Enabling a LoRA changes model behavior and may change speed and memory. Use appropriate, lawful behavioral evaluation and compare with/without. Adapter support here verified on one backend build; regard this as load/apply/coexistence confirmation, not a guarantee of any specific behavioral outcome.

12. Manual llama-server launch example

bash

/path/to/prism/llama-server \
  -m /path/to/Bonsai-2-27B-PTQ1_0-MTP-Q8_0.gguf \
  -c 40960 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 \
  --spec-type draft-mtp --spec-draft-n-max 1 \
  --jinja --reasoning-effort medium \
  --host 0.0.0.0 --port 8080

13. Compatibility and known limitations

—Requires the Prism llama.cpp fork with PTQ1_0 + MTP support; not stock llama.cpp.
—qwen35 hybrid backbone; quantization variants/backends are not interchangeable.
—MTP acceptance varies with prompt/content; speedups are workload-dependent.
—12 GB VRAM is the design envelope; not every 12 GB GPU/back-build matches the measured throughput. Concurrent workloads reduce headroom, especially at n-max=4.
—The upstream Python gguf library may not parse the custom ternary type tags; use the Prism loading path.
—Vision projector (mmproj) and tool-calling are supported by the backend and by Prism Model Manager but were not re-validated for this specific merged file.
—Uncensored behavior is optional and off by default; it depends on the separate OrcaRouter adapter and was verified only for load/apply/coexistence on one build.

14. Tested hardware/software environment

Item	Value
GPU	NVIDIA GeForce RTX 4070 SUPER, 12 GB (12,282 MiB), Ada (sm_89)
Driver	610.57.04
Backend	Prism llama.cpp fork, build `10718`, commit `3443ddece` (PR #218 + Hadamard fix)
OS	Linux (Arch/Omarchy), x86_64
Generation settings	temp 0, seed 42, 500 tokens/run, `--reasoning-effort medium`, `--jinja`

15. Model provenance and contribution breakdown

<img src="assets/architecture-provenance.png" alt="Architecture and provenance diagram" width="88%"> Diagram for illustration; see the tables below for exact revisions and checksums.

Component	Origin	Upstream revision	SHA256
Backbone (PTQ1_0, 64 blocks)	`prism-ml/Ternary-Bonsai-2-27B-gguf`	`6ed5e12bf84b7a63069882c91dd9e9218647d17b`	`53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3`
MTP head donor (PQ2_0 backbone + Q8_0 MTP)	`ProCreations/Ternary-Bonsai-2-27B-MTP`	`efffdea64c1f9e93cc7fa6bb24f72ae9d66ecf51` (r3-mtp)	`3cb3f0056d2e34ee44245a64396004a21f8492573d6ce1266ec4b7222c131dd4`
MTP trained-head checkpoint	`ProCreations/Ternary-Bonsai-2-27B-MTP`	same HEAD (r3-mtp)	`7a4a18b2d02116ef184d1b0ee4af46d829825ff2c042f79cf37ef8a03c399218`

What the author of this merge contributed:

—Combining the PTQ1_0 backbone with ProCreations' trained MTP head into a single Q8_0-head GGUF and applying the compatibility integration (metadata alignment).
—Byte-level per-tensor validation of all 866 tensors and reproducible benchmarks.
—This consumer deployment workflow and documentation.

What the author did NOT do: train the MTP head (ProCreations), invent PTQ1_0 quantization (PrismML), or develop PR #218's CUDA kernels (sudoingsx / PrismML PR #218).

16. License, NOTICE, and credits

This distribution is provided under the Apache License 2.0 (see LICENSE), consistent with both upstream sources.

—Original model: Bonsai 2 27B by PrismML, Inc. (Apache-2.0). Attribution requested: "Created using Bonsai by Prism ML."
—MTP head + integration: ProCreations (Apache-2.0). Fine-tuned the Qwen3.8-27B MTP head against frozen Bonsai features; added runtime inverse-rotation handling for the MTP embedding lookup. Main Bonsai tensor payloads unchanged.
—Qwen lineage: built from Qwen3.8-27B, Copyright 2026 Alibaba Cloud (Apache-2.0).
—PR #218 CUDA kernels: developed by sudoingsx within the Prism llama.cpp fork.
—OrcaRouter (optional adapter): Continuum-AI-Corp / OrcaRouter (Apache-2.0) — refusal-direction ablation LoRA; provided separately, not bundled here.

See NOTICE for the required attribution text.

17. Reproducible benchmark methodology

For every number above the following are fixed and recorded:

—GPU model & VRAM, driver version.
—Backend build id and commit.
—Model file name + SHA256 (section 2).
—Prompt token count (56-token short / 17,531-token long) and generated token count (500/run).
—Maximum configured context (40,960) and actual occupied context (556 / 18,031).
—KV cache quantization (q8_0/q8_0), Flash Attention (on), -ngl 99.
—MTP draft length (--spec-draft-n-max N) and GGML_CUDA_BATCH_INVARIANT (off).
—VRAM measurement method (absolute nvidia-smi samples around each request, not isolated allocation deltas). Prompt processing reported separately from generation; prompt processing speed is never presented as generation speed.
—Number of measured runs (3) and median (not max) reported.
—All runs had zero CUDA errors and zero CPU fallback.

Full per-run detail is in the raw benchmark records (see REPRODUCE.md).

18. Independent community project disclaimer

This is an independent, community-made project. It is not affiliated with, endorsed by, or maintained by PrismML, ProCreations, Qwen/Alibaba, sudoingsx, the OrcaRouter team, or Omarchy. Brands, models, and code remain the property of their respective owners and keep their own licenses. Nothing here is provided with warranty; validate everything for your own hardware.