Jerrybro/bonsai2-27b-pq2-vs-fable-rtx5090

Bonsai 2 27B PQ2_0 vs Fable on a Single RTX 5090

This benchmark and reproducibility artifact compares single-GPU local LLM inference for Bonsai 2 27B / Qwen3.8 27B PQ2_0 GGUF through the Bonsai llama.cpp fork, its low-VRAM Q4_0 KV-cache quantization route with multimodal vision, and a Fable groupwise-int baseline. The controlled 210-question comparison on one RTX 5090 separates a quality-first Fable route from a 15,595 MiB sampled-peak Bonsai PQ2_0 + Q4_0-KV route.

[image]

Headline inference benchmark results

With the same PQ20 weights, Q40 KV reduced sampled peak VRAM by 4,206 MiB relative to FP16 KV while both profiles scored 190/210 in this run.

Q4_0 KV is cache compression, not Q4 weight quantization. Every Bonsai profile here uses the same PQ2_0 weights at 2.13 bits per weight.

Hardware under test

This table records the hardware and current host state; the currently observed driver must not be assumed to be the driver used for every historical measurement. The files under results/ are authoritative for the published benchmark values. This Bonsai package did not retain a complete runtime software-version manifest, so exact historical runtime and driver reconstruction is not available from this dataset alone.

Search terms: Bonsai 2 27B, Qwen3.8 27B, PQ20 2-bit quantization, GGUF, llama.cpp, RTX 5090, Q40 KV-cache quantization, low-VRAM local LLM inference, 256K / 262,144-token long context, multimodal vision, Fable, and groupwise-int benchmarking.

What is downloadable

• —results/comparison-summary.json: sanitized four-profile accuracy, latency, vision, VRAM, and paired-test aggregates.
• —results/long-context-summary.json: low-VRAM runtime and zero-cache long-context evidence.
• —experiment/controlled-contract.json: seed, sampling, output, prompt, and inclusion/exclusion contract.
• —assets/bonsai2-pq2-vs-fable-results.svg: reusable results visual.
• —Citation and split licensing files.

This dataset does not contain model weights, converted Fable artifacts, raw model outputs, private prompts, client data, chat history, credentials, local paths, service URLs, container images, or machine logs. The 210 raw questions remain withheld until per-source provenance and redistribution terms are cleared.

Download this research package

hf download Jerrybro/bonsai2-27b-pq2-vs-fable-rtx5090 \
  --repo-type dataset \
  --local-dir bonsai2-27b-pq2-vs-fable-rtx5090

Download weights only from their upstream publishers:

hf download prism-ml/Ternary-Bonsai-2-27B-gguf \
  Ternary-Bonsai-2-27B-PQ2_0.gguf \
  Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf \
  --local-dir bonsai2-27b-pq2

• —Bonsai 2 weights: prism-ml/Ternary-Bonsai-2-27B-gguf
• —Bonsai runtime: PrismML-Eng/Bonsai-demo
• —Fable upstream card: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF

The reported Fable baseline used a separate groupwise-int-v3 runtime artifact. Linking the upstream card identifies lineage; it does not imply that the exact converted artifact is redistributed here.

Reproduce the Bonsai runtime routes

1. Install the required fork runtime

Bonsai 2 PQ2_0 requires the Bonsai demo's fork build and activation transform. Do not run it with stock llama.cpp.

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
Set-Location Bonsai-demo
pwsh -File .\setup.ps1

The upstream setup script manages its supported binaries. Verify the selected release and backend before comparing results.

2. Launch PQ2_0 with FP16 KV

$env:BONSAI_FAMILY = 'bonsai2'
$env:BONSAI_MODEL = '27B'
$env:BONSAI_GGUF = "$PWD\..\bonsai2-27b-pq2\Ternary-Bonsai-2-27B-PQ2_0.gguf"
$env:BONSAI_MMPROJ = "$PWD\..\bonsai2-27b-pq2\Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf"
$env:BONSAI_IMAGE_MAX_TOKENS = '1024'
Remove-Item Env:BONSAI_KV4 -ErrorAction SilentlyContinue
pwsh -File .\scripts\start_llama_server.ps1

3. Launch the low-VRAM Q4_0-KV route

Use the same PQ2_0 model and projector, then change only the KV cache:

$env:BONSAI_KV4 = '1'
pwsh -File .\scripts\start_llama_server.ps1

For the published extended-context validation, set the explicit runtime context before launch:

$env:BONSAI_CTX = '262144'
$env:BONSAI_KV4 = '1'
pwsh -File .\scripts\start_llama_server.ps1

The extended run validated a 110,038-token zero-cache request in 65.856 seconds at 1,675.6 prompt tok/s. A separate 40,918-token low-VRAM request measured 2,340.6 prompt tok/s and 49.6 decode tok/s with a CPU-resident Q8_0 vision projector.

Controlled comparison method

The contract in experiment/controlled-contract.json fixes:

• —seed 42;
• —temperature 0, top-p 1.0, top-k 0, and min-p 0.0;
• —maximum output of 8 tokens;
• —thinking disabled;
• —the same exact-label system instruction for controlled profiles.

The controlled set contains PQ2 + FP16 KV, PQ2 + Q4_0 KV, and the Fable baseline. The Fable-inspired prompt profile is intentionally excluded because it changes the prompt and sampling policy.

Paired evidence

• —FP16 KV vs Q4_0 KV: one case moved in each direction; exact McNemar p = 1.0.
• —PQ2 Q4_0 KV vs Fable: Fable corrected 14 PQ2 misses while PQ2 corrected 5 Fable misses; exact McNemar p = 0.063568115234375.

The second result points toward Fable but does not cross the conventional 0.05 threshold in this one 210-case run.

Prompt steering did not transfer Fable's learned behavior

The Fable-inspired prompt profile scored 184/210, below the controlled PQ2 profiles at 190/210, and passed 2/3 rather than 3/3 vision checks. A system prompt can copy an explicit response policy; it cannot copy learned representations, distilled behavior, tool reliability, safety boundaries, or domain knowledge stored in model weights.

Treat prompt steering as a descriptive experiment. Weight-level transfer would require a separately licensed distillation or fine-tuning program and a new evaluation.

Recommended deployment split

These are workload routes, not universal model rankings.

Autotuning and portability boundary

No MoE autotune artifact is claimed for the Bonsai comparison. The companion RTX 5090 study tested device-local tuning directly and found that copying an RTX PRO 4000 Blackwell seed reduced matched throughput by 29.43%, despite both arms passing the same smoke gate.

That result supports a strict distinction:

• —Portable: experiment contract, launch method, measurement scripts, and configuration hypotheses.
• —Device-local: tuned kernel tables and any claimed optimum.

See Jerrybro/qwen38-flash-next-rtx5090-research for the underlying autotune evidence.

Benchmark boundaries and limitations

• —The 210 questions are a local exact-label selection suite, not a comprehensive intelligence or safety benchmark.
• —Raw questions are withheld pending per-source provenance review; this package therefore supports audit of aggregates and method, not independent rescoring of the exact suite.
• —The three generated vision cases are wiring and basic-perception checks, not a general VLM benchmark.
• —VRAM is sampled peak usage, not a hardware-instrumented allocation trace.
• —Results come from one host and one run family. Driver, runtime, thermal, and background-load changes require remeasurement.
• —P95 latency and peak VRAM were not retained for the prompt-steered profile.
• —The exact Fable converted artifact and its conversion procedure are not distributed here.

The raw suite drew from public benchmark families including AI2 ARC, BoolQ, HellaSwag, OpenBookQA, PIQA, and WinoGrande. Their questions are not republished in this dataset.

Licensing and attribution

• —Dataset documentation, aggregate results, experiment contract, and original visual: CC BY 4.0.
• —Original code, if added to this package, is covered by Apache License 2.0.
• —Upstream model weights, runtimes, benchmark sources, CUDA components, and conversion tools retain their upstream licenses. Nothing here relicenses them.

Citation

Use CITATION.cff, and cite the relevant upstream model, runtime, and benchmark sources when reporting derivative work.

Author

Jerry Sheen · Hugging Face: Jerrybro