jinreiyu/clone-Ternary-Bonsai-8B-gguf

<p align="center"> <img src="./assets/bonsai-logo.svg" width="280" alt="Bonsai"> </p>

<p align="center"> <a href="https://prismml.com"><b>Prism ML Website</b></a> &nbsp;|&nbsp; <a href="https://github.com/PrismML-Eng/Bonsai-demo/blob/main/ternary-bonsai-8b-whitepaper.pdf"><b>White Paper</b></a> &nbsp;|&nbsp; <a href="https://github.com/PrismML-Eng/Bonsai-demo"><b>Demo &amp; Examples</b></a> &nbsp;|&nbsp; <a href="https://discord.gg/prismml"><b>Discord</b></a> </p>

Ternary-Bonsai-8B-gguf

Ternary (1.58-bit) language model in GGUF Q2_0 format for llama.cpp

<p align="center"> <img src="./assets/frontier.svg" width="680" alt="Pareto Frontier"> </p>

Resources

• —[White Paper](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/ternary-bonsai-8b-whitepaper.pdf)
• —[Demo repo](https://github.com/PrismML-Eng/Bonsai-demo) — examples for serving, benchmarking, and integrating Bonsai
• —[Discord](https://discord.gg/prismml) — community support and updates
• —Kernels: Q20 is not yet in mainline `llama.cpp`. Use our fork at [PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp) (`prism` branch, default) which adds Q20 support for CPU (NEON/generic) and Metal. Upstream PR coming soon.

Model Overview

Quantization Format: GGUF Q2_0 (g128)

Each weight takes a value from {-1, 0, +1}, with one shared FP16 scale per group of 128 weights:

w_i = scale_g * t_i,    t_i in {-1, 0, +1}

Q2_0 encodes each weight as a 2-bit code q in {0, 1, 2, 3}, dequantized via w = (q - 1) * scale. One 128-element block is 34 bytes (2 bytes FP16 scale + 32 bytes of packed 2-bit codes) for an effective 2.125 bits/weight. The fourth code point (q = 3, reconstructing to +2 * scale) is reserved for future extensions; for ternary weights it is unused.

Memory

Files in this repo

Quickstart

Build from the Prism fork

git clone https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp
cmake -B build -DGGML_METAL=ON   # or -DGGML_CUDA=ON, -DGGML_VULKAN=ON
cmake --build build -j

llama.cpp CLI

./build/bin/llama-cli \
  -m Ternary-Bonsai-8B-Q2_0.gguf \
  -p "Explain quantum computing in simple terms." \
  -n 256

llama.cpp server

./build/bin/llama-server -m Ternary-Bonsai-8B-Q2_0.gguf -c 4096

Throughput (llama.cpp, Apple M4 Pro 48 GB)

Flags: -ngl 99 -fa 1 for Metal; -ngl 0 -fa 1 -t 10 for CPU.

Benchmarks

Evaluated with EvalScope v1.4.2 + vLLM 0.15.1 on NVIDIA H100 under identical infrastructure, generation parameters, and scoring. All models are in the 6B-9B parameter range.

Ternary Bonsai 8B ranks 2nd among all compared models despite being 1/8th the size.

Intelligence Density

density = -ln(1 - score/100) / size_GB

Citation

@techreport{ternarybonsai,
    title   = {Ternary Bonsai: 1.58-bit Language Models at 8B, 4B, and 1.7B Scale},
    author  = {Prism ML},
    year    = {2026},
    month   = {April},
    url     = {https://prismml.com}
}

Contact

For questions, feedback, or collaboration inquiries: contact@prismml.com