CoolFace
Modelpublic

darkmaniac7/TokForge-Bonsai-2-27B-GGUF

sourceHugging Faceapache-2.0updated 3d agoView on Hugging Face
0likes30downloads
Model Card

TokForge

  • —Website: https://tokforge.ai
  • —Discord: https://discord.gg/Acv3CBtfVm
  • —Google Play: https://play.google.com/store/apps/details?id=dev.tokforge
  • —iOS TestFlight: https://testflight.apple.com/join/jnufjzRr
  • —Phone speed benchmarks: https://tokforge.ai/leaderboard/

Runs on-device in the TokForge app.

TokForge-Bonsai-2-27B-GGUF

Prism ML's Bonsai 2 27B ternary model, packaged for on-device Android inference in TokForge, with the settings we measured to run best on phones.

Created using Bonsai by Prism ML. All credit for the model, the ternary training and the GGUF packings goes to Prism ML; the base model is Qwen3.8-27B by Alibaba Cloud. The two GGUF files here are byte-identical to the upstream release. What TokForge adds is the Android runtime (Vulkan and CPU kernels for these packings) and the tested phone settings below.

Which file to use

FileSizePackingBest route in TokForge
Ternary-Bonsai-2-27B-PQ2_0.gguf7.21 GBPQ2_0, 2.13 bits per weightGPU (Vulkan) on supported Snapdragon phones
Ternary-Bonsai-2-27B-PTQ1_0.gguf5.95 GBPTQ1_0, 1.75 bits per weightCPU
  • —Have a Snapdragon 8 Elite Gen 5 or 8 Gen 3 phone with 16 GB of RAM? Pick PQ2_0. It is the larger file, but it is about 2.6x faster because it runs on the GPU.
  • —The original Snapdragon 8 Elite (SM8750) is not on the GPU list yet and runs the CPU route.
  • —Otherwise pick PTQ1_0 for the smaller download and the CPU route.

TokForge chooses the route and the settings for you when it loads either file. You need TokForge 1.3.4 or newer (currently on the Play beta track and as an APK on GitHub).

Measured speed

RedMagic 11 Pro (Snapdragon 8 Elite Gen 5, Adreno 840, 16 GB), short chat answers inside the app:

FileRouteDecode
PQ2_0Vulkan (GPU)5.6 tok/s
PQ2_0CPU2.13 tok/s
PTQ1_0CPU2.01 tok/s
PTQ1_0Vulkan (GPU)0.82 tok/s
  • —On Vulkan, the same short prompts finished in about 14.5 s, against 31.7 s on CPU.
  • —These numbers come from the TokForge GPU lab build, which uses the same profile TokForge 1.3.4 now applies automatically.
  • —They are short repeated tests under the phone's own clock management, not an all-day soak. Your phone's cooling and clocks will move them.
  • —Other phones:
  • —The Snapdragon 8 Gen 3 (SM8650) uses the same GPU route, but this file was not separately benchmarked there.
  • —Xiaomi phones on HyperOS cap an app at about 6 GiB of memory and closed both files in our tests.
  • —12 GB phones are untested.

More phones and models: tokforge.ai/leaderboard.

Settings TokForge uses

These are applied automatically; you do not need to set them by hand. They are also in manifest.json for other runtimes.

PQ2_0 on the GPU (Snapdragon 8 Elite Gen 5 / 8 Gen 3, non-Samsung)

SettingValueWhy
BackendVulkan, all layers on the GPU2.6x faster decode than CPU
Context2048 tokenslarger contexts faulted the GPU at the first prefill on this driver
Prefill batch128same reason
Microbatch1 tokenbatched prefill lost the GPU device on Adreno
KV cachef16 (keys and values)a quantized KV cache faulted the GPU on this driver
Flash attentionon
CPU threads64 was clearly slower, 8 was no better
GGML_VK_DISABLE_FUSION_MASK5normalization fusion is not correct on this driver
Thinkingoffkeeps answers short on a phone
Speculative decodingoffno measured speedup

PTQ1_0 on the CPU

SettingValue
BackendCPU
Context2048 tokens
Prefill batch128
CPU threads6
Thinkingoff
Speculative decodingoff
  • —Vulkan stays available for PTQ1_0 as an explicit choice, but it is slower today: the fast PTQ integer-dot GPU kernel is not yet correct on Adreno, so the GPU route falls back to a correct but slow path.
  • —Temperature and the other sampler settings are your own choice. The numbers above were measured at temperature 0 with seed 42 for repeatability, not as a chat recommendation.

Using it in TokForge

  1. 1.Install TokForge 1.3.4 or newer.
  2. 2.Open Models, then Browse, and search for TokForge-Bonsai-2-27B, or download a file from this page and import it.
  3. 3.Load the model. TokForge picks the GPU or CPU route and the settings above.

Files

FileBytesSHA-256
Ternary-Bonsai-2-27B-PQ2_0.gguf7,206,168,9283907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1
Ternary-Bonsai-2-27B-PTQ1_0.gguf5,946,648,92853107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3
manifest.jsonTokForge route, settings and measurements, machine-readable
LICENSE, NOTICE.txtApache-2.0 license and notice, copied from upstream

Both GGUF files are byte-identical to upstream `prism-ml/Ternary-Bonsai-2-27B-gguf` at revision 6ed5e12bf84b7a63069882c91dd9e9218647d17b. The upstream repo also carries the F16 weights and the vision projector (mmproj) files, which TokForge does not use yet.

Credit chain

  1. 1.Base model: `Qwen/Qwen3.8-27B` by Alibaba Cloud (Apache-2.0)
  2. 2.Bonsai 2 27B ternary model and GGUF packings (PQ2_0, PTQ1_0): `prism-ml/Ternary-Bonsai-2-27B-gguf` by Prism ML (Apache-2.0). See their whitepaper and demo repository.
  3. 3.Prism ML's llama.cpp fork with the ternary kernels (CUDA, Metal): `PrismML-Eng/llama.cpp`
  4. 4.TokForge: the Android port of those packings (CPU, OpenCL and Vulkan kernels, including the Adreno and Mali Vulkan repairs) and the phone settings above.

License

Apache-2.0, inherited from Prism ML's release and the Qwen base model. LICENSE and NOTICE.txt are included unchanged. Per Prism ML's notice: created using Bonsai by Prism ML.

Community