CoolFace
Apppublic

tengmoon68/bonsai-27b-1bit-demo

sourceHugging Faceupdated 28d agoView on Hugging Face
0likes
App README

Bonsai 27B · 1-bit MLX Demo

An interactive showcase and live chat console for `prism-ml/Bonsai-27B-mlx-1bit` — the first 27B-class model that runs on a phone (~3.9 GB, 1.125 bits per weight).

What's inside

  • —Live console — a streaming chat UI that connects to an OpenAI-compatible MLX server running on your own Apple Silicon Mac. Everything runs in your browser; no requests go through this Space.
  • —Why 1-bit — how binary weights ({−1,+1} + shared scale) shrink a 54 GB FP16 model to ~3.9 GB.
  • —Performance — measured benchmarks, memory footprints, and device speeds.
  • —Run locally — copy-paste setup for the official Bonsai demo repo and for mlx_lm.server directly.

Quick start (macOS, Apple Silicon)

bash
git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
BONSAI_FAMILY=bonsai ./setup.sh
./scripts/start_mlx_server.sh        # serves http://127.0.0.1:8081

Then open this Space, click Connect in the console, and chat with the model through your local server.

Notes

  • —MLX runs only on Apple Silicon. The same 1-bit weights are also published in GGUF (prism-ml/Bonsai-27B-gguf) for llama.cpp on CUDA / Metal / CPU.
  • —The 1-bit MLX pack needs the PrismML MLX fork until mlx#3161 merges upstream; setup.sh builds it automatically.
tengmoon68/bonsai-27b-1bit-demo · CoolFace