CoolFace
Apppublic

knoxel/bitnet-cpp-explorer

sourceHugging Faceupdated 5mo agoView on Hugging Face
0likes
App README

⚡ BitNet b1.58 2B4T — CPU Inference with bitnet.cpp

The fast version. This Space compiles Microsoft's bitnet.cpp from source and runs the official GGUF model with the I2_S lossless kernel — achieving 4-10× speedup over the transformers-based version.

Architecture

┌─────────────────────────────────────────────────┐
│  Gradio UI (Python)                             │
│    ↕ OpenAI-compatible streaming API            │
│  llama-server (bitnet.cpp, port 8080)           │
│    ↕ I2_S ternary kernel (additions only)       │
│  ggml-model-i2_s.gguf (1.1 GB, lossless)       │
└─────────────────────────────────────────────────┘

Why bitnet.cpp?

EngineSpeedLossless?Notes
bitnet.cpp I2_S~10-15 tok/s✅ 100%Optimized ternary kernel
transformers (bf16)~1.4 tok/s✅No ternary optimization
llama.cpp TQ1_0~2 tok/s❌ 1.4%Quality degraded

Features

  • —💬 Streaming Chat — Real-time conversation with live tok/s stats
  • —📊 Benchmark — Greedy generation with detailed performance metrics
  • —📈 Paper Results — Published comparison table from the technical report
  • —🏗️ Architecture — How ternary weights enable multiplication-free inference
  • —⚙️ System Info — Live CPU/memory stats

How it works

  1. 1.Docker build compiles bitnet.cpp with I2_S kernel optimizations
  2. 2.Container startup launches llama-server on localhost:8080
  3. 3.Gradio app connects via OpenAI-compatible streaming API
  4. 4.All inference happens on CPU with addition-only matrix operations

References