CoolFace
Apppublic

akhaliq/Ternary-Bonsai-27B

sourceHugging Faceupdated 2mo agoView on Hugging Face
3likes
App README

Ternary-Bonsai-27B · Hugging Face Space

A text-generation chat app for `prism-ml/Ternary-Bonsai-27B-gguf`, served through Hugging Face Inference Providers (the OpenAI-compatible router at https://router.huggingface.co/v1) and built with `gradio.Server`.

No local GPU or model weights are needed — generation runs on the provider and tokens are streamed back over HTTP. The model is pin-routed to the Together provider as prism-ml/Ternary-Bonsai-27B-gguf:together.

The backend is a gradio.Server FastAPI app that exposes a queued, concurrency-controlled, streaming chat endpoint via @app.api(name="chat") and serves a hand-written Hugging-Face-branded chat UI from @app.get("/"). The frontend talks to the backend through the Gradio JS client (@gradio/client), so requests go through Gradio's queue and tokens stream to the browser over SSE.

Files

  • —app.py — gradio.Server backend: an openai.OpenAI client pointed at the HF router, a streaming /chat API (text in, text out), and index.html served at / as HTMLResponse.
  • —index.html — custom HTML/CSS/JS chat UI with the 🤗 logo, streaming response rendering, and generation controls (max tokens, temperature, top-p).
  • —requirements.txt — dependencies (gradio, openai).

Configuration

Set the Hugging Face token as the HF_TOKEN environment variable. On a Space, add it under Settings → Repository secrets → New secret named HF_TOKEN (read scope is sufficient). Without it, requests to the router will fail to authenticate.

Run locally

bash
pip install -r requirements.txt
export HF_TOKEN=hf_xxx     # your HF access token
python app.py
# open http://localhost:7860

No GPU hardware is required on the Space either — it runs on the default CPU hardware since inference is delegated to the provider.

Notes

  • —Text in, text out. Type a message and the model streams a reply back.
  • —Generation streams chunk-by-chunk from the provider; the frontend renders each delta incrementally.

Config reference: https://huggingface.co/docs/hub/spaces-config-reference Inference Providers: https://huggingface.co/docs/inference-providers