Apppublic

Shaankar39/vaaani-flagship

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes

App README

Vaaani Flagship — CPU inference (free tier)

Serves the Vaaani curriculum tutor (Qwen2.5-3B base GGUF + the Root-Bridge / sound-spiral LoRA) entirely on CPU via llama-cpp-python. No GPU. The CPU-only design is the moat — and it is exactly what lets this run on Hugging Face's free 2-vCPU / 16-GB Space.

Endpoints

Method	Path	Purpose
GET	`/`	health (returns instantly; model loads lazily on first chat)
POST	`/chat`	`{ "messages": [...] }` → `{ "reply": "..." }`
POST	`/chat/stream`	Server-Sent Events, token-by-token (use this in the UI)
POST	`/v1/chat/completions`	OpenAI-compatible drop-in for the gateway

Configuration (Space → Settings → Variables, not Secrets)

Var	Default	Notes
`VAAANI_MODEL_REPO`	`Shaankar39/vaaani-flagship-gguf`	HF model repo holding the GGUF files
`VAAANI_BASE_FILE`	`vaaani-base-q4_k_m.gguf`	Q4KM for free CPU; swap to Q8 on a paid Space
`VAAANI_LORA_FILE`	`vaaani-flagship-lora-f16.gguf`	the curriculum adapter
`N_THREADS`	`2`	= the free tier's 2 vCPUs
`N_CTX`	`2048`	curriculum prompts are short; keep it small for speed
`MAX_TOKENS`	`512`	lesson turns are brief

How the GGUF gets here

The Space does not bundle the 2–3 GB model. It pulls it at first request from the HF model repo above (hf_hub_download). So: after training, upload the two GGUF files to that model repo once, and this Space serves them. Swapping quant = change one Variable, no rebuild.

Free-tier reality (honest)

—Accuracy: identical to local — same weights; Q4KM ≈ Q8 for this scripted task.
—Speed: a few tok/s on 2 vCPU (~10–25 s/turn). Streaming hides most of it.
—Sleep: free Spaces idle-sleep after 48 h → ~30–60 s cold start on the next hit.
—Scale: one request at a time. For concurrent classrooms, upgrade to a paid CPU Space (never a GPU) — funded from pilot revenue, not upfront.