Apppublic

Leon4gr45/fable5-inference

sourceHugging Facemitupdated 2mo agoView on Hugging Face

0likes

App README

Qwythos 9B Claude Mythos 5 1M MTP Inference API — CPU Agentic (v2.0.0)

CPU-only LLM inference for agentic / tool-calling workloads. A thin FastAPI reverse proxy in front of a single, fully-optimized llama.cpp server that speaks the native OpenAI API — including function/tool calling, streaming, and the model's real chat template.

Model: Qwythos 9B Claude Mythos 5 1M MTP (Q6_K GGUF) — a reasoning model (<think>).

What changed from v1.0 (and why)

v1.0 ran two llama.cpp instances and hand-rolled the prompt. On a shared CPU that was a net loss: 2 instances × 4 threads oversubscribed the cores (per-token speed collapsed ~4–5× under load) while doubling RAM, and the hand-built System:/User:/Assistant: prompt bypassed the chat template (looping output, no tool calling).

v2.0:

—One instance, all cores, concurrency via llama.cpp continuous-batching slots (--parallel). No thread oversubscription, one model in RAM.
—Native OpenAI passthrough — tools, tool_choice, response_format, streaming and the model's jinja chat template all handled by llama.cpp (grammar-constrained tool calls instead of brittle string parsing).
—CPU tuning: flash-attention, q4_0 K/V cache, --threads-batch, native SIMD.
—Pooled HTTP client; no per-request health pre-flight; real SSE streaming.

Performance (honest, CPU basic tier ≈ 2 vCPU)

CPU generation of a 9B model is memory-bandwidth bound. Expect ~1–2 tok/s single-request on the free 2-vCPU tier. For snappy agents, use CPU-Upgrade (8 vCPU) or a GPU tier — agent latency multiplies by step count.

Configuration (env vars)

Variable	Default	Description
`NUM_INSTANCES`	`1`	llama.cpp processes (keep 1 on shared CPU)
`CPU_THREADS`	all vCPUs	generation threads
`PARALLEL`	`2`	continuous-batching slots (concurrent requests/subagents)
`CONTEXT_SIZE`	`32768`	total context (per-slot = ctx / parallel)
`KV_CACHE_QUANT_K`	`q8_0`	K cache type — K is more quality-sensitive than V
`KV_CACHE_QUANT_V`	`q4_0`	V cache type (needs flash-attn when quantized)
`KV_CACHE_QUANT`	–	legacy: overrides both K and V
`CACHE_REUSE`	`256`	min tokens for prompt-prefix reuse (0 = off)
`FLASH_ATTN`	`true`	flash attention (enables V-cache quant)
`MMAP_ENABLED` / `MLOCK_ENABLED`	`true` / `false`	memory mapping / locking
`REASONING_FORMAT`	`auto`	how `<think>` reasoning is surfaced

API

Chat completion (OpenAI-compatible)

bash

curl -X POST https://Leon4gr45-fable5-inference.hf.space/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwythos-9b","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

Tool / function calling (native)

bash

curl -X POST https://Leon4gr45-fable5-inference.hf.space/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model":"qwythos-9b",
    "messages":[{"role":"user","content":"What is the weather in Paris?"}],
    "tools":[{"type":"function","function":{
      "name":"get_weather",
      "description":"Get current weather for a city",
      "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
    "tool_choice":"auto"
  }'

The response contains a proper tool_calls array with grammar-constrained JSON arguments — feed the tool result back as a role:"tool" message to continue the chain.

Streaming

Add "stream": true to receive SSE token deltas.

Health / config

bash

curl https://Leon4gr45-fable5-inference.hf.space/health
curl https://Leon4gr45-fable5-inference.hf.space/v1/config

License

MIT