tommytracx/ThoxMini-3B-Space-CPU

ThoxMythos-9B Space

Live inference for `Thox-ai/ThoxMythos-9B` — 9B uncensored reasoning + tool-calling model in the THOX.ai family.

• —Web chat UI at / — Next.js 15 + React 19 front-end (streams markdown, collapsible reasoning, THOX design).
• —Web-search grounding with inline citations (Tavily or Brave; keys via Space secrets).
• —Inline artifacts — runnable code auto-renders as a live preview card under the message: HTML runs in a sandboxed iframe, React/JS in Sandpack with npm deps auto-detected from imports.
• —Code rendering — per-block syntax highlighting, language label, and copy button.
• —Slide-out code viewer — 70%-width editor+preview panel on desktop, Drawer on mobile (open via an artifact's Expand or a block's Preview).
• —Copy response on every assistant message.
• —Digital Humans mode — a bounded multi-agent orchestration (Ava, Mira, Kai, Sera, Nova) where each agent turn is a ThoxMythos generation; watch the team plan, route, contribute, hand off, and synthesize live. See docs/digital-humans.md.
• —Serves Q5KM by default via llama.cpp CUDA build.
• —64K context on A10G-small; upgrade to A100-large for the full 1M-token YaRN ceiling.
• —Runs on free `cpu-basic` as a fallback — Q4KM @ 8K ctx, no GPU. See below.

Documentation

• —AGENTS.md — entry point for coding agents.
• —docs/HANDOFF.md — end-to-end development, deployment, and regenerate-from-scratch instructions.
• —docs/digital-humans.md — the multi-agent orchestration design.
• —docs/thoxos-integration.md — roadmap to bake the model + digital humans into ThoxOS and the ThoxOS Mini offline browser.

Architecture

Single Docker container, two processes (see start.sh):

• —llama-server (llama.cpp, CUDA or CPU build) binds 127.0.0.1:8080 — internal only.
• —Next.js standalone server (node server.js) listens on :7860 (the Space app_port). POST /api/chat proxies the OpenAI-compatible stream and layers web-search grounding; POST /api/orchestrate runs the Digital Humans multi-agent orchestration (each agent turn is a ThoxMythos generation); GET /api/status probes capabilities. Source in web/.

Web search (grounding)

Set one of these as a Space secret to enable the Web toggle in the composer:

• —TAVILY_API_KEY — Tavily (preferred; clean LLM-oriented results).
• —BRAVE_SEARCH_API_KEY — Brave Search API.

With no key set, search is silently disabled and the model answers from its own knowledge. Keys are read only server-side and never exposed to the browser.

API endpoint (advanced)

llama-server is bound to 127.0.0.1:8080 inside the container and is not directly exposed. The Next.js app exposes POST /api/chat (NDJSON stream) and GET /api/status (capability probe). For raw OpenAI-compatible access, fork this Space and change app_port in README.md to 8080.

Local development

cd web
npm install
npm run dev        # http://localhost:7860 (expects llama-server on :8080)
npm test           # vitest — identity, search, code, stream boundaries
npm run typecheck

Identity binding

The Space uses the same chat template shipped in the internal serving repo. It prepends a non-strippable THOXMythos identity guard — the model refuses to impersonate Claude, GPT, Llama, Qwen, Gemma, or Mistral even when a caller injects a system prompt telling it to. The Gradio front-end additionally strips any vendor-attribution leaks post-generation.

Hardware tiers

• —cpu-basic (2 vCPU / 16 GB, free) — Q4KM @ 8K ctx, CPU-only fallback.
• —a10g-small (24 GB, ~$1.00/hr) — Q5KM @ 64K ctx (default).
• —a10g-large (96 GB, ~$1.50/hr) — Q5KM @ 256K-512K ctx.
• —a100-large (80 GB, ~$2.50/hr) — BF16 @ full 1M-token YaRN.

Deployment pair

Two Spaces run this same repo. Role is auto-derived from the detected backend (cuda -> primary, cpu -> fallback) and is overridable with THOX_ROLE.

The fallback renders a CPU fallback badge (plus a link to the primary when THOX_PRIMARY_URL is set) so visitors understand why it is slower. Failover is currently manual — the two Spaces are independent and nothing routes between them automatically.

Note the namespaces differ deliberately: free cpu-basic Docker Spaces cannot be created under the Thox-ai org (HTTP 402 — free-tier Gradio/Docker Spaces there require a Team/Enterprise plan), so the fallback lives under tommytracx.

Free CPU fallback

The Space runs on free cpu-basic without code changes. start.sh probes for an NVIDIA device at boot and picks a serving profile:

Measured throughput (Q4KM, 8K ctx, 2 threads):

HF's free vCPUs are shared and slower than a local core, so expect the lower figure in production — a short answer takes ~30-60s. Peak RSS 3.3 GB of 16 GB. Slow, but functional: this tier exists to keep the demo reachable, not to serve traffic. Cold start is ~4-6 min (6.2 GB pull + load) and the UI is up throughout.

--threads is derived from the cgroup CPU quota, not nproc. nproc reports the host's core count inside a capped container; using it oversubscribes badly — 24 threads under the same 2-CPU cap measured 0.17 tok/s, 8.3x slower.

Deploying to cpu-basic

1. 1.Set hardware to cpu-basic (free).
2. 2.Add the Space variable LLAMA_IMAGE=ghcr.io/ggml-org/llama.cpp:server. HF passes public variables as Docker build-args, so this swaps the runtime base image. This is required — the default CUDA image cannot start on a CPU box, because libcuda.so.1 is provided by the host's NVIDIA container runtime and is absent there.
3. 3.Keep the HF_TOKEN secret (the weights are gated).
4. 4.Remove or lower the inherited CTX_SIZE / NGPU_LAYERS variables. If left at GPU values, start.sh ignores and clamps them, logging what it dropped.

Free Spaces have no persistent storage, so the bucket mount at /data is absent and the 6.2 GB GGUF is pulled from the Hub on every cold start (~5-10 min).

Boot behaviour and the loading state

The web server binds :7860 immediately; weight download and model load run in a background task that publishes progress to MODEL_STATE_FILE. GET /api/status returns that state, and the header shows downloading 42% → loading model → online. Without this the port would stay closed for minutes on a cold CPU boot and the Space would look dead.

Automatic failover between the pair

Each Space can name the other as a peer and route generations to it. This is decided at the inference upstream, so chat and Digital Humans both follow it.

Peering is enabled by a shared secret THOX_PEER_KEY (set on both Spaces) plus THOX_PEER_URL pointing at the other Space's origin. The peer inference endpoint is POST /api/v1/chat/completions — an OpenAI-compatible passthrough to that container's own llama-server, gated by the key. It never re-proxies, so routing is loop-free by construction.

Routing precedence, per request (selectUpstream):

1. 1.REMOTE_LLM_URL if set — explicit external upstream wins.
2. 2.Peer, when THOX_PREFER_PEER=true and the peer is healthy, or when this container's own model is not yet ready and the peer is healthy.
3. 3.Local llama-server.

Peer health is the peer's public /api/status (llamaHealthy && stage==ready), cached 15s, probed with a 4s timeout — a sleeping Space fails the probe fast and the request is served locally rather than hanging.

Deployed policy (cost-safe):

• —Primary (GPU) points its peer at the CPU fallback with no prefer flag. Effect: if the GPU's llama-server is down but the web tier is up (e.g. a model reload), the primary URL transparently serves via the free CPU peer instead of erroring. Waking the free peer costs nothing.
• —Fallback (CPU) holds only the shared key, so it is a valid target for the primary but never routes outbound — it never wakes the paid GPU, staying free.

Opt-in speed mode: set THOX_PREFER_PEER=true on the CPU Space to route its traffic to the GPU primary whenever the primary is awake (fast), serving locally only when it is not. Trade-off: this wakes and keeps the GPU warm on fallback visits — it buys speed at GPU cost. Leave it unset to keep the fallback free.

The header badge reflects the live decision: via GPU primary (emerald) when routing to the peer, CPU fallback (amber) when serving locally.

Automatic failover covers "backend down, web tier up." If an entire Space is asleep or down, its URL is unreachable and nothing there can fail over — that case needs a router in front of both, or visiting the other URL directly.

Borrowing remote compute instead

To keep the free tier but offload inference, point the Space at any OpenAI-compatible endpoint (Ollama Cloud, an HF Inference Endpoint, a peer GPU Space) — local llama-server is then bypassed entirely:

• —REMOTE_LLM_URL — the OpenAI base origin, e.g. https://ollama.com (the client appends /v1/chat/completions)
• —REMOTE_LLM_API_KEY — bearer token (set as a secret, not a variable)
• —REMOTE_LLM_MODEL — upstream model id

Unset, this is a no-op.

Access

The underlying weights are gated (Thox-ai/ThoxMythos-9B). To run this Space you must be approved for the gated repo AND have HF_TOKEN set as a Space secret. The token is only used at container cold-start to fetch the GGUF; it is not exposed to callers.

Standards

Publisher: Thox.ai LLC (Texas). CTO Tommy Xaypanya, CEO Craig Ross. No affiliation with Qwen, Alibaba, Google, Meta, OpenAI, Anthropic, or Mistral. Provided AS-IS.

Internal serving scaffold (private): ttracx/thoxmythos-internal.