Apppublic

Thox-ai/thox-mesh-node

sourceHugging Faceapache-2.0updated 2d agoView on Hugging Face

0likes

App README

⚠️ Org runtime paused (quota/hardware). Temporary: use personal tommytracx mirrors when available; org Team billing required to restore.

THOX Mesh Node — HF CPU Space

A free-tier CPU node in the THOX mesh. It serves an OpenAI-compatible endpoint from a THOX GGUF and self-registers so ThoxRoute can route traffic to it.

Endpoints

Path	Purpose
`POST /v1/chat/completions`	OpenAI-compatible chat completion
`GET /v1/models`	Advertised model
`GET /health`	Liveness, mesh registration state, live telemetry

Quick check

bash

curl -s https://<space-host>/health | jq .

curl -s https://<space-host>/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"Say hello from the THOX mesh."}],"max_tokens":48}'

Configuration

Set these as Space Variables, except HF_TOKEN which must be a Secret.

Variable	Default	Meaning
`THOX_MODEL_REPO`	`Thox-ai/ThoxMini-3B`	Hub repo holding the GGUF
`THOX_MODEL_FILE`	`thoxmini-3b-Q4_K_M.gguf`	GGUF filename
`THOX_MODEL_ID`	`thoxmini-3b`	Model id advertised to the mesh
`THOX_ALIAS`	`thoxmini-3b-space`	Endpoint alias, unique per device
`THOX_CAPABILITIES`	`chat,completion`	Capability flags ThoxRoute matches on
`THOX_CONTROLLER_URL`	(unset)	Mesh controller to register with
`THOX_MESH_DEVICE_ID`	(unset)	Paired MeshStack device UUID
`THOX_MESH_DEVICE_KEY`	(unset)	Secret. Base64 Ed25519 seed
`THOX_N_GPU_LAYERS`	`auto`	`auto` \	`0` (force CPU) \	`-1` (all) \	an integer
`HF_TOKEN`	(unset)	Secret. Required — the model repo is private

CPU and GPU paths

The same agent runs on both. Nothing needs reconfiguring to move between them — THOX_N_GPU_LAYERS=auto resolves per machine.

	CPU (this Space, free Colab)	GPU (Colab T4/L4/A100)
Model	ThoxMini-3B Q4KM (~2 GB)	ThoxMythos-9B Q4KM (~6.2 GB)
Offload	`n_gpu_layers=0`	`n_gpu_layers=-1` (all, clamped to VRAM)
llama.cpp build	CPU wheel/build	rebuilt with `-DGGML_CUDA=ON`
Throughput	~1.5 tok/s	roughly an order of magnitude faster

A CPU-only environment is fully supported, not a degraded one. GPU detection goes through thoxmesh_node.gpu, which treats a missing nvidia-smi, a broken driver and a hung probe all as "no GPU" and never raises. If a CUDA build is asked to offload and cannot, the node retries CPU-only rather than refusing to serve — a slower mesh member beats an absent one.

Check any machine before starting:

bash

python -m thoxmesh_node doctor

It reports Python, the crypto/web stack, llama-cpp-python, HF_TOKEN, GPU state, cloudflared and the resolved config. Exit 0 means the node can serve.

Why this Space registers with the controller, not MeshStack v2 directly

meshstack-device-v2 validates every base_url against isPrivateHost and accepts only RFC1918, loopback, link-local, CGNAT and .local addresses. An *.hf.space origin is public and is refused by that rule — deliberately, as an anti-SSRF control on the mesh.

The node detects that specific refusal and falls back to the controller transport rather than retrying a request that can never succeed. See ARCHITECTURE.md for the contract and the proposed resolution.

Cost

Runs on the free cpu-basic tier and must stay there. ThoxMini-3B Q4KM is ~2 GB and fits the 16 GB / 2 vCPU allocation; nothing here requests a GPU.

Expect roughly 1.5 tokens/second. That is the honest free-CPU number, and it is why THOX_MAX_TOKENS_CAP defaults to 256 — the node reports its real latency as telemetry, so ThoxRoute ranks it behind faster nodes rather than being surprised by it.

ThoxLLM-327M-v2 was tried first because it is far smaller. It loads and serves correctly but its output is degenerate ("The capital of France is" → "]]]]]]]]]]"): a 327M base model with no instruction tuning and no chat template. Mechanically fine, useless for a demo.

thox-micro-125m is not published on the Hub at all. It was the package default until v1.1.0, which meant anything relying on defaults pointed at nothing.