Thox-ai/ThoxAir-Accel
⚠️ Org runtime paused (quota/hardware). Temporary: use personal tommytracx mirrors when available; org Team billing required to restore.ThoxAir Accel — offload endpoint for the RV1103
Serves `Thox-ai/ThoxAir-16M-role` so the ThoxAir can hand off long turns.
Why CPU, not a paid GPU
Measured on this Space, on the free cpu-basic tier it actually runs on:
cpu-basic is the correct tier: a 5.5 MB model is not GPU work, and even the low end of that range is a real margin over the device. The spread is genuine — this is a shared free tier, and two measurement runs an hour apart gave 148–182 and 191–258 tok/s. Quote the range, not a single number.
⚠️ An earlier version of this README claimed ~1600 tok/s and a 146x speedup. That was wrong — it did not come from this hardware. Sustained throughput here is ~150–260 tok/s. The endpoint is still clearly worth offloading to, but at ~14–23x, not 146x.
The first call after a cold start is not a speedup at all (7–16 tok/s, at or below the device), because the model load happens inside the request. The Space sleeps when idle, so a real ThoxAir offload pays that cost on its first request after any quiet period. Route accordingly — or keep it warm.
The offload rule is about LENGTH, not content
ThoxAir is bandwidth-bound: ~5.5 MB of weight traffic per token on a Cortex-A7. Short turns finish on-device in under a second, and offloading them would add latency — a network round trip costs more than the tokens saved. Long turns are where the device falls behind linearly.
expected output <= ~24 tok -> stay on RV1103
expected output > ~24 tok -> offload hereThe C6 (Thox-ai/ThoxMesh-Head-C6) already triages events, so it makes the call and routes via route.thox.ai.
Endpoints
Responses carry a non-standard thox_offload.tok_per_s so the router can verify offload actually beat the ~11 tok/s it would have got locally.
Serving quant
Serves the Q8_0, not the TQ10 device artifact: `llama-cpp-python`'s prebuilt wheels carry no ternary kernels, and TQ10 is what ships to the board rather than what serves here. Same weights, different container.