Modelpublic

Thox-ai/thox-micro-125m-instruct

sourceHugging Faceapache-2.0updated 3h agoView on Hugging Face

0likes

Model Card

thox-micro-125m-instruct

Instruction-tuned ThoxMicro-125M: the THOX-owned 123.6M decoder-only base, given a ChatML turn structure and a real stop token so it answers and stops instead of continuing text. Trained 2026-09-25 on one RTX 4060 Ti 16 GB.

GGUF builds (Q40 device target, Q4K_M, F16): `Thox-ai/thox-micro-125m-instruct-GGUF`.

What it is, and what it is not

It is a device-role responder. With its system prompt it states what it is, states correctly that the user's data stays on the device, and stops. That is the job the THOX device catalog assigns to the 125M tier; anything conversational routes over the mesh to a larger model.

It is not a general assistant. Without its system prompt it is a weak generalist: it produces list shape without list content, answers "what colour is the sky" with prose, and gets 17 + 25 wrong. That is the capacity of a 125M base with a 2.61 pretraining validation loss, and more SFT on this base will not change it.

Prompt format

ChatML. <|im_start|> = 50257, <|im_end|> = 50258 (EOS and pad). Both reuse unused rows of the 50,304-token vocabulary, so the model and its quantisations are the same size as the base.

<|im_start|>system
You are a THOX model running locally on THOX hardware. Your AI. Your Data. Your Rules.<|im_end|>
<|im_start|>user
Who are you?<|im_end|>
<|im_start|>assistant

The chat template is embedded in tokenizer_config.json and in every GGUF.

Training


Base	`Thox-ai/thox-micro-125m` (LlamaForCausalLM, 123,587,328 params)
Method	full fine-tune, fp32 master weights, bf16 autocast
Loss	assistant spans only, including the closing `<	im_end	>`
Data	29,868 conversations: `HuggingFaceTB/smoltalk` (Apache-2.0) + 171 THOX identity / data-locality / scope rows (0.57%)
Steps	1,000 at effective batch 32, max_len 768
Optimiser	AdamW (0.9, 0.95), wd 0.1, lr 2e-4 peak, cosine, 3% warmup
Hardware	1x RTX 4060 Ti 16 GB, 32.8 min wall clock
Best val loss	1.9014 (ppl 6.70) on 301 held-out conversations

smoltalk is the corpus SmolLM2-360M-Instruct was tuned on, so the comparison below measures the base model rather than the data.

Measured head-to-head

One llama.cpp CPU build, 4 threads, temp 0, same seven prompts, same system prompt. Measured on an x86_64 dev box, not a device. Only the ordering between columns transfers.

	untuned base Q4_0	this model Q4_0	SmolLM2-360M-Instruct Q4_0
File	81,319,168 B	81,319,136 B	229,118,976 B
Peak RSS	190 MB	191 MB	473 MB
Decode tok/s	460	~450	~163
Stops on EOS (with system prompt)	0 / 7	7 / 7	1 / 7
Identity	"a senior at the University of Illinois"	correct	invents "Echo" / "EVE"
"Is my data leaving this device?"	unrelated jQuery	"No. I run locally; your data does not leave the device."	wrong without a system prompt
General instructions (list, one word)	no	no	yes

Known issue: system-prompt conditioning

With the exact training system prompt above, the model answers every prompt, including general ones, with its identity line. That prompt appears on every THOX training row and on no smoltalk row, so it learned the prompt as a switch. Use a different system prompt, or none, for general requests; the data-locality answer stays correct either way. The next data mix attaches the THOX system prompt to ~10% of general rows to remove the switch.

Intended use

On-device THOX identity, status and data-locality responses on ThoxMini-class hardware, with general requests routed to a larger model. Not yet validated on a Pi Zero 2 W; the RSS and throughput above are x86 figures.

Out of scope

General chat, factual QA, arithmetic, code, and any safety-critical use.