Thox-ai/thox-micro-125m-instruct-GGUF

thox-micro-125m-instruct-GGUF

GGUF builds of `Thox-ai/thox-micro-125m-instruct`. Read that card for what the model is and is not: a THOX device-role responder, not a general assistant.

Q40 is the device target for the same reason as the base: the FFN dim (2048) divides cleanly into Q40 blocks and the format is scalar-safe on ARMv6. The files are the same size as the untuned base because the ChatML tokens reuse unused vocabulary rows.

Embedded metadata: tokenizer.ggml.eos_token_id = 50258 (<|im_end|>), padding_token_id = 50258, tokenizer.chat_template (ChatML). Both <|endoftext|> and <|im_end|> are end-of-generation tokens.

Run

llama-completion -m thox-micro-125m-instruct.Q4_0.gguf -no-cnv --temp 0 -n 96 -p $'<|im_start|>system\nYou are a THOX model running locally on THOX hardware. Your AI. Your Data. Your Rules.<|im_end|>\n<|im_start|>user\nIs my data leaving this device?<|im_end|>\n<|im_start|>assistant\n'

Or with the embedded template: llama-server -m thox-micro-125m-instruct.Q4_0.gguf --jinja.

Measured (x86_64 dev box, 4 threads; not a device figure)

Q40: 191 MB peak RSS, ~450 tok/s decode, 7 of 7 clean stops with its system prompt. SmolLM2-360M-Instruct Q40 on the same harness: 473 MB peak RSS, ~163 tok/s, 1 of 7. Not yet validated on a Pi Zero 2 W.

License

Apache-2.0. Copyright (c) 2026 THOX.ai LLC.