glassbox/gpt-alpha-bg-14m-onnx

gpt-alpha-bg-14m-onnx

A 13.77M-parameter Bulgarian language model, trained from scratch, exported to ONNX and quantized to int8, 17.3 MB. Small enough to ship as a static asset and run in a browser tab with onnxruntime-web, which is where it already runs: restorer-interp-lab.iksnerd.workers.dev/live-lm.

What it is for

• —Generation of stylistically Bulgarian text, in the register you prompt toward.
• —Perplexity scoring, ranking candidate sentences by how natural they are, offline and client-side. Useful as a fluency gate over OCR, ASR or MT output.

It generates fluent Bulgarian and it does not know things:

Столицата на България е  ->  Столицата на България е най-добрата част от населението на България.

Fluent, grammatical, and wrong. That is what 13.77M parameters buys, and the limits section measures it rather than hedging about it. If fluency judging is the actual goal, the 91.26M sibling is measurably better at it (92.3% paired preference on real error/correction pairs, a statistical tie with a frontier model). This model is the one that fits in a browser tab.

Usage

Python, with onnxruntime and tokenizers:

import numpy as np, onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("bg15m.tokenizer.json")
sess = ort.InferenceSession("bg15m.int8.onnx")

ids = tok.encode("Столицата на България е").ids
for _ in range(12):                                   # greedy continuation
    logits = sess.run(None, {"idx": np.array([ids[-256:]], dtype=np.int64)})[0]
    ids.append(int(logits[0, -1].argmax()))
print(tok.decode(ids))

The graph is one stateless function:

idx: int64 [batch, seq]  ->  logits: float32 [batch, seq, 8192]

Opset 17, both axes dynamic, no KV cache in the graph, so the same call serves prompt ingestion and one-token-at-a-time decode. At a 256-token context a full recompute per token is fast enough in a browser, and it keeps the artifact a single file with no session state to manage.

In the browser it is the same graph under onnxruntime-web, with the byte-level BPE ported to TypeScript. That port is pinned: the tokenizer is fixed id-for-id against the Python implementation on 512 real sentences, and the runner against reference logits and perplexities.

One backend caveat, found the hard way. ONNX Runtime's wasm and CPU int8 kernels disagree by up to 0.12 on a logit. Across 72 greedy steps that flipped exactly one argmax, at a 0.028 margin, and one flip redirects the whole continuation. Identical greedy output across backends is not something this model, or any int8 model, guarantees. If you need reproducibility, condition each step on a fixed prefix and compare distributions rather than sampled strings.

Results

The data diet, not the parameter count

It saw its training data exactly once: 18,300 iterations over a cleaned mixed Bulgarian corpus (books, news, wiki, forum), ~300M tokens, no example twice. On a shared external held-out set it reaches 2.0620 bits per character, beating both 29M models trained on a narrower fiction-only diet at under half their size:

Single seed per configuration, so read this as a strong result about data diet rather than a measured margin. Bits per character rather than validation loss, because these models do not share a tokenizer and loss across tokenizers is not comparable.

The quantization cost, measured

Both graphs are verified against the PyTorch original on the same held-out text:

<!-- BEGIN generated: card-verification (uv run python datatools/gencardverification.py) --> | | size | max abs logit difference vs PyTorch | top-1 agreement | held-out bpc | |---|---|---|---|---| | bg15m.onnx (fp32) | 68.0 MB | 1.7e-05 | 1.00 | 2.0620 | | bg15m.int8.onnx (dynamic int8) | 17.3 MB | 0.55 | 1.00 | 2.0638 |

int8 costs +0.0018 bpc, measured on the same held-out slice in the same invocation as fp32 — scoring the two separately is how a moved --max-chars default once read as a headline failing to reproduce. <!-- END generated: card-verification -->

For scale: at this model size, switching the training corpus from fiction-only to a mixed diet was worth 0.086 bpc on the same metric, so quantizing costs about a fiftieth of what the data choice bought. Quantization-aware training was on the roadmap and got cut on the strength of this measurement: at 14M there is nothing for it to recover.

Note the two columns that disagree. int8's worst logit moves by 0.55, yet the argmax never changed on the probe: individual logits shift, the ranking mostly does not. Do not read "top-1 agreement 1.00" as determinism, per the backend caveat above.

Limits

• —At chance on knowledge benchmarks. Zero-shot on EXAMS (Bulgarian school exams, 1472 items, 25.0% chance) it scores 25.48%, inside the noise. The 91.26M sibling reaches 27.11% and clears an untrained network of its own shape by 5.0 points (z = 3.1); this model clears neither chance nor the control. On Belebele the untrained control matches or beats every trained model, so that benchmark measures nothing about models at this scale and any above-chance number on it is an artifact of the scoring procedure.
• —No factual reliability, no instruction following, no long-range coherence beyond the 256-token context. It is a stylistic language model.
• —Not a script or gibberish detector. Trained on web text, so romanized Bulgarian can outscore natural Cyrillic. It compares Bulgarian candidates against each other, nothing more.
• —Bulgarian only, one corpus, single seed.

How it was built

384 embedding dim, 6 heads, 6 layers, block size 256, vocab 8192. RoPE, RMSNorm, SwiGLU, weight tying, and highway-gated residuals (a gate on the residual stream, validated against the plain stack in a multi-seed ablation before being adopted). Trained on a cloud GPU with AdamW, 200 warmup steps then cosine decay from lr 3e-4, batch 64, dropout 0.2. 13,772,172 parameters.

Files

Part of a family

<!-- BEGIN generated: family (uv run python datatools/genpublishingdoc.py) --> Six models from one project on small models doing narrow tasks, these among them:

• —`gpt-alpha-bg-91m` - the 91M flagship: a fluency judge that ties a frontier model on Bulgarian error/correction pairs
• —`gpt-alpha-bg-restorer` - shlyokavitsa to Cyrillic, character by character, running in your browser
• —`gpt-alpha-bg-14m-onnx` - you are here

The other three appear under huggingface.co/glassbox as each ships. <!-- END generated: family -->

Findings, the interpretability lab and the full model roster: restorer-interp-lab.iksnerd.workers.dev.

License

MIT.