Modelpublic

Tokenade/potion-multilingual-128M-i8-tokenade

sourceHugging Facemitupdated 2mo agoView on Hugging Face

0likes1.2kdownloads

Model Card

potion-multilingual-128M-i8-tokenade

An int8 quantisation of `minishlab/potion-multilingual-128M`, plus precomputed tokenizer artifacts for low-memory loading.

All credit for the model itself goes to the Minish Lab — this repo adds no training, no distillation and no new data. It only changes how the same weights are stored and loaded.

Why

model2vec inference is tokenize → look up rows → mean-pool: no context, no forward pass. Loading it should not cost like a transformer, but in practice a stock load decodes the whole weight matrix to f32 and inflates tokenizer.json into an in-memory trie — together about 1 GB of RSS and ~930 ms before the first vector. That is fine for a long-lived server and prohibitive for a short-lived CLI process.

These artifacts are the same model arranged so it can be memory-mapped instead:

	stock load	these artifacts
RSS	~1056 MB	~4.5 MB
first embedding	~930 ms	~1 ms
per embedding	~0.03 ms	~0.11 ms

The per-embedding cost regresses (no pre-decoded matrix, no cache); a process needs roughly 12 700 embeddings in one run before that outweighs the load-time saving.

Files

file	what it is
`model.safetensors`	the embedding matrix, `I8`, `[500353, 256]`
`vocab.fst`	the Unigram vocab as an FST: token bytes → id
`vocab.fst.scores`	`f64` unigram log-probs, one per id, in id order
`vocab.fst.norm`	the normalizer / pre-tokenizer spec and decoding metadata
`tokenizer.json`	upstream's, unchanged — kept so the FST can be re-derived and verified
`config.json`	upstream's, unchanged

Quantisation

Symmetric, one global scale (max|W| / 127). The scale is deliberately not stored: mean-pooling is linear and model2vec L2-normalises its output, so a single global factor cancels exactly. Rows can therefore be read straight from disk as raw i8.

This is measured, not assumed. Against model2vec-rs reading the f32 original, over 100 133 text segments extracted from real PDF / DOCX / XLSX / EPUB / ODT / ODS / ODP / PPTX files:

—token ids: 0 divergences
—embeddings: worst cosine 0.999994

Retrieval quality on our internal evaluation is unchanged within noise (0.570/0.413 vs 0.571/0.414 for f32).

Caveats

—[UNK] handling follows model2vec-rs exactly, including that it does not drop [UNK] rows for a Unigram tokenizer (it looks for a string unk_token field, which Unigram does not emit). If you consume these artifacts with your own loader and drop [UNK] instead, you will get different vectors on unk-heavy text.
—vocab.fst maps duplicate tokens to their last id, matching how tokenizers builds its token_to_ids map.
—The FST/score/norm layout is ours, not a model2vec standard. Anything reading only model.safetensors + tokenizer.json (including model2vec itself) works with this repo unmodified — the i8 tensor is a standard safetensors dtype.

License

MIT, inherited from the upstream model.