oddadmix/Arabic-Tokenizer-Comparison
Arabic Tokenizer Comparison — كم رمزاً يكلّفك النص العربي؟
Every model card quotes parameter count and context length. None of them tell you that the same Arabic paragraph costs 50% more context on one tokenizer than another — a tax paid on every prompt, every document and every embedding you ever run.
Paste Arabic — MSA, dialect, or code-switched — and see what nine tokenizers actually spend on it.
The point isn't vocabulary size
Mistral-7B and Emhotob both have ~32K vocabularies. On Modern Standard Arabic, Mistral needs 4.5× more tokens for identical text. The difference isn't how many entries a vocabulary has, it's what those entries are spent on.
Measured on a 1,024-word MSA paragraph:
Emhotob beats vocabularies 8× its size on Arabic — and loses badly on English, which it was never built for. That trade is the whole design.
The token-split view shows why. On the same sentence, Mistral emits one token per Arabic letter, Gemma-4 breaks words into fragments (أ عل نت), and Emhotob keeps them whole.
What to try
- Dialect: Emhotob's lead narrows from 1.52× to ~1.19×. Dialectal orthography is where an MSA-trained vocabulary is weakest.
- Code-switched: Gemma-4 wins. Half the text is English.
- English: Emhotob costs 2.2× what Gemma-4 does. Right tool, wrong job.
About Emhotob
The Arabic-only byte-level BPE behind `50M-2048-Emhotob` and the Nawah family. At hidden size 512 its 32,000 entries cost 16.4M embedding parameters; a 262K vocabulary would cost 134M — more than twice the entire model.
© KAND CA 2026 — PROJECT NAWAH