oddadmix/Arabic-Tokenizer-Comparison

1---2title: Arabic Tokenizer Comparison3emoji: 🔤4colorFrom: green5colorTo: indigo6sdk: gradio7sdk_version: 6.22.08app_file: app.py9pinned: false10license: apache-2.011short_description: "How many tokens does your Arabic actually cost?"12---13 14# Arabic Tokenizer Comparison — كم رمزاً يكلّفك النص العربي؟15 16Every model card quotes parameter count and context length. None of them tell17you that the same Arabic paragraph costs **50% more context** on one tokenizer18than another — a tax paid on every prompt, every document and every embedding19you ever run.20 21Paste Arabic — MSA, dialect, or code-switched — and see what nine tokenizers22actually spend on it.23 24## The point isn't vocabulary size25 26**Mistral-7B and Emhotob both have ~32K vocabularies.** On Modern Standard27Arabic, Mistral needs **4.5× more tokens** for identical text. The difference28isn't how many entries a vocabulary has, it's what those entries are spent on.29 30Measured on a 1,024-word MSA paragraph:31 32| tokenizer | vocab | tokens/word | vs Emhotob |33|---|---|---|---|34| **Emhotob 32K** (Arabic-only) | 32,000 | **1.391** | 1.00× |35| Qwen3.6 | 248,044 | 1.885 | 1.36× |36| Gemma-4 | 262,144 | 2.109 | 1.52× |37 38Emhotob beats vocabularies **8× its size** on Arabic — and loses badly on39English, which it was never built for. That trade is the whole design.40 41The token-split view shows why. On the same sentence, Mistral emits **one token42per Arabic letter**, Gemma-4 breaks words into fragments (`أ` `عل` `نت`), and43Emhotob keeps them whole.44 45## What to try46 47- **Dialect**: Emhotob's lead narrows from 1.52× to ~1.19×. Dialectal48  orthography is where an MSA-trained vocabulary is weakest.49- **Code-switched**: Gemma-4 *wins*. Half the text is English.50- **English**: Emhotob costs 2.2× what Gemma-4 does. Right tool, wrong job.51 52## About Emhotob53 54The Arabic-only byte-level BPE behind55[`50M-2048-Emhotob`](https://huggingface.co/oddadmix/50M-2048-Emhotob) and the56[Nawah](https://huggingface.co/oddadmix/Nawah-50M-RAG-Support-2K) family. At57hidden size 512 its 32,000 entries cost 16.4M embedding parameters; a 262K58vocabulary would cost 134M — more than twice the entire model.59 60© KAND CA 2026 — PROJECT NAWAH61