TigreGotico/arabic-stem-lexicon
Arabic Diacritized-Stem Lexicon An undiacritized Arabic surface form → its most frequent diacritized stem. Standard Arabic writes no short vowels, so anything that has to pronounce Arabic must first put them back. A neural diacritizer does that well on rare words, where inference is the only thing there is. On common words it is the wrong tool: which vowels كتاب carries is not a thing to be inferred, it is a thing to be looked up — and models get exactly these wrong, reading… See the full description on the dataset page: https://huggingface.co/datasets/TigreGotico/arabic-stem-lexicon.
Arabic Diacritized-Stem Lexicon
An undiacritized Arabic surface form → its most frequent diacritized stem.
Standard Arabic writes no short vowels, so anything that has to pronounce Arabic must first put them back. A neural diacritizer does that well on rare words, where inference is the only thing there is. On common words it is the wrong tool: which vowels كتاب carries is not a thing to be inferred, it is a thing to be looked up — and models get exactly these wrong, reading كِتَاب as كَتَاب and مَدْرَسَة as مُدْرَسَة. This is the lookup.
The stem, not the word form
The source corpus is classical and writes the full iʿrāb, so the same word appears as كِتَابٌ / كِتَابَ / كِتَابِ depending on syntax. The case ending is stripped before counting, and what is stored is the stem. That is deliberate: the ending is the part a pausal (spoken) reading drops anyway, and the stem is the part a diacritizer gets wrong. Collapsing the case variants also concentrates the counts, so the evidence per entry is much stronger.
Format
key<TAB>stem<TAB>count<TAB>runner_up, UTF-8, one entry per line.
count and runner_up are the entry's own evidence, kept so a human can audit it. A large gap is a confident entry (كتاب → كِتَاب, 19390 vs 36); a narrow one is a genuine ambiguity the corpus cannot settle (كاتب → كَاتَب, 1220 vs 806 — kaːtab 'he corresponded' against kaːtib 'writer', which only the sentence decides).
Provenance
Mined from `TigreGotico/arabic_diacritized_text` — the diacritized MSA corpus the rawi diacritizer models were trained on. Building the lexicon from the same distribution the model learned means the lookup and the model agree about what Arabic looks like, and the words the model gets wrong are exactly the ones the lexicon is there to catch.
Used by
`arbtok`, which consults it before its neural diacritizer. It is a diacritization lexicon, not a pronunciation lexicon — the stem is fed back through the phonological engine, so a looked-up word still receives emphasis spread, gemination, stress and pausal treatment, and still comes out as Najdi under a Najdi spec and MSA under an MSA one from the same entry.
On WikiPron ara_arab_broad (17,563 words), consulting this lexicon before the model moves arbtok from WER 0.7405 → 0.6780 and PER 0.1842 → 0.1719, past espeak-ng (0.7305 / 0.2069).