ahmedsamirtarjama/Tashkeel-v3

Tashkeel-v3: Arabic Diacritization (MARBERTv2 + BiLSTM + Linear-Chain CRF)

Tashkeel-v3 adds full diacritics (تشكيل) to Arabic text: Classical and Modern Standard Arabic, including case endings. It is a character tagger: it predicts one diacritic class per input character. So it never adds, drops or changes letters, digits or punctuation (0% hallucination by construction), and it's fast enough for batch processing.

Results

All numbers use the official SadeedDiac-25 evaluator (ArabicDiacritizationEvaluator, gt_missing_diacritic_is_error=False). Total = including case endings; Morph = excluding the last letter. Every sentence is scored (no hallucinations).

These scores include the built-in إ post-processing step (see below).

Throughput (one NVIDIA L40S, diacritize(..., batch_size=64), including Python preprocessing): ~34 sentences/s on SadeedDiac-25 (long, multi-clause sentences) and ~105 sentences/s on CATT (shorter news sentences).

<details> <summary>Other systems (third-party figures, not re-evaluated by us)</summary>

Generative LLMs are scored only on the sentences where their output kept the input letters intact. A strong LLM remains more accurate on SadeedDiac-25. Tashkeel-v3 is far cheaper and faster, and it never alters the input text. </details>

Comparison with the models in the CATT paper

The CATT paper (Alasmary et al., 2024) compares 13 systems on its benchmark. We ran Tashkeel-v3 on the paper's released input file and scored it with the authors' own evaluation script (compute_der.py from abjadai/catt), next to the model outputs the authors released. We checked that the script reproduces the paper's published numbers exactly (e.g. CATT ED: 8.624 / 34.191).

This protocol scores every letter, including letters the reference deliberately leaves unmarked. That's why all numbers here are higher than the CATT row in the Results table above, which uses the SadeedDiac-25 evaluator. CE = with case endings.

Tashkeel-v3 has the lowest error with case endings (DER and WER) and the lowest WER without case endings. Without case endings, its DER (7.40) is behind the two CATT models (6.99 / 7.09). Part of that gap is convention: the CATT references often leave the letter before a long vowel unmarked and mark the wasl alef, while Tashkeel-v3 diacritizes fully. † The CATT script flags sentences whose output text changed (GPT-4: 11, Command R+: 77, Sakhr: 53, Mishkal: 1); Tashkeel-v3: 0. Our training data was checked against the CATT benchmark for 10-word overlap (see Training).

Usage

pip install torch transformers

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "ahmedsamirtarjama/Tashkeel-v3"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device).eval()

print(model.diacritize("ذهب الطالب المجتهد إلى المدرسة صباحا", tokenizer=tokenizer, device=device))

texts = ["قل إن صلاتي ونسكي ومحياي ومماتي لله رب العالمين", "سطا لصوص على منزل وزيرة الدفاع التشيلية"]
print(model.diacritize(texts, tokenizer=tokenizer, device=device, batch_size=64))

Input should be undiacritized Arabic text. Characters that aren't Arabic letters pass through unchanged.

Model

• —Encoder: UBC-NLP/MARBERTv2 (12 layers, 768-d). Its token vectors are mapped onto characters.
• —Character features: character, position-in-word and clitic-segment (proclitic / stem / enclitic) embeddings.
• —Sequence layers: 2-layer BiLSTM with a residual projection. Two heads: one for word-internal diacritics, one for case endings on word-final letters.
• —Decoding: linear-chain CRF over 15 diacritic classes (none, fatha, damma, kasra, sukun, the three tanweens, shadda, and shadda combinations). Globally optimal decoding with Viterbi.
• —Post-processing: a bare hamza-below alef (إ) always receives its kasra. This is linguistically always correct, and no other character is modified.
• —Size: ~171M parameters (fp32 weights).

Training

The model was trained in stages, starting from a model fine-tuned on Misraj/Sadeed_Tashkeela. Each stage was trained 1 epoch per seed. The final checkpoint of each of 2 seeds was kept and the two were weight-averaged. This rule was fixed in advance, and no checkpoint was ever selected on the benchmark test sets.

1. 1.Continued training on `Misraj/Sadeed_Tashkeela` (the in-domain training set).
2. 2.Complete classical text: new sentences from the original Tashkeela corpus (Zerrouki, v0.3) and from other public diacritized Arabic corpora, kept only if they were
3. 3.fully diacritized (every word-final letter marked, ≤2% unmarked interior letters),
4. 4.free of misplaced marks, and
5. 5.not already used in earlier training.

Sukun before hamzat al-wasl was normalized to the connecting-vowel convention (مِنَ الـ، عَنِ الـ).

1. 1.Modern Standard Arabic, which drove most of the CATT gain:
2. 2.self-training on ~154k news sentences from arbml/SANAD, labeled by the model and kept only where an independent earlier model agreed on ≥97% of words,
3. 3.~29k Wikipedia chunks whose own diacritics closely matched the model's.

Decontamination: every added sentence was checked against SadeedDiac-25 and the CATT benchmark, and any sentence sharing a 10-word sequence with either was removed.

Limitations

• —Rare and foreign words are the main remaining error source. Words seen 50 times or fewer in training cause about 42% (SadeedDiac-25) and 49% (CATT) of the errors. That includes transliterated names and brands (e.g. فيسبوك، رويترز) and uncommon classical vocabulary.
• —Case endings still depend on syntax the model sometimes misreads, for example the object vs. the subject in long clauses.
• —Conventions: the model writes the connecting vowel before hamzat al-wasl (مِنَ الْ). References that use sukun there (مِنْ الْ) count these words as errors. The CATT references also mark the wasl alef itself (اِحْتَفَلَ), which this model doesn't.
• —Scoring note: the official evaluator drops a shadda written after its vowel in a reference, so a few correct predictions count as errors on SadeedDiac-25 (≈0.05 DER).
• —Dialectal Arabic and poetry-specific diacritization weren't targets of this model.

License

Apache-2.0 for the model weights and code. The training data comes from publicly released corpora; see their respective licenses.