yuujiElfahkrany/tashkeel

Arabic Tashkeel Dataset — Al-Maktaba Al-Shamela

A large-scale Arabic diacritization (tashkeel) dataset derived from Al-Maktaba Al-Shamela (المكتبة الشاملة), a comprehensive digital library of classical Islamic texts. The dataset pairs undiacritized Arabic sentences with their fully diacritized equivalents, enabling training and evaluation of automatic tashkeel systems.

Dataset Summary

• —Languages: Arabic (classical / Islamic corpus)
• —Source: Al-Maktaba Al-Shamela HTML exports
• —Format: JSONL (one record per line)
• —Uncompressed size: ~1.29 GB

Dataset Structure

Data Fields

Example

{
  "raw": "وقد أخرج مالك في الموطأ هذه الزيادة عن الزهري أيضا،",
  "tashkeel": "وَقَدْ أَخْرَجَ مَالِكٌ فِي الْمُوَطَّأِ هَذِهِ الزِّيَادَةِ عَنِ الزُّهْرِيِّ أَيْضًا،",
  "ratio": 0.878
}

Data Splits

The dataset is split 80 / 10 / 10 using a fixed random seed (42) for reproducibility:

Dataset Creation

Source Data

All text originates from Al-Maktaba Al-Shamela, an authoritative digital collection covering classical Islamic jurisprudence, hadith sciences, Quranic exegesis, Arabic linguistics, history, and related disciplines. Books were exported as HTML and processed with a custom extraction pipeline.

Extraction & Filtering

Sentences are extracted from HTML exports, then filtered according to:

• —Minimum tashkeel ratio: 0.90 — at least 90 % of Arabic base letters must carry a diacritic mark, ensuring high annotation density.
• —Minimum length: 20 Arabic letters per sentence.
• —Maximum length: 400 characters per sentence (longer sentences are sub-segmented on ، / ; boundaries).
• —Deduplication: exact-match on the diacritized string; duplicate sentences are dropped.

Diacritics covered include all standard Arabic harakat: fatha (َ), damma (ُ), kasra (ِ), fathatan (ً), dammatan (ٌ), kasratan (ٍ), sukun (ْ), shadda (ّ), and their combinations.

Sample Statistics (train split, first 10 k sentences)

Usage

Loading with 🤗 Datasets

from datasets import load_dataset

ds = load_dataset("yuujiElfahkrany/tashkeel")
print(ds["train"][0])
# {'raw': '...', 'tashkeel': '...', 'ratio': 0.867}

Manual loading

import json

def load_jsonl(path):
    with open(path, encoding="utf-8") as f:
        return [json.loads(line) for line in f]

train = load_jsonl("train.jsonl")
val   = load_jsonl("val.jsonl")
test  = load_jsonl("test.jsonl")

Character-level label scheme

The companion dataset.py maps each Arabic base character to one of 15 diacritic classes for sequence-labeling models:

Intended Uses & Limitations

Intended uses:

• —Training and benchmarking automatic Arabic diacritization (tashkeel) models.
• —Pre-training or fine-tuning Arabic language models on classical text.
• —Linguistic research into classical Arabic morphology and orthography.

Limitations:

• —The corpus is dominated by classical and pre-modern Islamic texts; performance on modern Standard Arabic or dialectal Arabic may be lower.
• —Tashkeel annotations are those present in the original Shamela sources. Occasional OCR or editorial errors in the source library may propagate to this dataset.
• —Very short sentences (< 20 Arabic letters) and very long ones (> 400 characters) are excluded.

License

This dataset is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license, in keeping with the terms of Al-Maktaba Al-Shamela, which distributes its library as a charitable Islamic endowment (waqf) — free for non-commercial use. Commercial use of this dataset is not permitted.

Citation

If you use this dataset, please cite:

@dataset{shamela_tashkeel_2026,
  title     = {Arabic Tashkeel Dataset from Al-Maktaba Al-Shamela},
  year      = {2026},
  language  = {Arabic},
  note      = {Extracted from Al-Maktaba Al-Shamela (https://shamela.ws)},
}

Acknowledgements

All classical texts are sourced from Al-Maktaba Al-Shamela (الشاملة), which provides free access to thousands of classical Arabic and Islamic works.