SalahAbdoNLP/arabic-dataset-quality-assessments

Arabic NLP Dataset Quality Assessments

تقييمات جودة شاملة لمجموعات البيانات العربية في معالجة اللغات الطبيعية

Dataset Description

• —Homepage: Arabic Datasets Browser
• —Source Catalog: Masader (ARBML)
• —Language: Arabic (assessments), English (metadata)
• —License: CC-BY-4.0
• —Size: 331 assessed datasets

Dataset Summary

This dataset contains automated quality assessments of 331 Arabic NLP datasets from the Masader catalog. Each assessment evaluates a dataset across 7 quality dimensions using up to 500 data samples, producing detailed Arabic-language analysis including quality scores, statistical analysis, strengths, weaknesses, and usage recommendations.

The assessments were generated using an autonomous pipeline built on Claude (Sonnet) running inside isolated Docker containers. Each container loads the target dataset via the HuggingFace datasets library, runs statistical analysis scripts, inspects samples, and produces a structured quality report.

This is the first large-scale quality assessment of Arabic NLP datasets. The Masader catalog provides metadata about what datasets exist and where to find them — this dataset adds a quality layer on top, answering how good each dataset is.

Quality Grade Distribution

Seven Quality Dimensions

Each dataset is scored 0-100 on:

1. 1.سهولة الوصول (Accessibility) — Can the data be freely downloaded?
2. 2.التوثيق (Documentation) — Is there a paper, README, dataset card?
3. 3.السلامة الأخلاقية (Ethics) — Are there ethical risks in the content?
4. 4.الترخيص (Licensing) — Is the license clear and permissive?
5. 5.قابلية إعادة الإنتاج (Reproducibility) — Can results be reproduced?
6. 6.المراجعة العلمية (Peer Review) — Was the dataset peer-reviewed?
7. 7.جودة البيانات (Data Quality) — Are the actual samples high quality?

Dataset Structure

Data Fields

Data Splits

Usage

from datasets import load_dataset

# Load all assessments
ds = load_dataset("SalahAbdoNLP/arabic-dataset-quality-assessments")

# Filter excellent datasets
excellent = [d for d in ds["train"] if d["quality_grade"] == "ممتاز"]
print(f"Excellent datasets: {len(excellent)}")

# Get top 10 by quality score
top10 = sorted(ds["train"], key=lambda x: x["quality_score"], reverse=True)[:10]
for d in top10:
    print(f"{d['quality_score']}/100 - {d['name_original']}")

# Find datasets with data quality issues
issues = [d for d in ds["train"] if d["issues_found"]]
print(f"Datasets with issues found: {len(issues)}")

Dataset Creation

Methodology

1. 1.Source: 335 free MSA (Modern Standard Arabic) datasets from the Masader catalog with HuggingFace links
2. 2.Pipeline: Each dataset assessed by Claude (Sonnet) in an isolated Docker container (Ubuntu 22.04, Node.js 20)
3. 3.Inspection: Up to 500 samples loaded via datasets library in streaming mode
4. 4.Analysis: Embedded Python scripts run statistical analysis (duplication, encoding, text lengths, Latin punctuation)
5. 5.Output: Structured Arabic JSON with 7 quality dimensions, scores, strengths, weaknesses, and statistical evidence
6. 6.Scale: 20 parallel Docker workers, 15-minute timeout per dataset, ~6 hours total runtime
7. 7.Result: 331/335 completed (4 timeouts: clartts, doclang, mergedarabiccorpusofisolated_words, pm4bench)

Prompt Evolution

• —v1: English prompt, 10 samples → shallow assessments
• —v2: Arabic prompt, 10 samples → Arabic output but still shallow
• —v3: Arabic prompt, 500 samples, embedded statistical analysis scripts → deep assessments with precise numbers

Security Model

• —Each assessment runs in a fresh Docker container with no network access to sensitive services
• —Credentials stored on tmpfs, cleared after use
• —Containers killed after 15-minute timeout
• —Lock files (atomic mkdir) prevent parallel workers from assessing the same dataset

Limitations

• —Automated assessments: Quality judgments are made by an AI model, not human experts. Scores should be treated as indicative, not definitive.
• —MSA focus: Only Modern Standard Arabic datasets with free access and HuggingFace links were assessed. Dialectal, paid, or non-HF datasets are not included.
• —Snapshot in time: Assessments reflect dataset state as of February 2026. Datasets may be updated after assessment.
• —4 missing: clartts, doclang, mergedarabiccorpusofisolated_words, and pm4bench timed out and are not included.
• —Streaming limitations: Some datasets could not be loaded via streaming mode, resulting in assessments based on metadata and documentation only.

Citation

If you use this dataset, please cite both this work and the Masader catalog:

@misc{alyafeai2021masader,
    title={Masader: Metadata Sourcing for Arabic Text and Speech Data Resources},
    author={Zaid Alyafeai and Maraim Masoud and Mustafa Ghaleb and Maged S. Al-shaibani},
    year={2021},
    eprint={2110.06744},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

@article{altaher2022masader,
    title={Masader Plus: A New Interface for Exploring +500 Arabic NLP Datasets},
    author={Altaher, Ali and Fadel, Ali and Alharbi, Asma and Alnefaie, Ghadah
            and Alqahtani, Rawan and Aloraini, Sara and Alyafeai, Zaid},
    journal={arXiv preprint arXiv:2208.00932},
    year={2022}
}

Acknowledgments

• —Dataset metadata is sourced from the Masader catalog by ARBML, the largest public catalogue for Arabic NLP datasets
• —Quality assessments are original work produced using an automated pipeline built on Claude by Anthropic
• —Thanks to the 40+ Masader contributors who built the catalog that made this work possible