tiiuae/NativeQA

3LM Native STEM Arabic Benchmark

Dataset Summary

The 3LM Native STEM dataset contains 865 multiple-choice questions (MCQs) curated from real Arabic educational sources. It targets mid- to high-school level content in Biology, Chemistry, Physics, Mathematics, and Geography. This benchmark is designed to evaluate Arabic large language models on structured, domain-specific knowledge.

Motivation

While Arabic NLP has seen growth in cultural and linguistic tasks, scientific reasoning remains underrepresented. This dataset fills that gap by using authentic, in-domain Arabic materials to evaluate factual and conceptual understanding.

Dataset Structure

• —question_text: Arabic text of the MCQ (fully self-contained)
• —choices: List of four choices labeled "أ", "ب", "ج", "د"
• —correct_choice: Correct answer (letter only)
• —domain: Subject area (e.g., biology, physics)
• —difficulty: Score from 1 (easy) to 10 (hard)

{
  "question_text": "ما هو الغاز الذي يتنفسه الإنسان؟",
  "choices": ["أ. الأكسجين", "ب. ثاني أكسيد الكربون", "ج. النيتروجين", "د. الهيدروجين"],
  "correct_choice": "أ",
  "domain": "biology",
  "difficulty": 3
}

Data Sources

Collected from open-access Arabic textbooks, worksheets, and question banks sourced through web crawling and regex-based filtering.

Data Curation

1. 1.OCR Processing: Dual-stage OCR (text + math) using Pix2Tex for LaTeX support.
2. 2.Extraction Pipeline: Used LLMs to extract Q&A pairs.
3. 3.Classification: Questions tagged by type, domain, and difficulty.
4. 4.Standardization: Reformatted to MCQ and randomized correct answer positions.
5. 5.Manual Verification: All questions reviewed by Arabic speakers with STEM background.

Code and Paper

• —3LM repo on GitHub: https://github.com/tiiuae/3LM-benchmark
• —3LM paper: https://aclanthology.org/2025.arabicnlp-main.4/

Licensing

Falcon LLM Licence

Citation

@inproceedings{boussaha-etal-2025-3lm,
    title = "3{LM}: Bridging {A}rabic, {STEM}, and Code through Benchmarking",
    author = "Boussaha, Basma El Amel  and
      Al Qadi, Leen  and
      Farooq, Mugariya  and
      Alsuwaidi, Shaikha  and
      Campesan, Giulia  and
      Alzubaidi, Ahmed  and
      Alyafeai, Mohammed  and
      Hacid, Hakim",
    booktitle = "Proceedings of The Third Arabic Natural Language Processing Conference",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.arabicnlp-main.4/",
    doi = "10.18653/v1/2025.arabicnlp-main.4",
    pages = "42--63",
    ISBN = "979-8-89176-352-4",
}