LyngualLabs/Nigerian-Language-Exams-VQA

Nigerian Language Exams VQA Dataset

Dataset Summary

The Nigerian Language Exams VQA Dataset is a curated benchmark designed to evaluate the Document Visual Question Answering (VQA) capabilities of Multimodal Large Language Models (MLLMs) in indigenous West African languages. It consists of 837 manually cropped images of multiple-choice questions from standardized Nigerian examinations (WAEC and NECO) spanning the years 2008 to 2024.

This dataset addresses the critical data gap for African languages in multimodal AI, focusing specifically on Yoruba, Igbo, Hausa, and English.

Dataset Structure

The dataset is organized via a metadata.jsonl file that maps each question image to its corresponding linguistic and examination context.

Metadata Fields

• —image: The cropped .png image of the examination question.
• —question_id: Unique identifier (e.g., hau_2022_q11.png).
• —language: The language of the question (English, Yoruba, Igbo, or Hausa).
• —exam: The examination body (WAEC or NECO).
• —year: The year the exam was administered (2008–2024).
• —answer: The "Gold Standard" correct option (A, B, C, D, or E).
• —instruction: Task-specific instructions (e.g., "Choose the option nearest in meaning").

Data Composition

Maintenance & Context

• —Status: Actively Maintained.
• —Geographic Coverage: Nigeria / West Africa.
• —Source: Manually collected and scanned physical examination papers from 2008–2024.

Purpose & Impact

This dataset was developed as part of a research study titled "Challenging Multimodal LLMs with African Standardized Exams: A Document VQA Evaluation." The goal is to provide a rigorous, culturally relevant benchmark that tests a model's ability to handle complex orthographic features, such as diacritics and tone marks, which are essential for accurate Nigerian language processing.

Usage & Licensing

This dataset is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. It is intended for academic and research purposes.

Citation

If you use this dataset in your research, please cite:

@inproceedings{olufemi-etal-2025-challenging,
    title = "Challenging Multimodal {LLM}s with {A}frican Standardized Exams: A Document {VQA} Evaluation",
    author = "Olufemi, Victor Tolulope  and
      Babatunde, Oreoluwa Boluwatife  and
      Bolarinwa, Emmanuel  and
      Moshood, Kausar Yetunde",
    editor = "Lignos, Constantine  and
      Abdulmumin, Idris  and
      Adelani, David",
    booktitle = "Proceedings of the Sixth Workshop on African Natural Language Processing (AfricaNLP 2025)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.africanlp-1.22/",
    doi = "10.18653/v1/2025.africanlp-1.22",
    pages = "150--157",
    ISBN = "979-8-89176-257-2",
    abstract = "Despite rapid advancements in multimodal large language models (MLLMs), their ability to process low-resource African languages in document-based visual question answering (VQA) tasks remains limited. This paper evaluates three state-of-the-art MLLMs{---}GPT-4o, Claude-3.5 Haiku, and Gemini-1.5 Pro{---}on WAEC/NECO standardized exam questions in Yoruba, Igbo, and Hausa. We curate a dataset of multiple-choice questions from exam images and compare model accuracies across two prompting strategies: (1) using English prompts for African language questions, and (2) using native-language prompts. While GPT-4o achieves over 90{\%} accuracy for English, performance drops below 40{\%} for African languages, highlighting severe data imbalance in model training. Notably, native-language prompting improves accuracy for most models, yet no system approaches human-level performance, which reaches over 50{\%} in Yoruba, Igbo, and Hausa. These findings emphasize the need for diverse training data, fine-tuning, and dedicated benchmarks that address the linguistic intricacies of African languages in multimodal tasks, paving the way for more equitable and effective AI systems in education."
}