QCRI/ArGuard-Task1

ArGuard – Track A: Arabic Hateful Memes

This repository hosts the official dataset for Track A of the ArGuard shared task: multimodal hateful-meme detection in Arabic. Each instance is an Arabic meme (image + OCR-extracted overlaid text) manually annotated for hatefulness and fine-grained sub-types.

Content warning. The dataset contains text and imagery that is offensive, discriminatory, or otherwise harmful by design. Handle with care.

Track A subtasks

Given a meme (image + Arabic text):

• —Subtask A1 – Binary. Classify the meme as Hateful or Not Hateful.
• —Subtask A2 – Fine-grained category prediction (multi-label). Predict the applicable fine-grained sub-type(s) from a unified taxonomy that covers both hateful and non-hateful categories. Hateful memes draw labels from the hateful sub-type set (Mocking, Incitement, Dehumanization, Slurs, Contempt, Inferiority, Exclusion, …); non-hateful memes draw from Humor, Sarcasm, plus the shared Other. Both subtasks are evaluated on every meme.

Splits

• —dev_test was the leaderboard set for the development phase. That phase is now closed and its gold labels have been released, so dev_test is usable as additional labelled data. It is disjoint from test.
• —test was the blind test set for the final-evaluation phase. That phase has now closed and the gold labels are released, so the full dataset is labelled and the benchmark can be evaluated offline. All 500 records are triple-annotated with majority voting, which makes test the highest-quality split and the one to report results on.

Binary label distribution

Note that test is deliberately not distribution-matched to the other splits: it was drawn from the triple-annotated calibration pool, where majority voting resolves more borderline cases to Not Hateful.

Fine-grained sub-types (Subtask A2)

The Subtask A2 label space is one unified multi-label vocabulary that covers both hateful and non-hateful sub-types:

• —Hateful sub-types (active in the released data): Mocking, Incitement, Dehumanization, Slurs, Contempt, Inferiority, Exclusion.
• —Non-hateful sub-types: Humor, Sarcasm.
• —Shared: Other (used by both Hateful and Not-Hateful memes).

Five additional hateful classes appear in the annotation taxonomy but have zero training support in the released data: Extremism, Historical, Insults, Stereotyping, Threat. They are documented for completeness, accepted by the format checker, and ignored by the scorer.

In practice each meme's fine-grained labels are drawn from its own binary class: a Hateful meme will only carry hateful sub-types (and/or Other); a Not-Hateful meme will only carry Humor / Sarcasm / Other. Sub-types are multi-label, so per-class counts sum to more than the meme counts.

Record schema

{
    "id": "f9a8…b1.jpg",            # str – original image filename, unique
    "image": <PIL.Image.Image>,     # embedded bytes, decoded on access
    "text": "…",                    # str – OCR-extracted Arabic meme text
    "label": "Hateful" | "Not Hateful",  # every split is now labelled
    "fine_grained_label": [...],         # list[str], possibly empty
}

Usage

from datasets import load_dataset

ds = load_dataset("QCRI/ArGuard-Task1")
print(ds)

train_ex = ds["train"][0]
train_ex["image"].show()
print(train_ex["text"], train_ex["label"], train_ex["fine_grained_label"])

test_ex = ds["test"][0]
print(test_ex["label"], test_ex["fine_grained_label"])

Recommended benchmark protocol. Train on train + dev + dev_test (4,500 memes, single-annotator labels) and report on test (500 memes, triple-annotated majority vote). That is the split the shared-task leaderboard was scored on, so results are comparable to the published rankings.

Shared-task resources

• —Website: https://araieval.github.io/ArGuard2026/
• —Starter kit / baselines / scorers (GitHub): https://github.com/araieval/ArGuard-2026-tasks
• —Submission — Subtask A1: https://www.codabench.org/competitions/16909/
• —Submission — Subtask A2: https://www.codabench.org/competitions/16910/
• —Contact organisers: arguard2026-organizers@googlegroups.com

Annotation

• —All memes are manually annotated following the ArGuard guidelines.
• —train, dev, dev_test: single-annotator labels (bulk annotation).
• —test: triple-annotated. Binary label is the majority vote; the fine-grained label set is the union of sub-types selected by annotators whose binary label matches the majority.
• —Inter-annotator agreement on the calibration subset is above 0.81.

Intended use and limitations

• —Intended use. Research on Arabic multimodal hate speech detection, including binary classification, fine-grained sub-type prediction, and vision-language modelling.
• —Limitations. Memes reflect online discourse and contain offensive and harmful content. Annotations on train / dev / dev_test are single-annotator and may contain noise; only the held-out test split uses triple-annotated majority-voted labels.
• —Not for deployment. This dataset is for research and benchmarking; it is not a moderation tool.

License

Released under CC BY-NC 4.0 for non-commercial research use only. Not to be used for commercial purposes or for training systems that generate harmful content.

Citation

A citation will be provided when the shared-task overview paper is released. Until then, please cite this repository URL.

Contact

• —Email: arguard2026-organizers@googlegroups.com
• —Website: https://araieval.github.io/ArGuard2026/
• —GitHub: https://github.com/araieval/ArGuard-2026-tasks