QCRI/AHA-MEMES

AHA-Memes

A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes

Hateful memes carry their meaning in the interaction between an image and the text laid over it, and often through cultural references that neither modality states outright. Arabic has been badly served here: the meme resources that exist annotate propaganda or coarse "harmful content", not who is being attacked or how.

AHA-Memes is a benchmark of 5,000 Arabic memes, each annotated by trained native speakers for binary hatefulness and, where a meme is hateful, for the attack strategy it uses. A further 66,413 memes ship with labels and rich descriptive metadata generated by Gemini 3.1 Pro, for weakly supervised and semi-supervised work.

⚠️ Content warning. This dataset contains hateful, offensive and disturbing material, including slurs and dehumanising imagery targeting protected groups. It is released to enable research on detection and moderation, not to endorse any of it.

• —Paper: arXiv:2607.27393
• —Code, baselines and reproduction recipes: github.com/MohamedBayan/AHA-Memes
• —Licence: CC BY-NC 4.0 (non-commercial research)

Loading it

from datasets import load_dataset

test = load_dataset("QCRI/AHA-MEMES", split="test")
print(test[0]["label"], test[0]["fine_grained_label"])

# The silver corpus is ~5 GB; stream it rather than downloading it whole.
silver = load_dataset("QCRI/AHA-MEMES", split="silver", streaming=True)

If you want files on disk instead — which is what the image and fusion baselines in the code repository expect — use the downloader there:

python -m aha.download                  # train + dev + test
python -m aha.download --splits silver

Splits

The 5,000 human-annotated memes are the benchmark. They are stratified by the binary label, no meme appears in more than one split, and silver is disjoint from all three. All results in the paper come from the human splits.

Fields

Check `annotation_source` before treating a label as ground truth. It is the only thing distinguishing the human benchmark from the silver corpus once the splits are concatenated.

The taxonomy

Binary hatefulness sits at the top. A meme is Hateful if it attacks people, directly or indirectly, on the basis of a protected characteristic. Two boundaries are deliberate: attacks on groups that themselves perpetrate hate are not counted, and content that is merely rude or offensive without targeting a protected category is Not Hateful.

The ten fine-grained categories are multi-label and conditional on that decision:

Other is reachable from either side, which is why the paper's split table lists it twice. Full definitions and the bilingual guidelines the annotators worked from are in `docs/annotation-guidelines.md`.

Label distribution

Counts exceed the number of memes because the label is multi-label: 9.9% of train and 21.4% of test carry more than one category. The taxonomy is long-tailed — Exclusion has three test instances — which is why macro-F1 reads far below micro-F1 for every system.

The meta struct

Generated by Gemini 3.1 Pro for every row. On the human splits it was produced conditioned on the human label, so the model described the meme but never relabelled it; on silver the same call produced both label and description.

topic, mentioned_categories, dialect, visual_manipulation, ocr_text, ocr_english_translation, intent, stance, sentiment, emotion, text_image_relationship, meaning_type, cultural_references, context_scope, requires_current_event_knowledge, propaganda, propaganda_techniques, english_rationale, arabic_rationale, confidence, subtype_raw.

Two are easy to misread. meta.ocr_text is not the same as the top-level text: the former is what Gemini read off the image, the latter is EasyOCR's output, and they disagree often enough to be useful if you care about OCR quality. And mentioned_categories is a model guess, not a human target annotation — the human target labels are not part of this release.

Known facts in the silver split

• —116 rows have no label. For 115 the model's reply could not be parsed; one parsed but omitted the label. They still carry image and OCR text, so they are usable as unlabelled data. Filter with ds.filter(lambda x: x["label"] is not None).
• —Three rows got an out-of-taxonomy category (Criticism ×2, Satire ×1). fine_grained_label is left empty for those so the label space stays closed; the literal reply is preserved in meta.subtype_raw.
• —Silver fine-grained labels are single-label. The prompt asked for one subtype, so unlike the human splits these never have more than one element. Do not mix the two when computing multi-label statistics.
• —The class balance differs sharply. 0.9% of silver is hateful against 37% of the human splits. The human set was built by pre-selecting candidates with Gemma-3-12B to raise the positive rate; silver covers the broad pool.
• —It skews Egyptian. 56,697 of the 66,297 labelled silver rows are tagged egyptian dialect and daily_life is the dominant topic. Large, but not evenly spread across the Arabic-speaking world.

How it was built

Memes were collected from public pages and groups on Facebook, Instagram, Pinterest and Twitter/X, focused on public figures, politics and social commentary. Exact and near-duplicate images were removed using embeddings from a model fine-tuned on social-media imagery, treating pairs within Euclidean distance 3.6 as duplicates. Overlaid text was extracted with EasyOCR; memes with no detectable text were dropped, so every meme here has both modalities.

Because hateful content is rare in the wild, Gemma-3-12B assigned provisional binary labels to 71K memes and the 5,000-meme annotation set was sampled from that pool. Those provisional labels were hidden from annotators and discarded — all gold labels are human.

Annotation was done by a third-party company: three trained native Arabic speakers working from bilingual guidelines, after several rounds of training and guideline refinement, at a cost of roughly $4K. Agreement (Cohen's κ, macro-averaged over subtypes and annotator pairs) was 0.91 for binary hatefulness, 0.75 for hate type and 0.67 for the non-hateful subtypes.

Benchmark results

From the paper's Table 2, on the 1,000-meme test split. Regenerate any of these from the committed predictions in the code repository — no GPU needed.

Fine-tuning the open 8B VLM wins the binary task. Zero-shot models are badly under-sensitive — InternVL3.5-8B recalls 6% of hateful memes while posting 0.676 accuracy, barely above the majority baseline. And nothing clears 0.35 macro-F1 on the fine-grained task, where the best system is a prompted closed model: 3,500 training memes are not enough for the long tail.

Intended use and limitations

Intended for research on Arabic multimodal hate detection and safer content moderation. Not for commercial use, and not for targeting, profiling or otherwise harming individuals or groups.

The data comes from four public platforms and does not cover every dialect, region or platform community, nor private or ephemeral content. Some annotation boundaries are genuinely hard — offensive humour, satire, political criticism and protected-group hate shade into one another, which is what the detailed guidelines and the calibration rounds were for. The silver corpus broadens the resource but its labels are model-generated and unverified. Models trained here can both miss harmful content and over-flag legitimate speech, so they belong behind human oversight rather than in front of it.

Citation

@inproceedings{kmainasi2026aha,
  title={{AHA-Memes}: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes},
  author={Kmainasi, Mohamed Bayan and Shahroor, Ali Ezzat and Hasnat, Abul and Biswas, Md Rafiul and Zaghouani, Wajdi and Alam, Firoj},
  booktitle={Proceedings of the Fourth Arabic Natural Language Processing Conference (ArabicNLP 2026)},
  address={Budapest, Hungary},
  month=oct,
  year={2026},
  url={https://arxiv.org/abs/2607.27393}
}

Acknowledgments

Supported by NPRP grant 14C-0916-210015 from the Qatar National Research Fund, part of the Qatar Research Development and Innovation Council (QRDI). The findings reported here are solely the responsibility of the authors.