theonlysanjeev/nepal-ooc-misinformation

NepOOC: Bilingual Nepali-English Out-of-Context Multimodal Misinformation Benchmark

Dataset Description

NepOOC is the first publicly available Nepali-dominant, bilingual benchmark for out-of-context (OOC) multimodal misinformation detection. OOC misinformation pairs an authentic, unmanipulated image with a misleading caption to construct a false narrative, without any image manipulation — making detection a problem of image-caption semantic alignment rather than image forensics.

The benchmark contains 1,090 image-caption pairs (545 pristine, 545 OOC), each OOC instance annotated with one of five manipulation typologies, drawn from real-world Nepali misinformation incidents verified by fact-checking organisations, news portals, and social media archives.

• —Curated by: Sanjeev Khatiwada
• —Languages: Nepali (ne, 78.5%), English (en, 14.5%), code-switched Nepali-English (ne-en, 7.0%)
• —License: Research-only, non-commercial (CC BY-NC 4.0) — see License
• —Paper: NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection
• —Repository (code): https://github.com/SanjeevKCodes/nepooc

Dataset Summary

Each of 545 unique image-source pairs yields exactly one pristine (factually accurate) and one OOC (misleadingly recaptioned) sample using the same image, for a total of 1,090 rows. OOC instances are labelled with one of five typologies: Fabricated, Miscaptioned, Temporal Mismatch, Geographic Mismatch, and Identity Mismatch. Inter-annotator agreement reached Cohen's κ = 0.84 (typology) and κ = 0.81 (binary, non-fact-checker sources).

Supported Tasks

• —`text-classification`: binary Pristine vs. OOC classification using the caption text alone (text-only baselines in the paper achieve up to 94.65% Macro-F1).
• —`image-text-to-text` / multimodal classification: binary Pristine vs. OOC classification using the image-caption pair jointly (e.g. CNN+LSTM, ViT+TCN, ResNet-50+mBERT, CLIP, ViT+MuRIL).
• —Typology classification: multi-class classification of OOC instances into one of the five manipulation typologies (multi-label-classification tag reflects the typology annotation layer; the primary task is single-label).
• —Fact-checking research more broadly: studying caption-image consistency in a low-resource, code-switched, Devanagari-script setting.

Languages

Captions are written in Nepali (Devanagari script), English (Latin script), or a code-switched mixture of both, reflecting authentic Nepali digital-media discourse. Distribution: Nepali 856 (78.5%), English 158 (14.5%), code-switched 76 (7.0%).

Dataset Structure

Data Instances

A typical instance looks like:

{
  "post_id": "TC_facebook-spreading-misinformation-free-health-checks-spreading-aids",
  "image_url": "https://techpana.prixacdn.net/media/albums/fake-news-techpana-news_E0dWBBw0Hf.jpg.760x430_q100_crop.jpg",
  "caption": "घर-घरमा निःशुल्क स्वास्थ्य जाँच गर्न आउनेले एड्स सारिदिने भन्ने भ्रामक सूचना",
  "label": 1,
  "misinformation_type": "Fabricated",
  "named_entities": "Facebook",
  "categories": "Politics",
  "posted_date": "01/11/2024",
  "language": "ne",
  "article_url": "https://techpana.com/2024/147213/facebook-spreading-misinformation-free-health-checks-spreading-aids",
  "article_title": "घर-घरमा निःशुल्क स्वास्थ्य जाँच गर्न आउनेले एड्स सारिदिने भन्दै फेसबुकमा भ्रामक सूचना"
}

Each post_id maps to a pristine/OOC pair: the same post_id appears twice in nepOOC_full.csv, once with label=0 (pristine caption/context) and once with label=1 (OOC caption), sharing the same image_url.

Data Fields

Data Splits

The benchmark is partitioned via stratified sampling into train/validation/test splits, stratified by label and source cluster to avoid leakage (validated via event-cluster analysis; see paper Section III-F — all leakage deltas |Δ| < 1%).

OOC Typology Distribution (n = 545 OOC samples)

Source Distribution (n = 545 unique images)

Dataset Creation

Curation Rationale

OOC detection research is concentrated in high-resource, English-language settings (NewsCLIPpings, COSMOS, VERITE). No public benchmark previously existed for the "regional context gap" in Nepal: code-switching between Devanagari and Latin scripts, sparse named-entity coverage for Nepali figures/locations, and the absence of multimodal OOC resources adapted to regional discourse. NepOOC was curated to provide the first such resource, grounded in real Nepali misinformation incidents spanning elections, public health, natural disasters, infrastructure claims, and communal affairs.

Source Data

Initial Data Collection and Normalisation

545 unique image-source pairs were collected from three categories of sources: fact-checking organisations operating in Nepal, Nepali online news portals, and social media archives flagged for potential misinformation. For every source, both the false claim and the verified true context were documented from the same evidentiary reference, enabling direct derivation of pristine and OOC labels without post-hoc re-annotation.

Who are the source data producers?

The underlying images and claims originate from Nepali fact-checking organisations, news portals, and social media users/pages whose posts were subsequently fact-checked. The dataset curator did not produce the original images or claims; they were aggregated, paired, and annotated from public fact-checking outputs and news archives.

Annotation Process

Binary OOC labels for fact-checker-sourced pairs (86.1% of unique sources) were established by the fact-checkers themselves as part of their published verdicts, constituting expert-verified ground truth. For the remaining pairs from news portals and social media archives (n = 76 unique sources, 14.0%), two trained annotators independently confirmed the binary OOC/Pristine classification (Cohen's κ = 0.81).

All annotators independently assigned one of five typology labels to each OOC instance, using published fact-checker analyses as reference material. Disagreements were adjudicated by a third expert; pairs with unresolvable disagreement or insufficient provenance were discarded. Typology inter-annotator agreement reached Cohen's κ = 0.84.

Personal and Sensitive Information

All data were collected from publicly available sources (published fact-checks, public news articles, and publicly posted social media content). No private communications or non-public information are included. The dataset does, however, contain captions that name public figures (e.g. politicians) in the context of political misinformation claims; these names are part of the documented misinformation record and are essential to the dataset's purpose.

Bias, Risks, and Limitations

• —Source bias: 86.1% of unique source images come from fact-checking organisations, skewing the dataset toward professionally curated, salient, politically-themed content. High model performance may partly reflect lexical patterns specific to this source distribution rather than general OOC detection ability. Generalisation to organically-produced, non-fact-checked social media content has not been tested.
• —Dataset scale: At 1,090 samples (228 test samples), a single misclassification shifts Macro-F1 by roughly 0.4 percentage points, limiting statistical power for fine-grained comparisons.
• —Typology imbalance: Identity Mismatch comprises only 1.8% of OOC instances (n = 10 overall, n = 2 in the test split), so typology-level results for this class are not statistically reliable.
• —Text-dominant signal: Caption semantics carry most of the discriminative signal at this scale; conclusions about cross-modal fusion benefits may not generalise to larger datasets or different source distributions.
• —Closed-pair constraint: Models trained on this benchmark observe only the image-caption pair, with no access to external evidence, retrieval, or knowledge bases — real-world deployment would require such grounding, especially for Fabricated, Temporal, Geographic, and Identity typologies that depend on world knowledge.
• —Temporal validity: OOC manipulation tactics evolve; strategies prevalent at data collection time (2023-2025) may not represent future misinformation patterns. Periodic refresh is recommended.
• —Hosted images: image_url values point to externally hosted images (e.g. fact-checker CDNs, news sites). Some links may become unavailable over time (link rot), and users should expect to handle missing images gracefully in multimodal pipelines.

Recommendations

Users should treat this benchmark primarily as a tool for studying caption-image alignment in a low-resource, bilingual (Nepali-English, Devanagari/Latin) setting, rather than as a general-purpose misinformation detector for deployment. Given the text-sufficiency finding reported in the accompanying paper, researchers working on multimodal fusion should report dedicated text-only and image-only baselines alongside multimodal results on this dataset. Researchers extending this benchmark are encouraged to prioritise underrepresented typologies (Geographic and Identity Mismatch) and non-fact-checker sources to reduce source bias.

Citation

BibTeX:

@misc{khatiwada2026nepooc,
  author       = {Khatiwada, Sanjeev},
  title        = {NepOOC: A Bilingual Nepali-English Benchmark for Out-of-Context
                   Multimodal Misinformation Detection},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/theonlysanjeev/nepal-ooc-misinformation}},
  note         = {Dataset accompanying ``NepOOC-M: Bilingual Nepali-English Benchmark
                   and Comparative Analysis of Multimodal Architectures for OOC Detection''}
}

License

This dataset is released under a research-only, non-commercial licence (CC BY-NC 4.0), as stated in the accompanying paper. See License Recommendation in the deployment guide for rationale and how this differs from the previously published cc-by-4.0 tag.

Acknowledgements

NepOOC was built using verdicts and source material from Nepali fact-checking organisations and news portals. We thank these organisations for their public fact-checking work, which forms the evidentiary backbone of this benchmark.