neulab/merlin-rare

MERLIN-Rare

MERLIN-Rare contains the rare-entity evaluation slices introduced in Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking. It augments examples from MERLIN with 15 Wikipedia and Wikidata metrics, each example's percentile under each metric, and membership in the bottom-5% slice for each metric.

The companion evaluation framework is available at neulab/think-before-you-link.

Dataset structure

The dataset has one configuration for each MERLIN language.

There are 1,105 entity mentions and 790 unique images in total. Each row preserves the MERLIN input and label fields:

• —Article Title
• —Entity Name
• —Wikidata ID
• —English Wikipedia Title
• —Image Name, stored as a Hugging Face Image
• —Image Filename, a stable filename for local materialization

For each metric <metric>, the dataset adds its integer value, <metric>_percentile, and the Boolean rare_<metric>. Percentiles are computed within each language after excluding examples without a valid English Wikipedia title, matching the slice construction. The Boolean is the authoritative membership indicator for the paper's bottom-5% test-set slice. Ties at the cutoff can make a slice contain more than exactly 5% of examples.

The 15 metrics are:

• —Wikipedia pageviews over 90 days
• —Wikipedia backlinks
• —article size in bytes
• —reference count
• —revision count
• —unique editors
• —category count
• —image count
• —external links
• —Wikidata incoming links
• —Wikidata outgoing links
• —language editions
• —statement count
• —qualifier count
• —entity age in days

Metric collection ran from October 7 through October 8, 2025. Pageviews use the preceding 90-day window. Exact field names and per-language slice counts are in metadata/slice_counts.json.

Load the data

from datasets import load_dataset

dataset = load_dataset("neulab/merlin-rare", "hi", split="test")
print(dataset[0])

The companion repository includes scripts/materialize_hf_dataset.py, which writes evaluator-compatible JSON and image directories.

Predictions and traces

The release also includes complete per-example outputs for the paper experiments:

• —all 18 Qwen3-VL configurations from 2B, 4B, and 8B Instruct and Thinking models crossed with no retrieval, BM25 retrieval, and embedding retrieval
• —three complete GLM-4.6V-Flash configurations used in the analysis

Compact prediction tables are under predictions/<family>/<configuration>/<language>.parquet. Lossless run outputs, including reasoning and tool-call traces where produced, are deterministic gzip archives under traces/<family>/<configuration>/<language>.json.gz.

metadata/artifact_manifest.json records row counts and SHA-256 hashes for every source JSON and compressed trace. Sanitized run configurations are under metadata/run_configs. The unstable GLM non-thinking retrieval run is not included because it did not complete the evaluation protocol.

Licensing and attribution

MERLIN-Rare is derived from MERLIN and is released under CC BY-SA 4.0. Wikipedia text and images retain their original licenses. Users should follow the attribution and reuse requirements of MERLIN, Wikipedia, and each image's source page.

Acknowledgments

We thank Ibrahim AlRayes for his help with this project, and Jean de Dieu Nyandwi and Zaid Sheikh for sharing resources that supported this work. We also thank the members of NeuLab for their helpful feedback.

This work was supported in part by a research grant from the Defence Science and Technology Agency (DSTA), Singapore.

Citation

@misc{pengpun2026thinklinkrarityreasoning,
      title={Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking}, 
      author={Parinthapat Pengpun and Simran Khanuja and Graham Neubig},
      year={2026},
      eprint={2609.10745},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.10745}, 
}

@article{ramamoorthy-etal-2026-merlin,
    title = "{MERLIN}: A Testbed for Multilingual Multimodal Entity Recognition and Linking",
    author = "Ramamoorthy, Sathyanarayanan  and
      Shah, Vishwa  and
      Khanuja, Simran  and
      Sheikh, Zaid  and
      Jie, Shan  and
      Chia, Ann  and
      Chua, Shearman  and
      Neubig, Graham",
    journal = "Transactions of the Association for Computational Linguistics",
    volume = "14",
    year = "2026",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://aclanthology.org/2026.tacl-1.19/",
    doi = "10.1162/tacl.a.633",
    pages = "399--417",
    abstract = "This paper introduces MERLIN, a novel testbed system for the task of Multilingual Multimodal Entity Linking. The created dataset includes BBC news article titles, paired with corresponding images, in five languages: Hindi, Japanese, Indonesian, Vietnamese, and Tamil, featuring over 7,000 named entity mentions linked to 2,500 unique Wikidata entities. We also include several benchmarks using multilingual and multimodal entity linking methods exploring different language models like LLaMa-2 and Aya-23. Our findings indicate that incorporating visual data improves the accuracy of entity linking, especially for entities where the textual context is ambiguous or insufficient, and particularly for models that do not have strong multilingual abilities. For the work, the dataset, methods are available online.1"
}