naist-nlp/RedirectQA

RedirectQA

RedirectQA is an entity-based factual QA dataset for analyzing how large language models access the same fact through different surface forms of an entity.

This v1.0.0 release is aligned with the dataset described in the paper Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms. The public dataset contains:

• —61,120 question realizations in the test split
• —30,560 subject-surface instances
• —14,672 Wikidata factual triples
• —14,672 canonical surface instances
• —15,888 redirect surface instances
• —16 relation types
• —33 redirect categories

Each surface instance is rendered with two question templates:

• —original: the original PopQA-style template
• —paraphrased: a paraphrased template generated during dataset construction

Dataset Structure

This release provides one configuration:

• —default
• —split: test
• —file: data/test.parquet

The row unit in data/test.parquet is one question realization. Since each surface instance appears under two templates, every (triplet_id, redirect_id) pair appears exactly twice: once with template_type="original" and once with template_type="paraphrased".

Main data file

• —data/test.parquet
• —61,120 rows
• —one row per question realization

Metadata files

• —metadata/question_templates.json
• —the two public question templates for each predicate
• —metadata/redirect_category_to_type.csv
• —mapping from each redirect category to one of the higher-level types used in the paper

Data Fields

The main file contains the following columns.

High-Level Redirect Types

Redirect categories are grouped into the following high-level types:

• —Aliases_and_Abbreviations
• —Spelling_variants
• —Typical_Errors

Canonical rows are marked separately as Canonical in metadata/redirect_category_to_type.csv.

Statistics

The paper reports the following dataset statistics, which match this release:

• —30,560 surface instances derived from 14,672 factual triples
• —14,672 canonical surface instances
• —15,888 redirect surface instances
• —61,120 question realizations from two templates per surface instance
• —among redirect surface instances:
• —8,667 instances associated with Aliases_and_Abbreviations
• —4,928 instances associated with Spelling_variants
• —2,884 instances associated with Typical_Errors

Source Data and Licensing Notes

This release is distributed under CC BY-SA 4.0.

It is derived from multiple resources, including:

• —English Wikipedia redirects and titles
• —Wikidata factual triples
• —Wikimedia pageview statistics
• —the PopQA setup for relation selection and original question templates

Underlying source licenses differ by source:

• —Wikipedia text: CC BY-SA 4.0
• —Wikidata and pageviews: CC0 1.0
• —PopQA: MIT

Loading the Dataset

from datasets import load_dataset

dataset = load_dataset("naist-nlp/RedirectQA", split="test")
print(dataset)

Citation

If you use RedirectQA, please cite the paper:

@misc{nishida2026revisitingnonverbatimmemorizationlarge,
  title={Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms},
  author={Yuto Nishida and Naoki Shikoda and Yosuke Kishinami and Ryo Fujii and Makoto Morishita and Hidetaka Kamigaito and Taro Watanabe},
  year={2026},
  eprint={2604.21882},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2604.21882},
}