Hyukkyu/RAQUEL2-ICLR

RAQUEL2

Execution-grounded evaluation for machine unlearning. Each evaluation record is a question answered by a SQL query run against two databases: one built from the full corpus, and one with the forget-set facts removed. A record is affected when the two databases disagree, and unaffected when they agree — so the label is a measured property of the data, not an annotation.

Four configs, together enough to run the benchmark end to end:

qa   = load_dataset("Hyukkyu/RAQUEL2-ICLR", "source-qa", split="wmdp")
aff  = load_dataset("Hyukkyu/RAQUEL2-ICLR", "wmdp", split="affected")

# the muse-news source QA is split "muse_news": split names cannot take a dash
qa   = load_dataset("Hyukkyu/RAQUEL2-ICLR", "source-qa", split="muse_news")

The source QA is a separate config because its columns differ from the evaluation records', and datasets requires the splits of one config to share a schema.

Dataset statistics

13,210 evaluation records and 9,023 source QA.

Yes/No balance

A Boolean question is only informative if neither class can be answered by a constant. Nullification only removes facts, so an existence check is true before and false after — left alone, every affected Boolean answers "yes" and a model that always answers "no" scores perfect forgetting at no cost. These builds add negated twins over forgotten facts and positive controls over retained ones, so both classes carry both polarities.

Paraphrases

Every source-qa row carries one reworded question with the same reference answer, so a paraphrase is scored against the answer its original was scored against. TOFU's own paraphrases are used where it publishes them; the rest were generated with Qwen3.8-27B and screened.

A rewrite had to keep every proper noun and number, stay a question, avoid the answer's wording, and land inside a ROUGE-L band against the original. Failures were fed back once for revision; what still failed was withdrawn rather than published, which is why paraphrased_question is null on some rows and paraphrase_rejected_reason says why.

WMDP needed its own prompt: 530 of its 2,973 questions (17.8%) are multiple-choice stems rather than questions — "A 'dished face' profile is often associated with" — and a model told merely to reword one writes the question out of the answer. Its prompt asks for the question the stem implies, without describing the expected answer.

Fields

source-qa

quality_flags records defects inherited from the source benchmarks rather than hiding them. 62 of 9,023 rows carry one, and 15 — all WMDP — are marked usable_standalone: false, where the answer names multiple-choice options that are not present ("I, II, and III"). They are kept because the databases were built from them, so removing them would make this split disagree with the evaluation records it explains; filter on usable_standalone to exclude them.

affected / unaffected

answer is the fact alone — Taiwan, 3, Yes, or a comma-separated list where the query returns several values. verbose_answer is the same answer written as a sentence ("For Hsiao Yun-Hwa, the nationality is Taiwan.").

The concise form is the one to score against. In the sentence form roughly three quarters of the tokens are the question restated, so a model that has completely forgotten a fact still matches most of the expected answer under ROUGE or token-F1, and complete forgetting scores as partial. Measured across these records, 77.5% of sentence answers are at least half question words, against 4.7% of the concise ones. Exact match is only meaningful against the concise field.

answer_is_concise is false for 4.4% of records — mostly multi-column joins — where no concise form could be derived; there answer holds the sentence. Boolean polarity is taken from the sentence rather than the projected value, and is identical in both fields for every Boolean record.

question_variants holds validated rephrasings of the same query: each was generated from the slotted sentence without the model seeing any slot value, screened by an equivalence judge, and checked against the query's structure. Records average about five phrasings, so no single wording dominates — the most common one covers 4.7% of tofu, 6.0% of muse-news and 8.1% of wmdp.

Versions

To reproduce results published before 2026-09-24, pin the revision that predates this one:

load_dataset("Hyukkyu/RAQUEL2-ICLR", "tofu",
             revision="aee9a541f1ba58032e2eb62a1e56149475cd1126")

answer also changed meaning in this revision: it now holds the concise fact, and the sentence form moved to verbose_answer. A scorer written against the earlier revision's answer should read verbose_answer here.

That revision predates three fixes to label separability, so its affected Booleans skew heavily toward "yes" (79% on tofu) and its question wordings are far more concentrated. Its muse-news config was also built from a different source split, and it carries no source-qa. Numbers from it and from this revision are not comparable.

Caveats

• —wmdp is a QA recast, not WMDP's own unlearning setting: the forget set is WMDP-bio and WMDP-chem MCQ reduced to (question, correct choice), and the retain set is ten MMLU subjects converted the same way. Cyber is excluded. All facts derive from the public cais/wmdp MCQ; no hazardous corpus is redistributed.
• —muse-news source QA is synthesized from BBC news articles and refers to real people and real events, including deaths, attacks and convictions.
• —Paraphrases other than TOFU's were written by Qwen3.8-27B, the same model used elsewhere in the generation pipeline. paraphrase_source records this per row, so they can be filtered or regenerated.
• —Every evaluation label was verified by re-executing all 13,210 queries against both databases: every affected record's result changes and every unaffected record's does not.