irioder/littleHermione-benchmark

Dataset card: O.W.L. & N.E.W.T. Bench development set v0.4

Summary

Version 0.4 is a reviewed development benchmark of 75 short-answer factual questions about the seven English-language Harry Potter novels. It contains two separately scored examinations:

• —30 challenging O.W.L. questions covering recurring book canon beyond famous entrance-level facts;
• —45 frontier N.E.W.T. questions covering chapter-level prose details, minor names, precise objects, prices, and ordered recall.

Leakage warning

This is a public development set. littleHermione 0.8B was deliberately trained on every question and primary answer in this exact split. Its 100% score is therefore a contamination demonstration, not a valid held-out result. Do not use this split to claim generalisation by that model or by any model known to have trained on it.

Loading the dataset

from datasets import load_dataset

dataset = load_dataset(
    "irioder/littleHermione-benchmark",
    split="validation",
)

Each row contains:

• —id: stable question identifier;
• —exam: owls or newts;
• —category: editorial lore category;
• —canon: currently always books;
• —question: independently worded short-answer prompt;
• —answers: accepted exact-answer aliases;
• —answer_parts and answer_parts_ordered: optional structured requirements for multi-part answers;
• —source: canonical novel and chapter locator;
• —review_status: project curation status, not independent human validation.

Intended uses

• —Compare accessible and deep wizarding-lore knowledge in language models.
• —Compare a base model with a locally trained adapter.
• —Study knowledge acquisition, retention, calibration, and abstention.

It is not intended to measure general intelligence, reading comprehension, literary quality, or moral reasoning. It is not a proxy for overall model quality.

Canon policy

Version 0.4 uses the seven novels only. Film-only details, games, stage plays, interviews, and Wizarding World articles are out of scope even where commonly treated as canon. Future canon tracks should remain separately scored.

Construction and review

The questions were written specifically for this benchmark through an AI-assisted research and calibration process; they were not imported from an existing question-answer dataset and do not reproduce passages from the novels. The initial paper was drafted from the model's existing knowledge. The harder passes used The Harry Potter Lexicon as their principal discovery and fact-checking reference, then anchored retained items to the corresponding novel chapters.

v0.4 was rebuilt after the old O.W.L. paper and the first N.E.W.T. rewrite saturated on max-effort frontier models. Candidate pools were probed, apparent misses received item-level review, ambiguous floor-numbering and an incorrect Cattermole ordering were removed, and retained questions were checked against chapter-indexed references. The bundled items are marked reviewed; this means they passed the project's curation process, not that they received independent human or scholarly verification.

The O.W.L. paper promotes the useful middle-depth questions from the previous release. The N.E.W.T. paper is balanced across all seven books and emphasizes micro-details with clean, deterministic answers rather than trick wording.

The two difficulty assignments are editorial hypotheses. A stable release should add:

• —human-fan pilot accuracy;
• —baseline accuracy from diverse, untouched models;
• —inter-reviewer agreement;
• —ambiguity and adaptation-confusion flags;
• —fixed public-development and sealed-test splits.

Scoring

The primary score uses exact alias matching after conservative normalization. Optional ordered answer parts require every fact while tolerating natural labels or short explanations between facts. An unanswered question, malformed provider response, or request error scores zero. Reports expose coverage and errors so operational failures are not mistaken for knowledge deficits.

O.W.L. and N.E.W.T. percentages are reported separately. The overall benchmark score is their equal-weight macro average, emitted only for a complete run.

Limitations

• —Public questions can enter future model training data.
• —Exact matching can still reject a correct but unlisted paraphrase.
• —Difficulty has model calibration but not yet human-fan calibration.
• —Familiarity with adaptations can help or interfere despite the books-only policy.
• —English-only wording makes cross-language comparisons inappropriate.
• —The two exams contain different numbers of questions; their raw combined accuracy is not the official equal-weight benchmark score.
• —The research process did not preserve a reliable one-to-one URL ledger for every item. PROVENANCE.md documents the recoverable process-level record and representative reference pages without inventing false precision.

Rights and attribution

The seven Harry Potter novels by J. K. Rowling are the canonical source of the underlying fictional facts. The Harry Potter Lexicon was the principal research and fact-checking reference for the harder question passes. HP-Quiz and HarryPotterQA were reviewed only as related work; neither was imported or used to populate this dataset. See `PROVENANCE.md` for the full construction record and representative source pages.

The benchmark code is MIT licensed. To the extent that the project holds rights in the dataset's original wording, answer aliases, selection and arrangement, those contributions are also offered under the repository's MIT license. This does not grant rights in the novels, reference material, character names, places, marks or the underlying fictional universe. Users are responsible for ensuring that training corpora and model usage comply with applicable licenses and law.

This project is independent and unofficial, and is not affiliated with J. K. Rowling, Warner Bros., or Wizarding World.