luoojason/mm-long-storytelling-bench

MM Long Storytelling Bench — v3

⚠️ The 756 model-drafted questions have been WITHDRAWN from this dataset's splits (2026-08-05). They were drafted by a model that is also an evaluation target, which makes them circular as a measurement instrument. They are kept in full, with the reasoning, under `data/v3/archive/` — nothing was deleted. The splits currently hold 6 worked examples (status: "example"), which document the required format and are not a benchmark. Do not use this dataset for evaluation yet. Human-written replacements are in progress: 600 questions across a 100-source active cohort, each recording its author and a separate verifier. The corpus below — 125 sources, 13.5M tokens — is complete, verified, and unaffected.

The corpus is the finished part of v3. It roughly doubles v2 and covers every source.

What changed from v2

• —65 new sources. Sampled across 11 genre bands using Project Gutenberg's own subject headings and bookshelves, and across five era bands by author birth year — deliberately reaching past v2's 1890s–1920s concentration. Author cap enforced corpus-wide: no author appears twice.
• —Multi-volume works are now whole novels. v2 contained single volumes of multi-volume works, which are not novels — a middle volume has no beginning and no ending. Nine such fragments were found; eight had complete sets on Project Gutenberg and were merged into complete works, one was dropped. Notably farjeon_miser_farebrother and farjeon_miser_farebrother2 in v2 were volumes 1 and 3 of the same novel, with volume 2 missing entirely; v3 ships the complete three-volume novel, and its 12 questions were re-anchored to it with no answer changes.
• —Corrected count. v2's question files hold 354 questions across 59 sources — not the "348 across 58" previously recorded.
• —Chapter structure restored for every source (sources/structure.json); 101 of 125 have detectable chapters.

Verification

verify.py re-derives every claim from the corpus text rather than trusting stored metadata, and gates on 13 checks — all passing:

• —story ids unique; stored tokens/words match a recount; every source clears a 50,000-token floor; no source is a volume fragment; n_chapters consistent across files
• —every gold_story_id resolves; every question has a non-empty answer and ≥1 supporting quote
• —every supporting quote appears verbatim in its gold source, and exactly once — so each span is deterministically locatable
• —stored separation matches a recount
• —the answer never appears in its own question; no supporting quote appears in its question

Additionally, within a source no question contains another question's answer, since the eval packs a source's questions together.

Token counts use cl100k_base; word counts use \w+. Both reproduce all 60 of v2's stored values exactly.

Known limitations — please read before using

1. 1.Nothing is human-verified. All 756 items are status: draft.
2. 2.The difficulty gate is mostly unrun. Closed-book answerability has been checked on 30 items only. An item a model can answer without the text measures recall, not reading; that check is what makes this a reading benchmark, and it is incomplete.
3. 3.Four sources were drafted from a partial read. The largest merged works (halidom_wonder_club, russell_my_shipmate, marcet_berthas_visit, braddon_fatal_three, 210k–356k tokens) exceed a single context. Numeric-fact coverage is near-complete via whole-file sweeps, but narrative context for large stretches is unverified. All 24 affected questions carry a review_flags entry saying exactly what was and was not read. Their spans are verbatim, unique and correctly separated like any other — but a restatement elsewhere that would make an answer ambiguous cannot be ruled out.
4. 4.Separation threshold differs from v2. v3 requires multihop/computation spans ≥10% of the text apart (v2's README claims ≥20%). Note also that separation is normalised by total text length, so the merged multi-volume works score lower for the same absolute gap.
5. 5.No packed eval contexts. v2 shipped eval/ with pre-packed k-distractor contexts. v3 ships sources and questions only; packing is pending a decision about the new long tail (six sources now exceed v2's longest novel, up to 355,814 tokens).
6. 6.Question authoring is model-drafted. Every question was drafted by a model that read the source, then screened by a deterministic validator. review_flags records anything the drafter or the validator flagged.

Layout

data/v3/
  sources/stories.jsonl        125 records, full text in `text`
  sources/manifest.json        the same records without text
  sources/structure.json       per-source chapter offsets and counts
  sources/provenance.md        Gutenberg ids, licensing, counts
  sources/multilingual_pairs.json   3 verified EN↔FI translation pairs
  sources/{retrieval,multihop,computation}.jsonl   756 questions
  sources/stories/<id>.md      per-source text
  by_story/<id>/               story.md + questions.jsonl + rendered README
  verify_report.md

MM Long Storytelling Bench — v2

⚠️ Draft / preview. Every item ships status: "draft". The dataset is structurally verified (every supporting quote is a verbatim substring of its source; every multi-hop and computation item aggregates over passages ≥20% of the book apart; no question contains a word distinctive to its gold novel). It is not yet difficulty-gated — the closed-book G/E/P/S/L evaluation has been run only on a 30-item subset, not the full set. Treat this as a working research preview, not a finished benchmark. A model may never be the sole author or verifier of a benchmark item; a human verification pass is pending.

A long-context reading-comprehension benchmark built on stories — the follow-up to MUDDLE (Algoverse AI Research, in collaboration with researchers from PocketFM). Where MUDDLE tested finding one fact in a pile of research PDFs, this tests whether a model actually follows a story at length.

The core idea

Difficulty comes from aggregation over a long work, not from unseen text. A model may know a novel's plot; it does not know how many days separate two dated events, or what three sums scattered across the book add to. The benchmark is deliberately built on novels a model may have read — the guarantee is at the level of the question, not the story: each item is designed to be un-answerable without reading, and difficulty is carried by counting, state-tracking, interval arithmetic, and event ordering that no plot summary records.

Sources — 60 obscure full-length public-domain novels

4,852,314 words / 6,452,749 tokens across 30 authors, each novel 50k–202k tokens. These are obscure genre novels (Edwardian mystery, adventure, romance) by prolific but now-forgotten authors — chosen so the model has not absorbed a study-guide apparatus (Wikipedia/SparkNotes chapter summaries, character lists, timelines) that would let it answer an aggregation question from memory. All are US public domain (Project Gutenberg, header / trademark stripped); every author died before 1957. Provenance in data/v2/sources/provenance.md.

The three question types — 354 questions, 118 / 118 / 118

• —retrieval — the gold novel is identified by a paraphrased trait (never its title or a proper noun from it), then a specific incidental detail is asked. The context holds the gold novel beside a same-author, same-era sibling as a hard negative.
• —multihop — ≥2 hops, the bridge entity never named in the question; the supporting passages sit ≥20% of the book apart.
• —computation — aggregation (counting, summing, interval arithmetic, ordering) over ≥2 widely separated regions; no single chapter suffices.

Six questions per novel (2 per type), evenly spread. Multi-hop/computation passage separation: median 55%, up to 98%.

Loading

from datasets import load_dataset

ds  = load_dataset("luoojason/mm-long-storytelling-bench", "v2", split="computation")
src = load_dataset("luoojason/mm-long-storytelling-bench",
                   data_files="data/v2/sources/stories.jsonl", split="train")
text = {s["id"]: s for s in src}   # id -> full novel record

row = ds[0]
row["question"]          # the prompt
row["answers"]           # gold answer(s) as list<str>, e.g. ["20000"]
row["answer_format"]     # "string" | "number" | "list" — how to score
row["document_ids"]      # the packed context, in order (k1: gold + 1 same-author sibling)

# Reconstruct the >=100k-token context deterministically:
context = "\n\n".join(
    f'=== {text[d]["title"]} ===\n\n{text[d]["text"]}' for d in row["document_ids"]
)

The context is not inlined — at k1 each is a whole novel plus a sibling (~220k tokens), and every novel is the gold of six questions, so inlining would duplicate the 60 novels dozens of times (~300 MB). The document_ids give the exact packed order; the novels ship once under sources/. Larger distractor sets (k3, k7, all 60 novels ≈ 6.5M tokens) come from the same stories.jsonl with more document_ids; k1 is the config because it is the smallest packing in which every item clears the 100k-token long-context floor.

Score a prediction: for answer_format == "number" compare the number; for "list" require all items; for "string" accept a semantic match (an LLM judge is recommended for free-form answers). Every item is status: "draft" pending human verification.

Raw sources

data/v2/sources/ ships everything for reproduction: the 60 novels (stories.jsonl + one file each under stories/), the questions split by type with answer + reasoning + verbatim supporting spans + measured passage separation, per-novel provenance.md (Gutenberg ids, download counts, word/token/chapter counts), structure.json (chapter offsets), and multilingual_pairs.json (3 verified obscure English↔Finnish same-work translation pairs for a multilingual split).

Prior batch

sourced/ holds an earlier, superseded set of 11 obscure Gutenberg short stories and excerpts. v2 keeps the same obscurity property but uses full-length novels and aggregation questions, which fixes the length and difficulty limitations of that batch.

Licensing & attribution

All source text is US public domain (Project Gutenberg, header/trademark stripped); the curation artifacts are released CC0. Produced by a researcher at Algoverse AI Research, in collaboration with researchers from PocketFM — not a PocketFM product or endorsement.