anonymous12341952/Narrative-Infilling

Dataset Card for NarrativeInfilling Benchmark

Dataset Details

Dataset Description

The Narrative Infilling Benchmark is a large-scale evaluation dataset for narrative infilling the task of generating a missing span within a narrative while maintaining consistency with both the preceding and following context. The benchmark spans four narrative domains and contains 9,142 instances with systematic variation in blank position and span length, enabling controlled evaluation of language model infilling capabilities.

Each instance provides a narrative with one masked span (indicated by ____) and the corresponding gold answer. The benchmark is designed to evaluate instruction-tuned LLMs under varying prompt specificity and reasoning guidance.

Uses

Direct Use

• —Narrative infilling evaluation: Assess how well LLMs reconstruct missing narrative spans across diverse genres and blank positions.
• —Prompt sensitivity analysis: Study the effect of instruction specificity and reasoning paradigms on Story Completion.
• —Metric evaluation: Benchmark automatic metrics against qualitative human judgments for narrative infilling tasks.

Out-of-Scope Use

• —Not suitable for training language models directly without appropriate data splits to prevent leakage.
• —Not suitable for factual question answering or non-narrative text reconstruction.

Dataset Structure

Format: Single CSV file.

Fields

Splits

The dataset is distributed as a single file. Users can split by dataset column for domain-specific evaluation or by unit_idx for position-specific analysis.

Dataset Creation

Curation Rationale

No existing benchmark directly evaluates LLM narrative infilling across multiple genres with controlled variation in blank position and span length. This benchmark fills that gap by providing a standardized multi-domain evaluation setting for narrative reconstruction.

Source Data

Instances were derived from four publicly available datasets:

Annotation Process

No additional human annotation was performed during dataset construction. The gold answers are the original masked sentences from the source texts. Human evaluation of model outputs was conducted separately using a five-dimensional qualitative rubric (Fluency, Context Faithfulness, Bidirectional Coherence, Narrative Consistency, Informativeness) as described in the accompanying paper.

Bias, Risks, and Limitations

• —Domain shift: Performance varies substantially across domains; models strong on CNN/DailyMail may not generalize to SIND.
• —Pretraining overlap: Source corpora are publicly available and may appear in LLM pretraining data. Contamination analysis in the accompanying paper shows a very small fraction of responses exhibit near-verbatim reproduction.
• —Contiguous spans only: Real-world narrative gaps may involve non-contiguous or stylistic missing content not covered by this benchmark.

Citation

@inproceedings{narrativeinfilling2025,
  title  = {Evaluating Narrative Infilling in Large Language Models},
  year   = {2025},
}

More Information

This release supports the reproducibility of the results reported in the accompanying paper. The benchmark data, evaluation code, and model outputs are publicly available in the accompanying repository.