ChristineYe8/ReplicationBench

ReplicationBench

arXiv: ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?

GitHub: https://github.com/Christine8888/replicationbench-release

Dataset Description

The ReplicationBench dataset contains 111 astrophysics research replication tasks, spanning complete replications of 20 research papers. The dataset includes:

• —Original and masked manuscript text
• —Metadata (title, abstract, publication info, etc.)
• —Pointers to datasets and dataset access instructions
• —Additional specifications from the authors
• —Execution requirements
• —Detailed descriptions and grading guidelines for each task

Usage

from datasets import load_dataset

# Load papers
papers_ds = load_dataset("ChristineYe8/replicationbench", split="papers")

# Load tasks
tasks_ds = load_dataset("ChristineYe8/replicationbench", split="tasks")

You can load the dataset from HuggingFace into native ReplicationBench format using this script. However, if using RB's native formats, we recommend using the native data loading instead, described here.

Citation

If you use ReplicationBench in your research, please cite:

@misc{ye2025replicationbenchaiagentsreplicate,
      title={ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?},
      author={Christine Ye and Sihan Yuan and Suchetha Cooray and Steven Dillmann and Ian L. V. Roque and Dalya Baron and Philipp Frank and Sergio Martin-Alvarez and Nolan Koblischke and Frank J Qu and Diyi Yang and Risa Wechsler and Ioana Ciuca},
      year={2025},
      eprint={2510.24591},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.24591},
}

License

MIT License