ChristineYe8/ReplicationBench

1---2dataset_info:3  features:4  - name: paper_id5    dtype: string6  - name: title7    dtype: string8  - name: abstract9    dtype: string10  - name: publication_date11    dtype: string12  - name: paper_link13    dtype: string14  - name: code_available15    dtype: bool16  - name: code_link17    dtype: string18  - name: source19    dtype: string20  - name: dataset21    list:22    - name: kind23      dtype: string24    - name: dataset_name25      dtype: string26    - name: data_instructions27      dtype: string28  - name: execution_requirements29    struct:30    - name: code_language31      dtype: string32    - name: dependencies33      sequence: string34    - name: needs_gpu35      dtype: bool36  - name: other_instructions37    dtype: string38  - name: full_text39    dtype: string40  splits:41  - name: papers42    num_bytes: TBD43    num_examples: TBD44  - name: tasks45    num_bytes: TBD46    num_examples: TBD47  download_size: TBD48  dataset_size: TBD49configs:50- config_name: default51  data_files:52  - split: papers53    path: papers.jsonl54  - split: tasks55    path: tasks.jsonl56---57 58# ReplicationBench59**arXiv**: [ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?](https://arxiv.org/abs/2510.24591)60 61**GitHub**: [https://github.com/Christine8888/replicationbench-release](https://github.com/Christine8888/replicationbench-release)62 63## Dataset Description64 65The ReplicationBench dataset contains 111 astrophysics research replication tasks, spanning complete replications of 20 research papers. The dataset includes:66- Original and masked manuscript text67- Metadata (title, abstract, publication info, etc.)68- Pointers to datasets and dataset access instructions69- Additional specifications from the authors70- Execution requirements71- Detailed descriptions and grading guidelines for each task72 73## Usage74 75```python76from datasets import load_dataset77 78# Load papers79papers_ds = load_dataset("ChristineYe8/replicationbench", split="papers")80 81# Load tasks82tasks_ds = load_dataset("ChristineYe8/replicationbench", split="tasks")83```84 85You can load the dataset from HuggingFace into native ReplicationBench format using [this script](https://github.com/Christine8888/replicationbench-release/blob/main/src/dataset/hf/load_from_hf.py). However, if using RB's native formats, we recommend using the native data loading instead, described [here](https://github.com/Christine8888/replicationbench-release).86 87## Citation88 89If you use ReplicationBench in your research, please cite:90 91```bibtex92@misc{ye2025replicationbenchaiagentsreplicate,93      title={ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?},94      author={Christine Ye and Sihan Yuan and Suchetha Cooray and Steven Dillmann and Ian L. V. Roque and Dalya Baron and Philipp Frank and Sergio Martin-Alvarez and Nolan Koblischke and Frank J Qu and Diyi Yang and Risa Wechsler and Ioana Ciuca},95      year={2025},96      eprint={2510.24591},97      archivePrefix={arXiv},98      primaryClass={cs.CL},99      url={https://arxiv.org/abs/2510.24591},100}101```102 103## License104 105MIT License