kelseye/test_data

1---2license: other3pretty_name: RPC-Bench4task_categories:5  - question-answering6language:7  - en8tags:9  - research-paper10  - document-understanding11  - multimodal12  - benchmark13  - llm14  - vlm15---16 17<div align="center">18 19# RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension20 21</div>22 23<p align="center">24    🌐 <a href="https://rpc-bench.github.io/" target="_blank">Project Page</a> •25    💻 <a href="https://github.com/zai-org/RPC-Bench" target="_blank">GitHub</a> •26    📖 <a href="https://arxiv.org/abs/2601.14289" target="_blank">Paper</a>27</p>28 29<div align="center">30    <img src="assets/pipeline.png" width="100%" />31</div>32 33RPC-Bench is a fine-grained benchmark for research paper comprehension. It is built from review-rebuttal exchanges of high-quality academic papers and supports both text-only and visual evaluation through complementary paper representations.34 35## Data Structure36 37RPC-Bench is organized into `train`, `dev`, and `test` subsets. Split assignments are recorded in `manifest.jsonl`, and the original split JSON files are provided in `split_metadata/` (`train.json`, `dev.json`, `test.json`).38 39`md/` contains Markdown files parsed from each paper by MinerU. These files provide the text input for LLM-oriented evaluation.40 41`parse/` contains the full MinerU parsing outputs for each paper, including structured layout and content artifacts.42 43`pdf/` contains the original paper PDFs.44 45`vlm/` contains page images rendered from the PDFs with PyMuPDF at 200 DPI for VLM-oriented evaluation.46 47```text48RPC-Bench/49├── README.md50├── manifest.jsonl51├── split_metadata/52│   ├── train.json53│   ├── dev.json54│   └── test.json55├── parse/56│   ├── train/57│   │   └── <paper_id>/58│   ├── dev/59│   │   └── <paper_id>/60│   └── test/61│       └── <paper_id>/62├── md/63│   ├── train/64│   │   └── <paper_id>/65│   │       └── <paper_id>.md66│   ├── dev/67│   │   └── <paper_id>/68│   │       └── <paper_id>.md69│   └── test/70│       └── <paper_id>/71│           └── <paper_id>.md72├── pdf/73│   ├── train/74│   │   └── <paper_id>.pdf75│   ├── dev/76│   │   └── <paper_id>.pdf77│   └── test/78│       └── <paper_id>.pdf79└── vlm/80    ├── train/81    │   └── <paper_id>/82    ├── dev/83    │   └── <paper_id>/84    └── test/85        └── <paper_id>/86```87 88## Practical Uses89 90RPC-Bench can be used to try paper-centric systems that require broader document understanding rather than local snippet matching.91 92- Research paper comprehension: try models on full-paper understanding, including core concepts, methods, and experimental findings.93- Long-context evaluation: try whether longer context windows or long-context architectures improve document-level reasoning.94- Multimodal reasoning: try models that combine textual evidence with page-level figures, tables, and diagrams in the original PDF layout.95- RAG system diagnosis: try retrieval, chunking, and evidence-fusion strategies for paper-centric workflows beyond snippet-level retrieval accuracy.96 97## Citation98 99```bibtex100@article{chen2026rpc,101  title={RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension},102  author={Chen, Yelin and Zhang, Fanjin and Sun, Suping and Pang, Yunhe and Wang, Yuanchun and Song, Jian and Li, Xiaoyan and Hou, Lei and Zhao, Shu and Tang, Jie and others},103  journal={arXiv preprint arXiv:2601.14289},104  year={2026}105}106```107