caskcsg/LongBench-Pro
LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark LongBench-Pro, containing 1,500 samples, is entirely built on authentic, natural long documents and includes 11 primary tasks and 25 secondary tasks, covering all long-context capabilities assessed by existing benchmarks. It employs diverse evaluation metrics, enabling a more fine-grained measurement of model abilities, and provides a balanced… See the full description on the dataset page: https://huggingface.co/datasets/caskcsg/LongBench-Pro.
92.5k
1---2license: apache-2.03configs:4- config_name: default5 data_files:6 - split: test7 path: longbench_pro.json8task_categories:9- question-answering10- text-classification11- table-question-answering12- summarization13language:14- en15- zh16tags:17- Long Context18- Realistic19- Comprehensive20pretty_name: LongBench Pro21size_categories:22- 1K<n<10K23---24 25<div align="center">26 <img src="images/logo.png" width="80" alt="LongBench-Pro Logo"/>27 <h1>LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark</h1>28</div>29 30<div align="center">31 32[](https://huggingface.co/datasets/caskcsg/LongBench-Pro) 33[](https://github.com/caskcsg/longcontext/tree/main/LongBench-Pro) 34[](https://arxiv.org/abs/2601.02872) 35[](https://huggingface.co/spaces/caskcsg/LongBench-Pro-Leaderboard)36 37</div>38 39---40 41**LongBench-Pro**, containing **1,500 samples**, is entirely built on **authentic, natural long documents** and includes **11 primary tasks and 25 secondary tasks**, covering all long-context capabilities assessed by existing benchmarks. It employs **diverse evaluation metrics**, enabling a more fine-grained measurement of model abilities, and provides a balanced set of **bilingual samples in both English and Chinese**. 42 43In addition, **LongBench Pro** introduces a multi-dimensional taxonomy to support a comprehensive evaluation of models under different operating conditions:44 45- **Context Requirement**: *Full* context (global integration) versus *Partial* context (localized retrieval);46- **Length**: Six lengths uniformly distributed from *8k to 256k* tokens, used to analyze scaling behavior;47- **Difficulty**: Four levels ranging from *Easy to Extreme*, defined based on model performance.48 49<div align="center">50 <img src="images/bench_comparison.png" width="100%"/>51</div>52 53## 🧩 Task Framework54 55<div align="center">56 <img src="images/task_definition.png" width="100%"/>57 <br />58 <br />59 <img src="images/task_map.png" width="80%"/>60 <br />61 <b>Task mapping between LongBench Pro and existing benchmarks</b>62</div>63 64## 📊 Dataset Statistics65 66<div align="center">67 <img src="images/sample_distrubution.png" width="100%"/>68</div>69 70## 📝 Data Format71 72**LongBench Pro** organizes data in the following format:73 74```json75{76 "id": "Sample ID: unique for each sample.",77 "context": "Long context: 14 types of texts covering domains such as news, medicine, science, literature, law, and education, with various forms such as reports, tables, code, dialogues, lists, and JSON.",78 "language": "Sample language: English or Chinese.",79 "token_length": "Sample token length: 8k, 16k, 32k, 64k, 128k, or 256k (calculated using the Qwen tokenizer)",80 "primary_task": "Primary task type: 11 types.",81 "secondary_task": "Secondary task type: 25 types.",82 "contextual_requirement": "Contextual Requirement: Full or Partial.",83 "question_nonthinking": "Non-thinking prompt of the question: direct answer required.",84 "question_thinking": "Thinking prompt of the question: think first, then answer.",85 "answer": ["List of components that constitute the answer."],86 "difficulty": "Sample difficulty: Easy, Moderate, Hard or Extreme."87}88```89 90## 🧰 How to use it?91 92### Loading Data93 94You can download and load **LongBench Pro** data using the following code:95 96```python97from datasets import load_dataset98dataset = load_dataset('caskcsg/LongBench-Pro', split='test')99```100 101### Evaluation102 103Please refer to our [Github Repo](https://github.com/caskcsg/longcontext/tree/main/LongBench-Pro) for automated evaluation.104 105## 📖 Citation106 107```108@misc{chen2026longbenchprorealisticcomprehensive,109 title={LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark}, 110 author={Ziyang Chen and Xing Wu and Junlong Jia and Chaochen Gao and Qi Fu and Debing Zhang and Songlin Hu},111 year={2026},112 eprint={2601.02872},113 archivePrefix={arXiv},114 primaryClass={cs.CL},115 url={https://arxiv.org/abs/2601.02872}, 116}117```118