microsoft/XL-DocBench

1---2language:3- en4license: other5pretty_name: XL-DocBench6task_categories:7- question-answering8tags:9- document-understanding10- long-context11- multimodal-document-ai12- cross-document-qa13- benchmark14size_categories:15- 1K<n<10K16configs:17- config_name: single_doc18  default: true19  data_files:20  - split: test21    path: data/qa_single_doc.jsonl22- config_name: cross_doc23  data_files:24  - split: test25    path: data/qa_cross_doc.jsonl26- config_name: documents27  data_files:28  - split: documents29    path: data/documents.jsonl30- config_name: scores31  data_files:32  - split: results33    path: results/scores.jsonl34---35 36<div align="center">37 38<h1>XL-DocBench</h1>39 40<p>41  Evidence-grounded reasoning across hundreds or thousands of pages.<br>42  Fully verified by 194 human experts.43</p>44 45<p>46  Hongchen Wei<sup>1,†,‡</sup>, Yuanzhe Wang<sup>2,†,‡</sup>,47  Bei Liu<sup>2,*</sup>, Yifan Yang<sup>2</sup>, Qi Dai<sup>2</sup>,48  Ruichun Ma<sup>2</sup>, Kai Qiu<sup>2</sup>, Yunsheng Li<sup>2</sup>,<br>49  Dongdong Chen<sup>2</sup>, Chong Luo<sup>2</sup>,50  Zhenzhong Chen<sup>1</sup>, Baining Guo<sup>2</sup>51</p>52 53<p>54  <sup>1</sup>Wuhan University &nbsp; <sup>2</sup>Microsoft &nbsp;55  <sup>†</sup>Equal contribution &nbsp; <sup>‡</sup>Work done during an internship at MSRA &nbsp;56  <sup>*</sup>Project leader57</p>58 59<p>60  <a href="https://officeintelligence.github.io/xl-docbench/"><b>Project Page</b></a> ·61  <a href="https://arxiv.org/abs/2608.00036"><b>Paper</b></a> ·62  <a href="https://officeintelligence.github.io/xl-docbench/#leaderboard"><b>Live Leaderboard</b></a>63</p>64 65<p>66  <a href="https://arxiv.org/abs/2608.00036"><img src="https://img.shields.io/badge/arXiv-2608.00036-b31b1b.svg" alt="arXiv"></a>67  <img src="https://img.shields.io/badge/questions-1,345-3b5b92.svg" alt="1,345 questions">68  <img src="https://img.shields.io/badge/documents-292-6a4c93.svg" alt="292 documents">69  <img src="https://img.shields.io/badge/human_verified-194_experts-2d6a4f.svg" alt="Verified by 194 experts">70</p>71 72</div>73 74## TL;DR75 76This is the **conservative XL-DocBench release**. It contains77**1,345 QA rows over 292 documents** after78removing every question that touches an exact source URL marked79`RAG: Not Approved`.80 81## This release82 83**292 documents** · **1,191 single-document QA** ·84**154 cross-document QA** · **1,345 total QA**85 86- `data/documents.jsonl`: retained document metadata and source URLs.87- `data/qa_single_doc.jsonl`: retained single-document questions.88- `data/qa_cross_doc.jsonl`: retained cross-document questions.89- `manifest.json`: recomputed release statistics.90- `results/scores.jsonl`: per-question scores for the 13 reproducible systems.91- `results/summary.json`: aggregate scores for the same systems.92- `code/quickstart.py`: one-command data and evaluator smoke test.93- `code/evaluate.py`: self-contained deterministic evaluator.94 95The filter uses exact URL matching. This variant addresses exact `Not Approved`96RAG rows only and does not interpret separate `Portions Approved` entries.97 98## About XL-DocBench99 100XL-DocBench asks systems to **find the evidence, combine all required support,101apply the right rule, and know when to abstain**. This strict release contains1021,345 expert-verified questions from six professional domains. Among 1,280103records with parseable historical human page annotations, 975 (76.2%) use104multiple evidence pages. Released supporting evidence is multimodal for 429105questions (31.9%), 154 questions (11.4%) use cross-document contexts, and 188106require a `None` answer. Full-series contexts reach 2,935 pages.107 108## Reproducible strict results109 110All systems below have complete scores for the 1,345 retained IDs and use the111hardened evaluator shipped in this release.112 113| Rank | System | Input | Accuracy ↑ | Token F1 ↑ | ANLS ↑ |114|---:|---|:---:|---:|---:|---:|115| 1 | **GPT-5.4** | OCR | **38.36** | **39.70** | **34.17** |116| 2 | SimpleDoc + GPT-5.4 | Agent | 36.21 | 33.62 | 24.30 |117| 3 | Kimi-K2.5 | OCR | 36.06 | 38.24 | 32.64 |118| 4 | MDocAgent + GPT-5.4 | Agent | 31.82 | 30.58 | 22.64 |119| 5 | GPT-5.2 | OCR | 31.15 | 33.69 | 28.51 |120| 6 | DeepSeek-V3.2 | OCR | 29.67 | 32.99 | 28.91 |121| 7 | Qwen3.5-4B | OCR | 29.00 | 32.23 | 28.13 |122| 8 | DeepRead + GPT-5.4 | Agent | 27.73 | 26.99 | 21.17 |123| 9 | GPT-5.4 | Img | 26.77 | 29.76 | 26.33 |124| 10 | Kimi-K2.5 | Img | 26.02 | 27.59 | 23.04 |125| 11 | GPT-5.2 | Img | 21.71 | 24.95 | 21.92 |126| 12 | Qwen3.5-4B | Img | 20.52 | 20.86 | 18.26 |127| 13 | Qwen3.5-9B | Img | 20.22 | 22.69 | 20.18 |128 129## Per-question scores130 131Source documents are referenced by public URLs rather than redistributed.132Because some URLs may change or become unavailable over time, we also provide133the benchmark scores for every retained question ID. This gives future users a134stable comparison point even when a source URL is temporarily unavailable.135 136- `results/scores.jsonl`: one row for each of the 1,345 question IDs.137- `results/summary.json`: aggregate Accuracy, Token F1, and ANLS.138 139Scores are stored on a 0-to-1 scale. 140 141## Quick start142 143Validate the complete release and run a five-question evaluation fixture:144 145```bash146uv run --no-project python code/quickstart.py147```148 149Load all three JSONL tables with the included standard-library example:150 151```bash152uv run --no-project python code/examples/load_data.py153```154 155## Evaluation156 157Try the bundled five-question example:158 159```bash160uv run --no-project python code/evaluate.py \161  --gold-files code/examples/gold_sample.jsonl \162  --predictions code/examples/predictions_sample.jsonl163```164 165For a complete run, provide one prediction per question:166 167```jsonl168{"question_id": "adubench_single_000001", "prediction": "the biggest single risk to human health worldwide"}169{"question_id": "adubench_cross_000001", "prediction": "macroprudential measures"}170```171 172```bash173uv run --no-project python code/evaluate.py \174  --predictions predictions.jsonl \175  --output eval_report.json \176  --per-question-csv per_question.csv177```178 179The evaluator reports rule-based **Accuracy**, token-level **F1**, and **ANLS**.180Missing, failed, and unparsable predictions count as incorrect unless181`--ignore-missing` is enabled.182 183## Record structure184 185```text186question187├── answer: value + format + verification rule188├── document / documents189│   ├── document_id + public URL190│   ├── evidence_pages: one-based PDF/release page indices191│   └── evidence_items: annotator locator + page references + excerpt kind192└── metadata: domain + difficulty + reasoning type + answerability193```194 195`evidence_pages` locate pages in the released PDF context. Evidence-item196`pages` preserve annotator-supplied page references; printed pagination can197differ from PDF indices. The `page_numbering` and `evidence_kind` fields make198those cases explicit. Row-level `unassigned_evidence_items` are retained as199provenance and are not counted as released supporting evidence.200 201PDF binaries and local filenames are not included. Source documents remain202subject to their original licenses and terms; this release does not grant203redistribution rights for third-party PDFs.204 205## Citation206 207```bibtex208@article{wei2026xldocbench,209  title   = {XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding},210  author  = {Wei, Hongchen and Wang, Yuanzhe and Liu, Bei and Yang, Yifan and Dai, Qi and Ma, Ruichun and Qiu, Kai and Li, Yunsheng and Chen, Dongdong and Luo, Chong and Chen, Zhenzhong and Guo, Baining},211  journal = {arXiv preprint arXiv:2608.00036},212  year    = {2026}213}214```215