microsoft/XL-DocBench

<div align="center">

<h1>XL-DocBench</h1>

<p> Evidence-grounded reasoning across hundreds or thousands of pages.<br> Fully verified by 194 human experts. </p>

<p> Hongchen Wei^1,†,‡, Yuanzhe Wang^2,†,‡, Bei Liu^2,*, Yifan Yang², Qi Dai², Ruichun Ma², Kai Qiu², Yunsheng Li²,<br> Dongdong Chen², Chong Luo², Zhenzhong Chen¹, Baining Guo² </p>

<p> ¹Wuhan University &nbsp; ²Microsoft &nbsp; ^†Equal contribution &nbsp; ^‡Work done during an internship at MSRA &nbsp; ^*Project leader </p>

<p> <a href="https://officeintelligence.github.io/xl-docbench/"><b>Project Page</b></a> · <a href="https://arxiv.org/abs/2608.00036"><b>Paper</b></a> · <a href="https://officeintelligence.github.io/xl-docbench/#leaderboard"><b>Live Leaderboard</b></a> </p>

<p> <a href="https://arxiv.org/abs/2608.00036"><img src="https://img.shields.io/badge/arXiv-2608.00036-b31b1b.svg" alt="arXiv"></a> <img src="https://img.shields.io/badge/questions-1,345-3b5b92.svg" alt="1,345 questions"> <img src="https://img.shields.io/badge/documents-292-6a4c93.svg" alt="292 documents"> <img src="https://img.shields.io/badge/humanverified-194experts-2d6a4f.svg" alt="Verified by 194 experts"> </p>

</div>

TL;DR

This is the conservative XL-DocBench release. It contains 1,345 QA rows over 292 documents after removing every question that touches an exact source URL marked RAG: Not Approved.

This release

292 documents · 1,191 single-document QA · 154 cross-document QA · 1,345 total QA

• —data/documents.jsonl: retained document metadata and source URLs.
• —data/qa_single_doc.jsonl: retained single-document questions.
• —data/qa_cross_doc.jsonl: retained cross-document questions.
• —manifest.json: recomputed release statistics.
• —results/scores.jsonl: per-question scores for the 13 reproducible systems.
• —results/summary.json: aggregate scores for the same systems.
• —code/quickstart.py: one-command data and evaluator smoke test.
• —code/evaluate.py: self-contained deterministic evaluator.

The filter uses exact URL matching. This variant addresses exact Not Approved RAG rows only and does not interpret separate Portions Approved entries.

About XL-DocBench

XL-DocBench asks systems to find the evidence, combine all required support, apply the right rule, and know when to abstain. This strict release contains 1,345 expert-verified questions from six professional domains. Among 1,280 records with parseable historical human page annotations, 975 (76.2%) use multiple evidence pages. Released supporting evidence is multimodal for 429 questions (31.9%), 154 questions (11.4%) use cross-document contexts, and 188 require a None answer. Full-series contexts reach 2,935 pages.

Reproducible strict results

All systems below have complete scores for the 1,345 retained IDs and use the hardened evaluator shipped in this release.

Per-question scores

Source documents are referenced by public URLs rather than redistributed. Because some URLs may change or become unavailable over time, we also provide the benchmark scores for every retained question ID. This gives future users a stable comparison point even when a source URL is temporarily unavailable.

• —results/scores.jsonl: one row for each of the 1,345 question IDs.
• —results/summary.json: aggregate Accuracy, Token F1, and ANLS.

Scores are stored on a 0-to-1 scale.

Quick start

Validate the complete release and run a five-question evaluation fixture:

uv run --no-project python code/quickstart.py

Load all three JSONL tables with the included standard-library example:

uv run --no-project python code/examples/load_data.py

Evaluation

Try the bundled five-question example:

uv run --no-project python code/evaluate.py \
  --gold-files code/examples/gold_sample.jsonl \
  --predictions code/examples/predictions_sample.jsonl

For a complete run, provide one prediction per question:

{"question_id": "adubench_single_000001", "prediction": "the biggest single risk to human health worldwide"}
{"question_id": "adubench_cross_000001", "prediction": "macroprudential measures"}

uv run --no-project python code/evaluate.py \
  --predictions predictions.jsonl \
  --output eval_report.json \
  --per-question-csv per_question.csv

The evaluator reports rule-based Accuracy, token-level F1, and ANLS. Missing, failed, and unparsable predictions count as incorrect unless --ignore-missing is enabled.

Record structure

question
├── answer: value + format + verification rule
├── document / documents
│   ├── document_id + public URL
│   ├── evidence_pages: one-based PDF/release page indices
│   └── evidence_items: annotator locator + page references + excerpt kind
└── metadata: domain + difficulty + reasoning type + answerability

evidence_pages locate pages in the released PDF context. Evidence-item pages preserve annotator-supplied page references; printed pagination can differ from PDF indices. The page_numbering and evidence_kind fields make those cases explicit. Row-level unassigned_evidence_items are retained as provenance and are not counted as released supporting evidence.

PDF binaries and local filenames are not included. Source documents remain subject to their original licenses and terms; this release does not grant redistribution rights for third-party PDFs.

Citation

@article{wei2026xldocbench,
  title   = {XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding},
  author  = {Wei, Hongchen and Wang, Yuanzhe and Liu, Bei and Yang, Yifan and Dai, Qi and Ma, Ruichun and Qiu, Kai and Li, Yunsheng and Chen, Dongdong and Luo, Chong and Chen, Zhenzhong and Guo, Baining},
  journal = {arXiv preprint arXiv:2608.00036},
  year    = {2026}
}