Datasetpublic

nutrientdocs/doc-split-benchmark

Doc-Split Benchmark The evaluation slice for page-stream segmentation — the exact set behind the leaderboard and the cloud-VLM comparison. Self-contained (page images embedded), with a reference scorer so results are reproducible. This is the benchmark, not the training corpus (which stays private). 🏆 Leaderboard: doc-split-leaderboard 🎯 Demo: doc-split-demo 🟢 Model: doc-split-mini-e5 (open weights) 🌍 OpenPSS cuts: openpss-mirror (SHORT/LONG, self-contained)… See the full description on the dataset page: https://huggingface.co/datasets/nutrientdocs/doc-split-benchmark.

sourceHugging Facecc-by-sa-4.0updated 1mo agoView on Hugging Face

0likes52downloads

Dataset Card

Doc-Split Benchmark

The evaluation slice for page-stream segmentation — the exact set behind the leaderboard and the cloud-VLM comparison. Self-contained (page images embedded), with a reference scorer so results are reproducible. This is the benchmark, not the training corpus (which stays private).

—🏆 Leaderboard: doc-split-leaderboard
—🎯 Demo: doc-split-demo
—🟢 Model: doc-split-mini-e5 (open weights)
—🌍 OpenPSS cuts: openpss-mirror (SHORT/LONG, self-contained)

Cuts

Cut	Where	Notes
our-200	this repo (`our200` config)	200 license-clean synthetic-concat streams, ~78% boundary rate
OpenPSS short / long	openpss-mirror	Dutch; sparse- & dense-boundary regimes
TABME++ / Tobacco800	upstream (not redistributed)	reported for reference; licenses/gating prevent re-hosting

Load

python

from datasets import load_dataset
ds = load_dataset("nutrientdocs/doc-split-benchmark", "our200", split="test")
# one row = one page; group by stream_id, order by position.

Schema

Field	Type	Meaning
`stream_id`	string	groups pages into one ordered stream
`position`	int	page index within the stream (0-based)
`boundary`	int	1 = starts a new document (position 0 always 1)
`page_text`	string	page OCR text (may be empty)
`image`	image	page image (512px)
`source`	string	provenance

How to score

Metric: boundary page-F1 (internal positions, page 0 forced) + Cohen's κ, WindowDiff, Pk. The reference scorer ships in this repo:

bash

python score.py --pred your_preds.json      # preds: {stream_id: [0/1 per page]}

score.py uses docsplit.eval.metrics.aggregate (bundled). κ is reported because a degenerate predict-none/all can look fine on F1 but scores κ ≈ 0 on the sparse cuts. See SUBMISSION.md to add a row to the leaderboard.

manifest.json records provenance (per-source stream/page counts, boundary rates, seed).

License & attribution

CC-BY-SA-4.0 for the our-200 packaging (Nutrient-generated synthetic concatenations). OpenPSS cuts are in openpss-mirror under their upstream terms. TABME++/Tobacco800 are referenced, not redistributed.

About the author

This project is maintained and funded by Nutrient - The #1 PDF SDK library for viewing, editing, eSigning, and more.