abhishekjha1008/historical-reading-order-benchmark

Detection is easy, reading order is the tax

A multilingual benchmark of line detection and reading order on historical documents

Author: Abhishek Jha (GitHub · Hugging Face) Code, paper & harness: https://github.com/AbhiPandit1/historical-reading-order-benchmark Interactive demo: https://huggingface.co/spaces/abhishekjha1008/historical-reading-order-demo

The finding in one paragraph

On historical documents, finding the text lines on a page is close to solved; deciding the order to read them in is not — and it only bites once the layout has more than one column. This benchmark scores segmentation systems on two separate questions — did you find the lines? (detection F at IoU ≥ 0.5) and did you read them in the right order? (Kendall τ vs the ground-truth document order, on matched lines) — across eight languages and roughly four centuries.

The tax is refundable

A fast box detector loses on reading order only because of the naive top-to-bottom sort. Holding detection fixed and adding a trivial column-clustering ordering stage lifts multi-column τ from 0.50 → 0.99 (95% CI [0.98, 1.00]) — statistically indistinguishable from the heavy layout-aware segmenter, at ~9× the speed. Downstream, assuming perfect line recognition, naive ordering inflicts a median 82% word error on multi-column pages; the ordering stage cuts it to 2%.

No off-the-shelf system is robust across layouts

Transkribus nails handwriting but scrambles printed columns; PP-Structure does the mirror opposite; a learned reading-order model (LayoutReader, trained on modern documents) fails everywhere. Each off-the-shelf system carries the blind spot of its training domain — only a fast detector plus an explicit ordering stage holds across all of them. The finding is era-invariant (a 17th-century Swedish court hand and a 20th-century English print behave the same) and holds under scale-up: English ×8 and Swedish ×28, each across two independent archives.

What's in the dataset

bench/                40 scored pages + 5 excluded German pages (.jpg)
bench/gt/             ALTO / PAGE-XML ground truth (line boxes + document order)
bench/pred/           predictions for every system (kraken, yolo, surya, doctr,
                      tesseract, transkribus, ppstructure, layoutreader)
eng_scale/, scale_eval/  the English (×8) and Swedish (×28) scale-up pages + preds
figures/              comparison figures (incl. the ordering-control result)
harness/              scoring library + per-system runners + score.py
experiments/          ordering-control, downstream-WER, bootstrap CIs, and the
                      Azure job scripts for PP-Structure and LayoutReader
paper/paper.pdf       full write-up (8 systems, CIs, scaling, related work)
manifest.json         per-page language, era, layout, source

Reproduce

git clone https://huggingface.co/datasets/abhishekjha1008/historical-reading-order-benchmark
cd historical-reading-order-benchmark
pip install -r requirements.txt
python harness/score.py                 # detection F + reading-order tau tables
python experiments/exp1_run.py          # ordering-control: naive vs column-clustering vs kraken
python experiments/exp2_run.py          # downstream order-induced word error

Honesty notes

• —Reading order is scored only where detection succeeded (validity guard); τ on a handful of lines is never reported.
• —German is excluded from scoring (under-annotated ground truth).
• —The downstream analysis assumes perfect line recognition to isolate ordering.
• —No private or restricted material — openly licensed FONDUE / HTR-United and public Riksarkivet data. Provenance per language in data_note/SOURCES.md.

Citation

@misc{jha2026readingorder,
  title  = {Detection is easy, reading order is the tax: a multilingual benchmark
            of line detection and reading order on historical documents},
  author = {Jha, Abhishek},
  year   = {2026},
  howpublished = {\url{https://github.com/AbhiPandit1/historical-reading-order-benchmark}}
}

MIT licensed. Contributions and collaboration are welcome.