davanstrien/ncse-v2

NCSE v2.0 — OCR-processed 19th-century English newspapers (working mirror)

[!NOTE] Private working mirror, not an original work. Source: *Jonno Bourne, NCSE v2.0: A Dataset of OCR-Processed 19th Century English Newspapers, UCL Research Data Repository, 2025. [doi:10.5522/04/28381610.v1](https://doi.org/10.5522/04/28381610.v1) — CC BY 4.0*. Mirrored here for analysis convenience (parquet-native loading, Dataset Viewer). All credit to the original author.

The Nineteenth Century Serials Edition re-OCR'd with Pixtral 12B, reported at 1% median character error rate — 5× lower than the next best approach (paper, arXiv 2502.14901, code).

1,378,882 text segments across six periodicals, 1806–1890.

Contents

Main schema: issue_id · page_number · block · column · class · reading_order · content · segment_count · prompt_tokens · completion_tokens · total_tokens · box_page_id · page_id

⚠️ Verified structure notes (checked 2026-08-05, not in the original docs)

• —`ground_truth/` pairs with `cropped_images/`, NOT with `data/`. All 358 GT files have a matching crop ({issue}_page_{n}_{box}; the GT filename adds a _box_page_id_ infix). Together they are a standalone OCR eval set: run a model on the crop, score against the transcription.
• —*The GT does not join cleanly to the released text. Only 107 of 358 GT region ids exist in `data/`, and region numbering diverges (e.g. `CLD-1852-04-24_page_6` jumps R7 → R10). So the paper's 1% CER headline cannot be reproduced from this release by comparing published OCR against published GT — the two were evidently produced from different segmentation runs. This is a documentation/alignment gap, not* evidence that the OCR is poor.
• —GT text preserves original line breaks and end-of-line hyphenation (perma-\nnent); data/ content is post-processed and de-hyphenated. Normalise before scoring.

Why mirrored

1. 1.A rare high-quality historic-newspaper corpus — most heritage OCR is far worse than 1% CER, so this is a useful clean reference point against noisy corpora.
2. 2.`ground_truth/` + `cropped_images/` form a real paired eval set (image crop ↔ human transcription) for 19th-c English newsprint — scarce, and directly usable for calibrating OCR-quality metrics.
3. 3.It was published only to a university repository with zero Hub presence, so it is invisible to the ML ecosystem despite being CC BY.

Original page images are held by King's College London (Monthly Repository, Northern Star, Leader, English Woman's Journal, Tomahawk, Publishers' Circular) and are not included here.

Citation

@misc{bourne_ncse_v2,
  author    = {Bourne, Jonno},
  title     = {{NCSE v2.0}: A Dataset of {OCR}-Processed 19th Century English Newspapers},
  year      = {2025},
  publisher = {UCL Research Data Repository},
  doi       = {10.5522/04/28381610.v1}
}