biglam/europeana-newspapers-ground-truth

Europeana Newspapers — Historical Newspapers Ground Truth

50 pages of digitised historical German newspapers from the Berlin State Library (Staatsbibliothek zu Berlin), with PAGE XML ground truth produced for the EU Europeana Newspapers project.

Each row pairs three things: the page image, the human-corrected ground truth (regions, polygons, reading order, text), and the ALTO OCR output that ABBYY FineReader actually produced. That last column is what makes this an OCR evaluation set rather than just layout training data.

The gap is stark. Ground truth for one heading reads:

Der Anzeiger.

The OCR for the same region reads:

D c r An; c t g k r.

This is Fraktur, and it is hard.

Fields

• —image — the page scan (bitonal TIF, as deposited)
• —text — full ground-truth text, concatenated in region order
• —ocr_text — the ALTO OCR output, flattened to plain text
• —regions — list of {id, kind, type, text, bbox, polygon, n_lines}
• —reading_order — ordered region ids from the PAGE ReadingOrder block
• —page_id, image_filename, width, height

3,199 regions across the 50 pages: 2,458 TextRegion (typed paragraph, heading, page-number, header, caption, TOC-entry, drop-capital), 678 SeparatorRegion, 41 ImageRegion, 19 GraphicRegion, 3 TableRegion.

Caveats

• —The ground truth is REGION-level, not line-level. 0 of 50 pages carry TextLine elements — n_lines is 0 throughout. If you need line-level HTR ground truth, this is not that dataset. It is layout + region-text GT.
• —50 pages. This is an evaluation set, not a training corpus.
• —Images are bitonal (1-bit) TIFs. Greyscale/colour originals are not in the deposit.
• —Text is 18th–19th-century German in Fraktur, with historical orthography and long-s.

Load

from datasets import load_dataset

ds = load_dataset("biglam/europeana-newspapers-ground-truth", split="train")

# the point of this dataset: corrected GT against what the OCR actually produced
row = ds[0]
print(row["text"][:200])       # human-corrected ground truth
print(row["ocr_text"][:200])   # ABBYY FineReader output for the same page

Source & credit

Clemens Neudecker. Historical Newspapers Ground Truth. Zenodo, 2019-03-05. <https://zenodo.org/records/2583866> — CC0-1.0.

Produced in the EU ICT-PSP project Europeana Newspapers (<http://www.europeana-newspapers.eu/>). Images from the Berlin State Library.

This repository parses the PAGE XML and ALTO into Parquet. Please cite the Zenodo deposit.

@dataset{neudecker_2019_historical_newspapers_gt,
  author    = {Neudecker, Clemens},
  title     = {{Historical Newspapers Ground Truth}},
  year      = {2019},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.2583866}
}