ai-historian/german-newspaper-pages

### 📋 On the licensing of this data Every item here is labelled to the best of my ability. The rights statement is taken per issue from what the holding institution recorded in the Deutsche Digitale Bibliothek, captured at download time — never inferred, and never applied at newspaper level to issues that may differ. Labelling at this scale is imperfect. If you believe anything here is mislabelled, or you hold rights in any of this material, please write to [lorenz.hufe@posteo.de](mailto:lorenz.hufe@posteo.de) — I will resolve it immediately.

## ⚠️ Content notice · Inhaltshinweis This collection contains National Socialist propaganda and other historical material that is antisemitic, racist and otherwise discriminatory. It is published solely as a historical source, for research, teaching and documentation. The corpus includes the German press of 1933–1945 — 187,453 pages from 22 newspapers — comprising state and party propaganda, antisemitic agitation, denaturalisation and deportation notices naming persecuted individuals, and reporting that legitimised persecution and war. Material from the 18th and 19th centuries likewise carries the colonial, antisemitic and racist language of its time, and the OCR text reproduces it verbatim. Reproduction here is documentary and implies no endorsement. The views expressed are those of the historical publications, not of the depositors or the holding institutions. German law permits the use of such material for civic education, research, teaching, art, and reporting on contemporary events or history — the social-adequacy clause of §§ 86, 86a StGB. Users are responsible for lawful use in their own jurisdiction, including any restrictions on reproducing or displaying the symbols and texts contained here. Anyone training generative models on this corpus should expect this material to be reproduced, and should filter accordingly. --- Diese Sammlung enthält nationalsozialistische Propaganda sowie weiteres historisches Material antisemitischen, rassistischen und anderweitig diskriminierenden Inhalts. Sie wird ausschließlich als historische Quelle zu Zwecken der Forschung, der Lehre und der Dokumentation bereitgestellt. Enthalten ist die deutsche Presse der Jahre 1933–1945 (187.453 Seiten aus 22 Zeitungen): Staats- und Parteipropaganda, antisemitische Hetze, Ausbürgerungs- und Deportationsbekanntmachungen mit Namen verfolgter Personen sowie Berichterstattung, die Verfolgung und Krieg legitimierte. Auch das Material des 18. und 19. Jahrhunderts enthält die koloniale, antisemitische und rassistische Sprache seiner Zeit; der OCR-Text gibt sie wörtlich wieder. Die Wiedergabe erfolgt zu dokumentarischen Zwecken und stellt keine Billigung dar. Die geäußerten Auffassungen sind die der historischen Publikationen, nicht die der Bereitstellenden oder der besitzenden Einrichtungen. Die Nutzung ist im Rahmen der Sozialadäquanzklausel der §§ 86, 86a StGB zulässig (staatsbürgerliche Aufklärung, Wissenschaft, Forschung, Lehre, Kunst, Berichterstattung über Vorgänge des Zeitgeschehens oder der Geschichte). Für die Rechtmäßigkeit der Nutzung in der jeweiligen Rechtsordnung sind die Nutzenden selbst verantwortlich.

German Newspaper Pages (Public Domain, 1671–1964)

7,000,060 newspaper page scans from 650 German newspapers, each shipped with the layout boxes and OCR transcription produced for this corpus, and — for 38 % of pages — the digitising library's own ALTO XML.

Every page here comes from an issue whose own rights statement is CC Public Domain Mark 1.0 or CC0 1.0. Sources were harvested from the Deutsche Digitale Bibliothek newspaper portal.

Contents of a sample

Shards are data/pages-NNNNN.tar, ordered chronologically; an issue is never split across two shards. Each page is one WebDataset sample, key <date>_<issuehash>_<pageid>:

import webdataset as wds

ds = wds.WebDataset("data/pages-{00000..02522}.tar").decode("pil")
for s in ds:
    img, meta = s["jpg"], s["json"]
    for r in meta["ocr"]["regions"]:
        x1, y1, x2, y2 = r["bbox"]      # pixels of THIS jpg
        print(r["label"], r["text"][:60])

The pages are in date order, starting mid-corpus

Shard date ranges never overlap. The config lists them as a rotation: reading starts at 1880 — the middle of the corpus, where the pages actually are — runs forward to 1964, then wraps to 1671 and returns to 1880. The order is monotonic in date apart from that one wrap, and the preview opens on representative 19th-century material rather than on the 1670s (936 pages in the entire decade) or the 1960s (4 pages).

first read   data/pages-01100.tar   ≈1880
then         … → data/pages-02522.tar   1944-10-19 → 1964-12-03
wraps to     data/pages-00000.tar   1671-12-26 → 1710-11-17
last  read   data/pages-01099.tar   ≈1880

The tar numbering is strictly chronological (pages-00000.tar oldest), so shard_dates.tsv — not the read order — is the guide for picking a period.

To read a period, take only the shards covering it. shard_dates.tsv in this repo gives shard, date_min, date_max, pages for all 2,523 shards; the decades are:

import csv, tarfile
from huggingface_hub import hf_hub_download

rows = list(csv.DictReader(open(hf_hub_download(
    "ai-historian/german-newspaper-pages", "shard_dates.tsv", repo_type="dataset")), delimiter="\t"))
want = [r["shard"] for r in rows if r["date_min"] <= "1848-12-31" and r["date_max"] >= "1848-01-01"]

The Hub's viewer cannot sort or filter this dataset — it fails to convert WebDataset tars at all (int(f.size) on a streaming handle whose size is None). Use the chronological layout above, or the metadata index, which is fully sortable and filterable online.

Metadata index — browse and filter all 7 million pages

The tars are 7.96 TB, so the Hub cannot preview them usefully. A Parquet metadata index — one row per page, 97 MB — is published as its own dataset so the viewer can serve it in full, and the same files are mirrored here under index/ for offline use.

Browse it online: `ai-historian/german-newspaper-pages-index`

from datasets import load_dataset
idx  = load_dataset("ai-historian/german-newspaper-pages-index", split="train")
hits = idx.filter(lambda r: r["place"] == "Köln" and r["year"] == 1883)

(The index cannot be a second config of this repo: the Hub resolves one builder per repository, so a Parquet config inside a WebDataset repo gets parsed as tar and fails.)

Columns: key, issue_id, page_id, page_number, date, year, month, day, zdb_id, paper_title, place, language, provider_ddb_id, license, has_alto, shard, ddb_url.

shard names the tar holding that page, so a hit leads straight to its image:

from huggingface_hub import hf_hub_download
import tarfile, json
hit = hits[0]
tar = hf_hub_download("ai-historian/german-newspaper-pages", hit["shard"], repo_type="dataset")
with tarfile.open(tar) as t:
    img  = t.extractfile(hit["key"] + ".jpg").read()
    meta = json.loads(t.extractfile(hit["key"] + ".json").read())

⚠️ paper_title drifts — 650 newspapers carry 1,930 title strings because papers were renamed. Group on zdb_id.

Coordinate systems — read this before drawing boxes

• —`.json` boxes are in the coordinates of the `.jpg` in the same sample. They were produced on the full-resolution scan and rescaled by the same factor the image was, so they need no adjustment.
• —`.alto.xml` is verbatim from the library and keeps the ORIGINAL full-resolution coordinates. Its own <Page WIDTH=… HEIGHT=…> header states them, and they equal image.original_width/original_height in the JSON. To overlay ALTO on the shipped JPEG, multiply by image.scale_from_original.

.json schema

{
  "key": "1883-02-14_HR6PLRK…_p0001",
  "issue_id": "1883-02-14_HR6PLRK…", "page_id": "p0001", "page_number": 1,
  "date": "1883-02-14", "year": 1883,
  "zdb_id": "…", "paper_title": "Kölner Nachrichten", "place": "Köln",
  "language": "ger", "provider_ddb_id": "…",
  "license": "http://creativecommons.org/publicdomain/mark/1.0/",
  "ddb_url": "https://www.deutsche-digitale-bibliothek.de/newspaper/item/…?issuepage=1",
  "image": {"width": 1799, "height": 2400,
            "original_width": 4905, "original_height": 6543,
            "scale_from_original": 0.366769},
  "ocr": {
    "layout_model": "PP-DocLayout_plus-L", "text_model": "churro-3B",
    "bbox_space": "image",
    "regions": [
      {"order": 0, "label": "text", "bbox": [820.4, 694.5, 1163.5, 1023.2],
       "score": 0.93, "transcribed": true, "text": "…", "finish_reason": "stop"}
    ]
  },
  "alto": {"present": true, "coordinate_space": "original"}
}

Region labels: text, paragraph_title, image, header, table, doc_title, footer, number, figure_title, aside_text, footnote, seal, abstract, chart, formula, reference, content, and a few rarer ones.

How it was made

1. 1.Harvest — issues fetched from the DDB API, one directory per issue, with the page images, the library ALTO when the API offered one, and a manifest recording the per-issue rights statement at download time.
2. 2.Layout — PP-DocLayout_plus-L on the full-resolution scan; boxes merged so a region contained in another keeps the outer box.
3. 3.Transcription — each region cropped and read by churro-3B (a VLM fine-tuned for historical print). Regions labelled as pictorial, and crops under 8 px, are boxed but not transcribed. The model's XML wrapper is stripped; one line of text per printed line.
4. 4.Packaging — public-domain issues only, images resized to 2400 px long edge.

Limitations — please read

• —Not full resolution. Originals are ~2× larger (median long edge 4,450 px). 2400 px is a working resolution for layout and document-understanding work; for fine Fraktur re-transcription, fetch the original from the ddb_url. The release would be 22 TB at original size.
• —OCR is machine output, not ground truth. The transcription model loops or truncates on a minority of regions (~0.9 % of regions, but touching ~16 % of pages in a 10k-page audit). finish_reason is "length" where a region hit the token cap and "repetition" where the loop detector fired — filter on it if clean text matters.
• —Layout boxes are detections. One printed table or article may fall into several boxes, and classified-ad columns are sometimes read as tables. On a hand-audited quality-filtered sample the table label was 97 % precise; other labels are not separately audited.
• —`aside_text` is effectively never detected (recall ~0 in a small annotation study).
• —ALTO is only on 38 % of pages, is the library's own OCR (usually much older and weaker than the transcription here), and its quality varies by institution. A crawler bug deleted a further ~2.3 M valid ALTO files (they began with a UTF-8 BOM and failed an is-xml check); the bug is fixed but those files were not re-fetched.
• —The corpus is shaped by what is digitised and public domain, not by what was printed. Coverage is heavily west-German and Saxon; Berlin is almost absent before 1871; the 1700–1814 material is about two thirds Hamburg. Do not read page counts per year as publication volume. The handful of post-1950 pages are rights-statement outliers, not coverage.
• —Titles drift. 650 ZDB ids carry 1,930 distinct title strings, because papers were renamed; join on zdb_id, not on paper_title.

Licence

Every issue carries CC Public Domain Mark 1.0 (6,971,082 pages) or CC0 1.0 (28,978 pages), taken from that issue's own rights statement in the DDB, never from a newspaper-level default. The layout boxes and transcriptions added here are released under CC0 1.0.

Please credit the holding institution — provider_ddb_id and ddb_url identify it for each page.

Related

• —`ai-historian/german-newspaper-tables` — 1,044,813 table crops cut from these same layout boxes.

Citation

@misc{german_newspaper_pages_2026,
  title  = {German Newspaper Pages (Public Domain, 1671--1964)},
  author = {Hufe, Lorenz},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/datasets/ai-historian/german-newspaper-pages}}
}

Acknowledgements — thank you to the holding institutions

This corpus exists only because libraries and archives digitised these newspapers, cleared their rights, and published them openly. Thank you to every institution below, and to the Deutsche Digitale Bibliothek and the zeitpunkt.NRW portal for aggregating and serving them.

Nothing here was created by this project except the layout boxes and the OCR text. The page images, the cataloguing, and the rights clearance are theirs.

Counts are pages in this release; 17 DDB provider entries map to the 15 institutions above (SLUB Dresden and MARCHIVUM each appear under two). Every page record carries provider_ddb_id and a ddb_url, so the holding institution is identifiable per page — please credit it when you use or cite individual pages.

Thanks are also due to the readers and cataloguers whose work is invisible here: the newspapers were indexed, described and dated long before any of this could be automated.