histde/dta-documents

Deutsches Textarchiv (DTA) Documents

This datasets hosts all documents from the Deutsches Textarchiv (DTA).

One row per work of the Deutsches Textarchiv (DTA), built from the official DTA dump dta_komplett_2026-02-10 (TCF format). The dataset contains 5,480 documents spanning 1472 to 1987 with about 205M tokens and 1.4B characters of German text, with two text views per document:

• —text: the historical, layout-faithful transcription (line breaks, long s ſ, combining diacritics, original spelling);
• —text_normalized: the same text in modern orthography, produced by applying the DTA CAB orthography corrections (operation="replace") to the token layer.

Rich bibliographic metadata comes directly from the embedded TEI header: authors with GND identifiers, original publication date and place, publisher, holding library and shelfmark, typeface, four genre schemes, DTA subcorpus labels and the per-work license. Licenses vary across the corpus, so please check the license_family column before reuse (see Licensing).

Dataset structure

Configs and splits

There is only one split, because the dataset is a corpus and not a benchmark.

Loading

from datasets import load_dataset

ds = load_dataset("histde/dta-documents", "documents", split="train")
print(ds[0]["title"], ds[0]["year"])

# Only permissively licensed works, as a streaming iterator
permissive = load_dataset("histde/dta-documents", "documents", split="train", streaming=True)
permissive = permissive.filter(lambda d: d["license_family"] in {"cc0-1.0", "cc-by-4.0", "cc-by-3.0"})

With Polars, straight from the Parquet shards:

import polars as pl

df = pl.read_parquet("hf://datasets/histde/dta-documents/documents/train-*.parquet")
df.filter(pl.col("year") < 1700).select("id", "title", "license_family", "num_tokens")

Schema

Example row

{
  "id": "343016",
  "dta_id": "200006",
  "urn": "urn:nbn:de:kobv:b4-200006-1",
  "url": "https://www.deutschestextarchiv.de/343016",
  "title": "Kurtze Anleitung: Wie die jetzige böse Zeit/ darinnen zwar für sich selbst/ nichts/ alß eytel Klag/ Ach/ vnd Weh regieret",
  "authors": [{"surname": "Reideburg", "forename": "Christoph von", "gnd": "http://d-nb.info/gnd/128862580"}],
  "date_published": "1642",
  "year": 1642,
  "place_of_publication": "Breslau",
  "publisher": "Georgius Baumann",
  "repository": "Universitätsbibliothek Breslau",
  "typeface": "Fraktur",
  "genre_dtamain": "Gebrauchsliteratur",
  "genre_dtasub": "Leichenpredigt",
  "dta_corpus_labels": ["ready", "aedit"],
  "language": "deu",
  "language_note": "(Früh-)Neuhochdeutsch",
  "license": "http://creativecommons.org/licenses/by-sa/3.0/de/",
  "license_family": "cc-by-sa-3.0",
  "num_tokens": 11515,
  "num_sentences": 842,
  "text": "Kurtze Anleitung:\nWie die jetzige boͤſe Zeit/ darinnen zwar fuͤr ſich ſelbſt/\nnichts/ alß eytel Klag/ Ach/ vnd Weh regieret; ...",
  "text_source": "layout",
  "text_normalized": "Kurze Anleitung : Wie die jetzige böse Zeit / darinnen zwar für sich selbst / nichts / als eitel Klage / Ach / und Weh regieret ; ..."
}

Dataset creation

Source data

The DTA is a reference corpus of printed German from the late 15th to the early 20th century, curated by the Berlin-Brandenburg Academy of Sciences and Humanities (BBAW). Texts were transcribed by double keying from first editions wherever possible, and are published with TEI headers, sentence and token layers and orthographic normalization from the CAB tool chain.

This dataset was built from the complete TCF dump dta_komplett_2026-02-10, which comes in two variants:

• —simple/ (5,107 files): CMDI metadata and the layout-faithful historical text (<text> layer);
• —full/ (5,471 files): CMDI metadata and the annotation layers (tokens, sentences, orthography).

The union of both variants gives 5,481 works. One work (ford_pitty_1633) is excluded because its TCF file is not well-formed XML.

Processing

• —text comes from simple/ where available (text_source = "layout", 5,105 docs). For the 375 works only present in full/, it is reconstructed by space-joining the token layer (text_source = "tokens"), so line breaks and the original whitespace are lost there.
• —text_normalized is the full/ token layer with CAB replace corrections applied (multi-token corrections replace the whole span), space-joined. It is null for the 10 works only present in simple/.
• —Metadata is deliberately kept un-flattened: authors are a list of structs, all four genre schemes are separate columns, the raw license URL is kept next to a normalized license_family.
• —num_tokens, num_types, num_characters, num_images and num_pages are the extent measures from the DTA header, not recomputed values.

The conversion script and the statistics script are available in the dta-experiments repository.

Statistics

All tables below were produced with dta_dataset_stats.py from the released Parquet shards.

Overview

Temporal distribution (50-year bins)

Text source

Language

Genre (DWDS main)

<details> <summary>Genre (DWDS sub, top 10)</summary>

</details>

<details> <summary>Genre (DTA main)</summary>

</details>

<details> <summary>Genre (DTA sub, top 10)</summary>

</details>

Typeface

Values are taken verbatim from the headers, including the two typos (Antqiua, Frakur).

<details> <summary>DTA corpus labels (a document can carry several)</summary>

The core label marks the curated DTA core corpus (1,478 works, 63.6% of all tokens), which is the balanced selection by genre and period. The other labels denote extension corpora (DTAE) contributed by partner projects.

</details>

<details> <summary>Place of publication (top 10)</summary>

</details>

Most frequent authors (top 10)

Licensing

The DTA publishes every work under its own license, and this dataset keeps that information per row in the license (raw URL from the header) and license_family (normalized) columns. There is no single license for the whole dataset. Note that only cc0-1.0 is a public domain dedication; all cc-by-* licenses carry binding conditions (attribution, share-alike, or non-commercial use), and about 28% of the documents are restricted to non-commercial use (cc-by-nc-3.0, cc-by-nc-sa-4.0, noc-nc-1.0, out-of-copyright-nc).

To select a subset that fits your use case:

ds_sa = ds.filter(lambda d: d["license_family"] in {"cc0-1.0", "cc-by-4.0", "cc-by-3.0", "cc-by-sa-4.0", "cc-by-sa-3.0", "cc-by-sa-2.0"})

Considerations for using the data

• —Coverage is uneven. Roughly 60% of the tokens date from 1750 to 1899; the 15th and 16th centuries are thinly covered. Newspapers (Zeitung) dominate the document count but make up only 8% of the tokens, and a single project (Alexander von Humboldt's writings) contributes 183 short documents.
• —Historical content. The texts reflect the views, language and stereotypes of their time, including religious, political and colonial-era writing. They are provided for research on historical German and should not be taken as statements of fact or acceptable opinion.
• —Normalization is automatic. text_normalized comes from the CAB tool chain and is not manually corrected. Expect residual errors, especially for very early texts, Latin passages, and proper names. The normalized text is also whitespace-tokenized (punctuation is separated by spaces), whereas text keeps the original layout.
• —`text` is not uniform. 375 documents have their historical text reconstructed from the token layer and lack line breaks; check text_source if layout matters to you.
• —Metadata quirks. Genre fields are null for a large share of the works in the DTA scheme (use the DWDS scheme for complete main-genre coverage), typeface strings contain a few typos, and year is parsed heuristically from the free-text date_published.

Citation

Please cite the DTA when using this dataset:

@misc{dta2026,
  author       = {{Berlin-Brandenburgische Akademie der Wissenschaften}},
  title        = {Deutsches Textarchiv. Grundlage für ein Referenzkorpus der neuhochdeutschen Sprache},
  year         = {2026},
  address      = {Berlin},
  howpublished = {Herausgegeben von der Berlin-Brandenburgischen Akademie der Wissenschaften},
  url          = {https://www.deutschestextarchiv.de/}
}

Acknowledgements

A big thank you to the whole team of the Deutsches Textarchiv at the Berlin-Brandenburg Academy of Sciences and Humanities! Also many thanks to all the partner projects, libraries and archives that contributed texts to the DTA extension corpora.

This dataset only converts an official dump into Parquet for convenient use with the Hugging Face ecosystem.

AI disclosure

This dataset card was drafted with Claude Fable 5 (claude-fable-5) based on the statistics produced by dta_dataset_stats.py, and reviewed by the dataset author. The conversion and statistics scripts carry their own AI disclosure blocks, following the rules in the ai-disclosure repository.