CUI03/german-commons

German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models

A comprehensive collection of German-language text data under open licenses for training German language models.

• —Datasheet: DATASHEET.md.
• —Paper: arxiv.org/abs/2510.13996
• —Code: github.com/coral-nlp/llmdata
• —Bloom Filter (DOLMA-compatible): bloom_filter.bin

Dataset Description

This dataset is aggregated from 41 diverse sources and contains 154.56 billion tokens of German text data with 35.78 million documents spanning 7 thematic domains:

• —🌐 Web Commons: 19.89B tokens source from Wiki projects, online discussions, code repositories, social media posts, YouTube transcripts
• —💬 Political Commons: 3.57B tokens sourced from parliamentary documents, speeches, protocols, political vocabulary
• —⚖️ Legal Commons: 2.99B tokens sourced from court decisions, federal law, legal databases, EU legal documents
• —📰 News Commons: 72.67B tokens sourced from historical and current newspapers archives
• —🏦 Economics Commons: 0.11B tokens sourced from EU public tenders
• —📚 Cultural Commons: 54.49B tokens sourced from cultural heritage collections
• —🔬 Scientific Commons: 0.84B tokens sourced from scholarly papers, books, and technical journals

Dataset Features

Each record contains the following fields:

• —id: Unique identifier string, as per each documents' source dataset
• —source: Source dataset name
• —subset: Thematic subset (Cultural, Legal, Political, Scientific, News, Web, Economic)
• —text: Main text content; deduplicated, quality filtered, with consistent formatting and encoding. Can be split at newlines to obtain paragraph text.
• —license: List of applicable licenses for each document, given as canonical SPDX license URL.
• —num_tokens: GPT-2 token count
• —perplexity: Text perplexity measured with a KenLM model trained on German Wikipedia text
• —ocr_score: OCR quality score measured using OCRoscope

Dataset Usage

• —Load the entire dataset

  from datasets import load_dataset
  
  ds = load_dataset("coral-nlp/german-commons")

• —Load a thematic subset

  ds = load_dataset("coral-nlp/german-commons", "cultural")

• —Load individual source datasets

  wikipedia = load_dataset("coral-nlp/german-commons", "web", split="wikipedia")

Supported splits and constituent datasets are:

Citation

If you use this dataset, please cite the correponding paper:

@article{gienapp:2025d,
	title        = {{The German Commons -- 154 Billion Tokens of Openly Licensed Text for German Language Models}},
	author       = {Lukas Gienapp and
                    Christopher Schr\"oder and
                    Stefan Schweter and
                    Christopher Akiki and
                    Ferdinand Schlatt and
                    Arden Zimmermann and
                    Phillipe Gen\^et and
                    Martin Potthast},
	year         = 2025,
	month        = oct,
	journal      = {CoRR},
	volume       = {abs/2510.13996},
	url          = {https://arxiv.org/abs/2510.13996}
}

License

This dataset aggregation and metadata is released under ODC-BY license. Individual documents have their own specific licenses - please check the license field for each record.