ashtok897/european-hplt-v1

European HPLT v1

A multilingual pretraining corpus of 45,031,396 documents (~50.9B estimated tokens, ~190 GB raw JSONL) across 41 European languages, built from HPLT Monolingual v3 high-quality web crawl data.

The corpus spans Germanic, Romance, Slavic, Celtic, Baltic, Finno-Ugric, Greek, and other European language families. Every document has an HPLT WDS quality score of 10 or higher (the top of the quality distribution).

Quick Start

from datasets import load_dataset

# Streaming (recommended — corpus is ~190 GB)
ds = load_dataset("ashtok897/european-hplt-v1", split="train", streaming=True)
for row in ds.take(3):
    print(row["lang"], row["text"][:100])

# Filter by language
de_ds = ds.filter(lambda x: x["lang"] == "de")

# Full training split (requires disk space)
ds = load_dataset("ashtok897/european-hplt-v1", split="train")

Language Distribution

Token estimates use chars÷4. Actual count varies by tokenizer.

Notable observations

• —Russian has the highest average doc length (1,847 words) — Russian web articles in HPLT v3 tend to be substantially longer than other languages.
• —Icelandic has unusually long average doc length (1,073 words) for its size tier, similar to Tier 1 languages.
• —German has the highest total token share among Latin-script languages (~5.23B) despite ranking 3rd by doc count.
• —Estonian (et / est_Latn) was targeted but is not present in HPLT v3 sorted collection and was excluded.

Dataset Fields

Data Splits

Split ratios are 98% / 1% / 1% over Parquet shards (250,000 rows per shard, ~180 shards total).

Note on Data Completeness

One shard (train-00107-of-00181.parquet) was removed from the published train split because Hugging Face's ClamAV scanner flagged a false-positive malware signature (Email.Trojan.Trojan-939) on email-like text within the crawled web content. Parquet files contain only data and cannot execute code — the flag was a byte-pattern match against text in the text column, not executable malware. As a result, the train split contains 176 shards (~44.0M documents) rather than

1. 1.Remaining file names keep their original of-00181 suffix for consistency; the datasets loader globs all available shards, so this does not affect loading.

How It Was Built

Source: HPLT v3 sorted shards (https://data.hplt-project.org/three/sorted), which order documents by WDS quality score descending — every document in this corpus has a WDS score of 10 or higher.

Quality filtering (applied inline during download):

• —50–100,000 characters per document
• —Max 50% non-alphabetic characters (Unicode-aware)
• —Min average word length 2.0 characters

Deduplication:

• —Exact SHA-256 hash dedup on all languages
• —MinHash near-duplicate removal disabled for this release — HPLT v3 already applies global near-deduplication, so re-running MinHash on Indo-European languages adds significant compute for marginal gain. May be re-enabled in a future release.

Merge: Languages are interleaved according to configured fractions targeting a balanced multilingual corpus. The merge.py step uses an adaptive quota algorithm that:

• —Caps each language at its available document count
• —Auto-excludes languages with less than 10% of their target quota (none were excluded in this release except Estonian, which had 0 data)
• —Redistributes excluded quota across the remaining languages
• —Shuffles the final output (deterministic, seed=42)

Pipeline code: github.com/ashtok/build-multilingual-corpus

Web Register Distribution

HPLT v3 labels documents with a register/genre code:

To exclude machine-translated content, filter on web-register != "MT" (drops ~8.5%).

Source Collection Types

99.8% of documents are HTML.

Corpus Statistics Summary

Limitations

• —Web text only — no books, Wikipedia, or structured data
• —~8.5% machine-translated content — identifiable via web-register: MT
• —No Estonian — language unavailable in HPLT v3 sorted collection at corpus build time
• —No MinHash near-dedup — exact dedup only (HPLT v3 already applies global near-dedup)
• —Tier-6 languages are data-limited — Occitan, Scottish Gaelic, Faroese, Asturian have <250K docs each
• —Russian has only 1.05M docs — substantially less than other Tier-1 languages because HPLT v3 sorted shards exhausted at this quality level
• —No PII filtering beyond HPLT defaults
• —Language proportions are partially balanced, not natural — high-resource languages are slightly down-weighted versus their raw web share to give more space to mid/low-resource languages

Comparison with Related Corpora

This corpus trades scale for quality: every document is pre-filtered to the top WDS tier and the language mix is tilted to include lower-resource European languages that are underrepresented in other multilingual corpora.

Citation

@dataset{europeanHPLTv1_2026,
  author    = {Mahajan, Ashutosh},
  title     = {European {HPLT} v1: A 45M-document Multilingual Corpus for 41 European Languages},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/ashtok897/european-hplt-v1}
}

@article{oepen2025hplt,
  title   = {{HPLT} 3.0: Very Large-Scale Multilingual Resources for {LLM} and {MT}},
  author  = {Oepen, Stephan and others},
  journal = {arXiv preprint arXiv:2511.01066},
  year    = {2025}
}

License

CC0 1.0 Universal — inherited from HPLT v3.