LeMoussel/fra-hplt

French HPLT

A French pretraining corpus of 720 405 documents built from HPLT Monolingual v3 high-quality web crawl data. The corpus is streaming-ready and includes per-document metadata: language confidence, URL, quality score, character/word counts, and per-register scores. Every document has an HPLT WDS quality score of 10 or higher (the top of the quality distribution).

Quick Start

from datasets import load_dataset

# Streaming
ds = load_dataset("LeMoussel/fra-hplt", split="train", streaming=True)
for row in ds.take(3):
    print(row["url"], row["text"][:100])


# Full training split (requires disk space)
ds = load_dataset("LeMoussel/fra-hplt", split="train")

Token estimates use words×1.66. Actual count varies by tokenizer.

Dataset Fields

Data Splits

Split ratios are 98% / 1% / 1% over Parquet shards (50,000 rows per shard, 15 shards total).

┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┓ ┃ Split ┃ Documents ┃ Shards ┃ ┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━┩ │ train │ 620,405 │ 13 │ │ validation │ 50,000 │ 1 │ │ test │ 50,000 │ 1 │ └────────────┴───────────┴────────┘

How It Was Built

Source: HPLT v3 sorted shards (https://data.hplt-project.org/three/sorted), which order documents by WDS quality score descending — every document in this corpus has a WDS score of 10 or higher.

Quality filtering (applied inline during download):

• —50–100,000 characters per document
• —Max 50% non-alphabetic characters (Unicode-aware)
• —Min average word length 2.0 characters

Deduplication:

• —HPLT v3 already applies global near-deduplication.
• —MinHash near-duplicate removal (xxhash, similarity threshold 0.85, 128 permutations, 5-grams)

Web Register Distribution

HPLT v3 labels documents with a register/genre code:

┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Register ┃ Count ┃ % ┃ Meaning ┃ ┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ IP │ 131,035 │ 18.19% │ Informational persuasion │ │ NA │ 55,815 │ 7.75% │ Narrative │ │ IN │ 120,755 │ 16.76% │ Informational description │ │ MT │ 310,002 │ 43.03% │ Machine-translated │ │ OP │ 67,251 │ 9.34% │ Opinion │ │ HI │ 20,726 │ 2.88% │ How-to / instructional │ │ LY │ 221 │ 0.03% │ Lyrical │ │ SP │ 9,372 │ 1.30% │ Spoken │ │ ID │ 5,228 │ 0.73% │ Interactive discussion │ └──────────┴─────────┴────────┴───────────────────────────┘

To exclude machine-translated content, filter on the dominant register: ds.filter(lambda x: max(x["web_register"], key=x["web_register"].get) != "MT") (drops ~43.03%). The web_register field stores raw probability scores per register — the dominant register is the key with the highest value.

Source Collection Types

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┓ ┃ Collection ┃ Count ┃ % ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━┩ │ text/html │ 719,874 │ 99.93% │ │ application/xml │ 302 │ 0.04% │ │ application/atom+xml │ 84 │ 0.01% │ │ application/epub+zip │ 61 │ 0.01% │ │ (empty) │ 19 │ 0.00% │ │ application/xhtml+xml │ 18 │ 0.00% │ │ application/vnd.openxmlformats-officedocument.spreadsheetml.sheet │ 12 │ 0.00% │ │ application/rss+xml │ 10 │ 0.00% │ │ application/octet-stream │ 9 │ 0.00% │ │ application/vnd.openxmlformats-officedocument.presentationml.presentation │ 9 │ 0.00% │ │ text/plain │ 7 │ 0.00% │ └───────────────────────────────────────────────────────────────────────────┴─────────┴────────┘

Corpus Statistics Summary

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩ │ Total documents │ 720,405 │ │ Total characters │ 12.0B │ │ Total words │ 1.9B │ │ Estimated tokens │ ~3.2B │ │ Avg doc length (words) │ 2642 │ │ Median doc length (words) │ 2049 │ │ Min doc length (words) │ 361 │ │ Max doc length (words) │ 18,561 │ └───────────────────────────┴─────────┘

Limitations

• —Web text only — no books, Wikipedia, or structured data
• —machine-translated content — identifiable via dominant web_register score on key MT
• —No PII filtering beyond HPLT defaults

Citation

@dataset{frenchHPLT_2026,
  author    = {LeMoussel},
  title     = {French {HPLT}: French Corpus Document},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/LeMoussel/fra-hplt}
}

@article{oepen2025hplt,
  title   = {{HPLT} 3.0: Very Large-Scale Multilingual Resources for {LLM} and {MT}},
  author  = {Oepen, Stephan and others},
  journal = {arXiv preprint arXiv:2511.01066},
  year    = {2025}
}

License

CC0 1.0 Universal — inherited from HPLT v3.