HPLT/HPLT2.0_cleaned

NB: HPLT2.0 is now superseded by a newer release:

[HPLT3.0](https://huggingface.co/datasets/HPLT/HPLT3.0)

We recommed switching to v3.0, unless you have a compelling reason to stay on 2.0.

This is a large-scale collection of web-crawled documents in 191 world languages, produced by the HPLT project. The source of the data is mostly Internet Archive with some additions from Common Crawl.

For a detailed description of the dataset, please refer to our website and our pre-print.

The Cleaned variant of HPLT Datasets v2.0

This is the ``cleaned`` variant of the HPLT Datasets v2.0 converted to the Parquet format semi-automatically when being uploaded here. The original JSONL files (which take ~4x fewer disk space than this HF version) and the larger non-cleaned version can be found at https://hplt-project.org/datasets/v2.0.

Dataset Performance

Internal Evaluation

We conducted the FineWeb-style ablation studies within the HPLT project with the focus on one high-resource and one low-resource language: English and Norwegian.

We train 1.7B decoder-only LMs using 100B/30B tokens sampled from the English/Norwegian parts of our HPLT v2 dataset respectively. We replicate the FineWeb corpora comparison design and train the models with a fixed pretraining setup except for the pretraining corpus (English: four corpora; Norwegian: five corpora). Please find the general description of the training and evalutaion setups below and refer to more details in Section 6.2 and Appendix I in our paper.

English

• —Corpora: HPLT v1.2, FineWeb and HPLT v2 (ours; deduplicated and cleaned versions).
• —Pretraining framework and infrastructure: We trained our English models using Megatron-LM on LUMI with 16 nodes, each with 4 AMD MI250x GPUs with dual-GCD (graphics compute die) design, amounting to 8 logical devices. In total, we used 128 devices and a single 64-core CPU for approximately 84 hours, totalling 11,008 GPU hours per model.
• —Evaluation tasks: ARC (Easy and Challenge), Hellaswag, PIQA, and OpenbookQA. We consider only the 0-shot evaluation regime.
• —Evaluation framework: LightEval.
• —Results: See the plot above. Our models trained on the HPLT v2 datasets reach similar performance to the models trained on FineWeb data and considerably outperform the models trained on HPLT v1.2.

Norwegian

• —Corpora: HPLT v1.2, FineWeb-2, mC4, CulturaX, and HPLT v2 (ours).
• —Pretraining framework and infrastructure: We trained our Norwegian models using Megatron-DeepSpeed on LUMI with 32 nodes, each with 4 AMD MI250x GPUs. The full pretraining run of each model took approximately 15 hours (wall-clock time), or 1,920 GPU-hours.
• —Evaluation tasks: NorCommonsenseQA, NorOpenBookQA, NRK-Quiz-QA, NCB, NorIdiom, and NorQuAD. We discarded tasks that provided a low signal based on the monotonicity and non-random performance criteria defined in the FineWeb-2 evaluation design. The resulting tasks were NCB, NRK-Quiz-QA, NorCommonsenseQA, and NorQuAD. We aggregated the performance using the average normalized score. We consider only the 0-shot evaluation regime.
• —Evaluation framework: NorEval, a Norwegian language understanding and generation evaluation benchmark based upon LM Evaluation Harness.
• —Results: See the plot above. The Norwegian models trained on FineWeb, CulturaX, and mC4 perform on par with HPLT v2 and outperform those trained on HPLT v1.2. Performance gains start to level off after 16B tokens, with the FineWeb and HPLT v2 scores being more stable during pretraining. This suggests that CulturaX, FineWeb, and HPLT v2 are more effective corpora for Norwegian, and their mixtures potentially provide further benefits.

External Evaluation

The HuggingFace team has compared the utility of various multilingual corpora for training large language models in their FineWeb2 initiative.

They found that the HPLT v2 datasets are next to their FineWeb-2, on par with the CulturaX dataset as shown in this figure produced by HuggingFace:

<img src="https://huggingface.co/datasets/HuggingFaceFW/admin/resolve/main/multilingualdatasetscomparison.png" width="800" height="800" />

This is a massive improvement compared to the HPLT v1 datasets, as can be seen on the plot above. In fact, it’s even better: if one looks at the language-specific results, it becomes clear that on Arabic, Hindi, Russian, Thai and Turkish (5 out of 9 languages HuggingFace evaluated on), HPLT v2 is on par or better than FineWeb 2. The average score is lower mostly because of Chinese, we expect it to improve a lot in HPLT v3. Note that the source of the FineWeb 2 (and CulturaX) data is exclusively CommonCrawl, while the HPLT datasets are to a large extent composed of Internet Archive crawls. Thus, FineWeb-2 and HPLT v2 are complementary to each other and should be used together.

Languages

The ``cleaned`` version of HPLT Datasets v2.0 consists of subsets corresponding to 191 language codes. Below we provide a list of language codes. For each language code the amount of text is shown as measured in:

• —segments: the number of sequences of characters (possibly empty) separated by the newline symbol,
• —wcwords: the number of words as defined by the Unix ``wc`` utility, i.e. the number of non-whitespaces with a whitespace or the beginning of document before,
• —chars: the number of characters,
• —docs: the number of documents, each document corresponds to an individual web page from the sourcing web crawls.

Terms of Use and Takedown

Terms of Use

These data are released under these Terms of Use:

• —We do not own any of the text from which these text data has been extracted.*
• —We license the actual packaging of these text data under the Creative Commons CC0 license ("no rights reserved").

Notice and take down policy

Notice: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:

• —Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
• —Clearly identify the work claimed to be infringed.
• —Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.
• —You can reach us at hplt-datasets@ufal.mff.cuni.cz

Take down: We will comply to legitimate requests by removing the affected sources from the next release of the corpora.

• —It is your responsibility that any use of the data complies with any applicable legal framework, such as, among others, the EU Copyright Directive 2019/790 and the General Data Protection Regulation 2018, as amended.

Cite us

@inproceedings{burchell-etal-2025-expanded,
    title = "An Expanded Massive Multilingual Dataset for High-Performance Language Technologies ({HPLT})",
    author = {Burchell, Laurie  and
      de Gibert, Ona  and
      Arefyev, Nikolay  and
      Aulamo, Mikko  and
      Ba{\~n}{\'o}n, Marta  and
      Chen, Pinzhen  and
      Fedorova, Mariia  and
      Guillou, Liane  and
      Haddow, Barry  and
      Haji{\v{c}}, Jan  and
      Helcl, Jind{\v{r}}ich  and
      Henriksson, Erik  and
      Klimaszewski, Mateusz  and
      Komulainen, Ville  and
      Kutuzov, Andrey  and
      Kyt{\"o}niemi, Joona  and
      Laippala, Veronika  and
      M{\ae}hlum, Petter  and
      Malik, Bhavitvya  and
      Mehryary, Farrokh  and
      Mikhailov, Vladislav  and
      Moghe, Nikita  and
      Myntti, Amanda  and
      O{'}Brien, Dayy{\'a}n  and
      Oepen, Stephan  and
      Pal, Proyag  and
      Piha, Jousia  and
      Pyysalo, Sampo  and
      Ram{\'i}rez-S{\'a}nchez, Gema  and
      Samuel, David  and
      Stepachev, Pavel  and
      Tiedemann, J{\"o}rg  and
      Vari{\v{s}}, Du{\v{s}}an  and
      Vojt{\v{e}}chov{\'a}, Tereza  and
      Zaragoza-Bernabeu, Jaume},
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.acl-long.854/",
    doi = "10.18653/v1/2025.acl-long.854",
    pages = "17452--17485",
    ISBN = "979-8-89176-251-0",
    abstract = "Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value."
}