epfml/FineWeb2-HQ

FineWeb2-HQ

Dataset summary

FineWeb2-HQ is a high-quality, model-filtered pretraining dataset derived as a subset of **FineWeb2**, spanning 20 languages. It enables around 6x faster pretraining compared to the base dataset. FineWeb2-HQ was created by selecting the top 10% quality documents of FineWeb2 in each language, based on scores assigned by a deep learning classifier trained to identify structured and knowledge-rich samples using **XLM-RoBERTa** embeddings.

<center> <img src="https://huggingface.co/datasets/epfml/FineWeb2-HQ/raw/main/aggscoreplot.svg" style="width: 70%;" /> </center>

Validation was performed by pretraining 1B-parameter LLM models (llama-like architecture) across multiple languages and writing systems (scripts). Evaluations on CMMLU (Chinese) and MMLU (German & French) demonstrate that FineWeb2-HQ matches FineWeb2 performance when trained with 6x fewer tokens, and outperforms it when fully trained. Additionally, improvements were observed across other benchmarks, such as outperforming its English cousins DCLM and FineWeb-Edu.

For more details, see our paper Enhancing Multilingual LLM Pretraining with Model-Based Data Selection.

Key features

• —High-quality selection: Top 10% of FineWeb2 documents by quality
• —Multilingual coverage: 20 languages, ensuring diverse linguistic representation
• —Model-based filtering: Uses an XLM-RoBERTa embedding-based classifier to score documents
• —Enhanced benchmark performance: Surpasses FineWeb2 benchmark performance
• —Fully open: Emphasis on transparency

Languages and subsets

The approach as described in the paper is easy to extend to other languages as well, and we might consider adding new languages to an upcoming version of the present dataset.

We also separately release the computed general-purpose embedding vectors for the the full sets of the original FineWeb2 dataset (not just the HQ subsets), in the respective languages, as they can be useful for other applications beyond quality filtering: FineWeb2-embedded.

Dataset structure

Data fields

Each data entry includes the original FineWeb2 data fields with the addition of:

• —quality_score: quality score obtained by the quality classifier
• —embeddings: array of float arrays containing 768-dimensional XLM-RoBERTa embeddings for every 512 token chunk of the tokenized text

Data instance

{
  "id": "<urn:uuid:f26003c7-6084-4791-b3fe-240eedc37e76>",
  "text": "Plutonium ist einer der gefährlichsten Stoffe der Welt. Es entsteht als hochgiftiges und radioaktives Nebenprodukt der Energiegewinnung in Atomkraftwerken. Wer nur ein Millionstel Gramm – ein kaum staubkorngroßes Teilchen – der Substanz einatmet, kann daran sterben. In der Natur kommt der Stoff nur in geringsten Mengen vor, wird aber künstlich hergestellt, weil man damit Bomben bauen kann. Je nach Reinheitsgrad reichen für eine Atombombe bereits fünf Kilogramm. Bis zum Beginn der achtziger Jahre des letzten Jahrhunderts hatten die Reaktoren weltweit bereits rund 300.000 Kilogramm erbrütet. Jährlich kommen etwa 20.000 Kilo hinzu. Genau dieser Stoff wird zu Land und zu Wasser um den ganzen Erdball herum transportiert. Legendär sind die Castor-Transporte, bei denen unter strengsten Sicherheitsvorkehrungen und entsprechenden Kosten abgebrannte Brennelemente aus deutschen Kernkraftwerken zur Wiederaufbereitung nach La Hague (Frankreich) oder Sellafield (Großbritannien) gebracht werden. Erst vergangenen Mai hat ein Frachter die größte Menge wiederaufbereiteten Mülls aller Zeiten von Frankreich nach Japan gebracht. Nicht auszudenken, was ein Unfall auf See bedeuten würde.",
  "date": "2014-03-16T08:53:38Z",
  "dump": "CC-MAIN-2014-10",
  "embeddings": [[ ... ]],
  "file_path": "s3://commoncrawl/crawl-data/CC-MAIN-2014-10/segments/1394678702159/warc/CC-MAIN-20140313024502-00039-ip-10-183-142-35.ec2.internal.warc.gz",
  "language": "deu",
  "language_score":  0.9983288645744324,
  "language_script": "Latn",
  "minhash_cluster_size": 2,
  "top_langs": {"deu_Latn_score": 0.9983288645744324},
  "url": "http://www.greenpeace.org/austria/de/themen/atom/probleme/atomtransporte/",
  "quality_score": 0.06472613662481308
}

Usage

You can load the dataset in Python using datasets:

from datasets import load_dataset

dataset = load_dataset("epfml/FineWeb2-HQ", "deu_Latn")

Licensing information

Like FineWeb2, this dataset is released under Open Data Commons Attribution License (ODC-By) v1.0 license and is subject to CommonCrawl's Terms of Use.

Dataset origin

Being a subset of FineWeb2, this data covers websites over the 2013-2024 time period.

FineWeb2 is sourced from the internet at large, it is very likely that some personable identifiable information (PII) will be present, even if the FineWeb2 processing has already anonymized email addresses and public IP addresses. If you find your own PII and would like it removed, please fill out the FineWeb2 PII removal/opt out form.

CommonCrawl respects robots.txt at crawl time, but if you are a webmaster and find your website in FineWeb2 and would like to have it removed, you may also use the FineWeb2 PII removal/opt out form.

Considerations for Using the Data

Before using this dataset for training models, we recommend performing additional filtering for sensitive content such as PII or harmful content. For the aspects of social impact, discussion of biases, and known limitations, we also refer to the FineWeb2 documentation.

Citation information

If you use this dataset in your research or applications, please use the following citation:

@article{messmer2025multilingdatacomp,
  title={Enhancing Multilingual LLM Pretraining with Model-Based Data Selection}, 
  author={Bettina Messmer and Vinko Sabolčec and Martin Jaggi},
  journal={arXiv},
  year={2025},
  url={https://arxiv.org/abs/2502.10361}, 
}