epfml/FineWeb2-embedded

1---2task_categories:3- text-generation4language:5- ru6- zh7- de8- ja9- es10- fr11- it12- pt13- pl14- nl15- id16- tr17- cs18- vi19- sv20- fa21- ar22- el23- da24- hu25pretty_name: FineWeb2-embedded26configs:27- config_name: rus_Cyrl28  data_files:29  - split: train30    path: rus_Cyrl/*31- config_name: cmn_Hani32  data_files:33  - split: train34    path: cmn_Hani/*35- config_name: deu_Latn36  data_files:37  - split: train38    path: deu_Latn/*39- config_name: jpn_Jpan40  data_files:41  - split: train42    path: jpn_Jpan/*43- config_name: spa_Latn44  data_files:45  - split: train46    path: spa_Latn/*47- config_name: fra_Latn48  data_files:49  - split: train50    path: fra_Latn/*51- config_name: ita_Latn52  data_files:53  - split: train54    path: ita_Latn/*55- config_name: por_Latn56  data_files:57  - split: train58    path: por_Latn/*59- config_name: pol_Latn60  data_files:61  - split: train62    path: pol_Latn/*63- config_name: nld_Latn64  data_files:65  - split: train66    path: nld_Latn/*67- config_name: ind_Latn68  data_files:69  - split: train70    path: ind_Latn/*71- config_name: tur_Latn72  data_files:73  - split: train74    path: tur_Latn/*75- config_name: ces_Latn76  data_files:77  - split: train78    path: ces_Latn/*79- config_name: vie_Latn80  data_files:81  - split: train82    path: vie_Latn/*83- config_name: swe_Latn84  data_files:85  - split: train86    path: swe_Latn/*87- config_name: fas_Arab88  data_files:89  - split: train90    path: fas_Arab/*91- config_name: arb_Arab92  data_files:93  - split: train94    path: arb_Arab/*95- config_name: ell_Grek96  data_files:97  - split: train98    path: ell_Grek/*99- config_name: dan_Latn100  data_files:101  - split: train102    path: dan_Latn/*103- config_name: hun_Latn104  data_files:105  - split: train106    path: hun_Latn/*107license: odc-by108size_categories:109- 1B<n<10B110---111# FineWeb2-embedded112 113## Dataset summary114 115FineWeb2-embedded is an extension of the [**FineWeb2**](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) dataset, annotated with **document-level** [**XLM-RoBERTa**](https://huggingface.co/FacebookAI/xlm-roberta-base) **embeddings** for **20 languages**, making the dataset **useful for a variety of tasks**, including document clustering, filtering, and other multilingual research.116 117Since XLM-RoBERTa has a sequence length limit of 512 tokens, each document's **embeddings are obtained by mean-pooling 512 token chunks of the XLM-RoBERTa output**. Therefore, longer texts have more embeddings available (one per 512 tokens).118 119The embeddings were initially computed as part of our [**FineWeb2-HQ**](https://huggingface.co/datasets/epfml/FineWeb2-HQ) dataset (a high-quality subset of FineWeb2). However, we believe that they can be useful for other multilingual research and applications.120 121For more details, see our paper [Enhancing Multilingual LLM Pretraining with Model-Based Data Selection](https://arxiv.org/abs/2502.10361).122 123## Languages and subsets124 125|Subset name|Language name|Number of documents|Disk size|126|----------|-----------------|------------:|----------:|127| rus_Cyrl | Russian         | 605,468,615 | 5.3T      |128| cmn_Hani | Chinese         | 578,332,129 | 4.4T      |129| deu_Latn | German          | 427,700,394 | 2.5T      |130| spa_Latn | Spanish         | 405,634,303 | 2.3T      |131| jpn_Jpan | Japanese        | 376,134,745 | 2.4T      |132| fra_Latn | French          | 332,646,715 | 2.0T      |133| ita_Latn | Italian         | 219,117,921 | 1.3T      |134| por_Latn | Portuguese      | 189,851,449 | 1.1T      |135| pol_Latn | Polish          | 138,337,436 | 794G      |136| nld_Latn | Dutch           | 133,855,612 | 720G      |137| ind_Latn | Indonesian      |  92,992,647 | 537G      |138| tur_Latn | Turkish         |  88,769,907 | 487G      |139| ces_Latn | Czech           |  62,703,458 | 390G      |140| arb_Arab | Arabic          |  57,752,149 | 363G      |141| fas_Arab | Persian         |  51,043,666 | 322G      |142| hun_Latn | Hungarian       |  46,879,826 | 328G      |143| swe_Latn | Swedish         |  45,329,979 | 261G      |144| ell_Grek | Greek           |  44,202,550 | 267G      |145| dan_Latn | Danish          |  42,975,661 | 262G      |146| vie_Latn | Vietnamese      |  40,741,340 | 298G      |147 148We might consider adding new languages supported by the XLM-RoBERTa model to an upcoming version of the present dataset.149 150## Dataset structure151 152### Data fields153 154Each data entry includes the original [FineWeb2 data fields](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2#data-fields) with the addition of:155- `embeddings`: array of float arrays containing 768-dimensional XLM-RoBERTa embeddings for every 512 token chunk of the tokenized text156 157 158### Data instance159 160```json161{162  "id": "<urn:uuid:f26003c7-6084-4791-b3fe-240eedc37e76>",163  "text": "Plutonium ist einer der gefährlichsten Stoffe der Welt. Es entsteht als hochgiftiges und radioaktives Nebenprodukt der Energiegewinnung in Atomkraftwerken. Wer nur ein Millionstel Gramm – ein kaum staubkorngroßes Teilchen – der Substanz einatmet, kann daran sterben. In der Natur kommt der Stoff nur in geringsten Mengen vor, wird aber künstlich hergestellt, weil man damit Bomben bauen kann. Je nach Reinheitsgrad reichen für eine Atombombe bereits fünf Kilogramm. Bis zum Beginn der achtziger Jahre des letzten Jahrhunderts hatten die Reaktoren weltweit bereits rund 300.000 Kilogramm erbrütet. Jährlich kommen etwa 20.000 Kilo hinzu. Genau dieser Stoff wird zu Land und zu Wasser um den ganzen Erdball herum transportiert. Legendär sind die Castor-Transporte, bei denen unter strengsten Sicherheitsvorkehrungen und entsprechenden Kosten abgebrannte Brennelemente aus deutschen Kernkraftwerken zur Wiederaufbereitung nach La Hague (Frankreich) oder Sellafield (Großbritannien) gebracht werden. Erst vergangenen Mai hat ein Frachter die größte Menge wiederaufbereiteten Mülls aller Zeiten von Frankreich nach Japan gebracht. Nicht auszudenken, was ein Unfall auf See bedeuten würde.",164  "date": "2014-03-16T08:53:38Z",165  "dump": "CC-MAIN-2014-10",166  "embeddings": [[ ... ]],167  "file_path": "s3://commoncrawl/crawl-data/CC-MAIN-2014-10/segments/1394678702159/warc/CC-MAIN-20140313024502-00039-ip-10-183-142-35.ec2.internal.warc.gz",168  "language": "deu",169  "language_score":  0.9983288645744324,170  "language_script": "Latn",171  "minhash_cluster_size": 2,172  "top_langs": {"deu_Latn_score": 0.9983288645744324},173  "url": "http://www.greenpeace.org/austria/de/themen/atom/probleme/atomtransporte/",174}175```176 177## Usage178 179You can load the dataset in Python using `datasets`:180 181```python182from datasets import load_dataset183 184dataset = load_dataset("epfml/FineWeb2-embedded", "deu_Latn")185```186 187## Licensing information188 189Like FineWeb2, this dataset is released under [Open Data Commons Attribution License (ODC-By) v1.0](https://opendatacommons.org/licenses/by/1-0/) license and is subject to [CommonCrawl's Terms of Use](https://commoncrawl.org/terms-of-use).190 191## Dataset origin192 193Being based on FineWeb2, this data covers websites over the 2013-2024 time period.194 195FineWeb2 is sourced from the internet at large, it is very likely that some personable identifiable information (PII) will be present, even if the FineWeb2 processing has already anonymized email addresses and public IP addresses. If you find your own PII and would like it removed, please fill out the [FineWeb2 PII removal/opt out form](https://forms.gle/VyNT3ZAUPZjPuWp39).196 197CommonCrawl respects robots.txt at crawl time, but if you are a webmaster and find your website in FineWeb2 and would like to have it removed, you may also use the [FineWeb2 PII removal/opt out form](https://forms.gle/VyNT3ZAUPZjPuWp39).198 199## Considerations for Using the Data200 201For the aspects of social impact, discussion of biases, and known limitations, we also refer to the [FineWeb2 documentation](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2).202 203 204## Citation information205If you use this dataset in your research or applications, please use the following citation:206```207@article{messmer2025multilingdatacomp,208  title={Enhancing Multilingual LLM Pretraining with Model-Based Data Selection}, 209  author={Bettina Messmer and Vinko Sabolčec and Martin Jaggi},210  journal={arXiv},211  year={2025},212  url={https://arxiv.org/abs/2502.10361}, 213}214```