HuggingFaceFW/fineweb

1---2license: odc-by3task_categories:4  - text-generation5language:6  - en7pretty_name: FineWeb8size_categories:9  - n>1T10configs:11  - config_name: default12    data_files:13      - split: train14        path: data/*/*15  - config_name: sample-10BT16    data_files:17      - split: train18        path: sample/10BT/*19  - config_name: sample-100BT20    data_files:21      - split: train22        path: sample/100BT/*23  - config_name: sample-350BT24    data_files:25      - split: train26        path: sample/350BT/*27  - config_name: CC-MAIN-2024-5128    data_files:29      - split: train30        path: data/CC-MAIN-2024-51/*31  - config_name: CC-MAIN-2024-4632    data_files:33      - split: train34        path: data/CC-MAIN-2024-46/*35  - config_name: CC-MAIN-2024-4236    data_files:37      - split: train38        path: data/CC-MAIN-2024-42/*39  - config_name: CC-MAIN-2024-3840    data_files:41      - split: train42        path: data/CC-MAIN-2024-38/*43  - config_name: CC-MAIN-2024-3344    data_files:45      - split: train46        path: data/CC-MAIN-2024-33/*47  - config_name: CC-MAIN-2024-3048    data_files:49      - split: train50        path: data/CC-MAIN-2024-30/*51  - config_name: CC-MAIN-2024-2652    data_files:53      - split: train54        path: data/CC-MAIN-2024-26/*55  - config_name: CC-MAIN-2024-2256    data_files:57      - split: train58        path: data/CC-MAIN-2024-22/*59  - config_name: CC-MAIN-2024-1860    data_files:61      - split: train62        path: data/CC-MAIN-2024-18/*63  - config_name: CC-MAIN-2024-1064    data_files:65      - split: train66        path: data/CC-MAIN-2024-10/*67  - config_name: CC-MAIN-2023-5068    data_files:69      - split: train70        path: data/CC-MAIN-2023-50/*71  - config_name: CC-MAIN-2023-4072    data_files:73      - split: train74        path: data/CC-MAIN-2023-40/*75  - config_name: CC-MAIN-2023-2376    data_files:77      - split: train78        path: data/CC-MAIN-2023-23/*79  - config_name: CC-MAIN-2023-1480    data_files:81      - split: train82        path: data/CC-MAIN-2023-14/*83  - config_name: CC-MAIN-2023-0684    data_files:85      - split: train86        path: data/CC-MAIN-2023-06/*87  - config_name: CC-MAIN-2022-4988    data_files:89      - split: train90        path: data/CC-MAIN-2022-49/*91  - config_name: CC-MAIN-2022-4092    data_files:93      - split: train94        path: data/CC-MAIN-2022-40/*95  - config_name: CC-MAIN-2022-3396    data_files:97      - split: train98        path: data/CC-MAIN-2022-33/*99  - config_name: CC-MAIN-2022-27100    data_files:101      - split: train102        path: data/CC-MAIN-2022-27/*103  - config_name: CC-MAIN-2022-21104    data_files:105      - split: train106        path: data/CC-MAIN-2022-21/*107  - config_name: CC-MAIN-2022-05108    data_files:109      - split: train110        path: data/CC-MAIN-2022-05/*111  - config_name: CC-MAIN-2021-49112    data_files:113      - split: train114        path: data/CC-MAIN-2021-49/*115  - config_name: CC-MAIN-2021-43116    data_files:117      - split: train118        path: data/CC-MAIN-2021-43/*119  - config_name: CC-MAIN-2021-39120    data_files:121      - split: train122        path: data/CC-MAIN-2021-39/*123  - config_name: CC-MAIN-2021-31124    data_files:125      - split: train126        path: data/CC-MAIN-2021-31/*127  - config_name: CC-MAIN-2021-25128    data_files:129      - split: train130        path: data/CC-MAIN-2021-25/*131  - config_name: CC-MAIN-2021-21132    data_files:133      - split: train134        path: data/CC-MAIN-2021-21/*135  - config_name: CC-MAIN-2021-17136    data_files:137      - split: train138        path: data/CC-MAIN-2021-17/*139  - config_name: CC-MAIN-2021-10140    data_files:141      - split: train142        path: data/CC-MAIN-2021-10/*143  - config_name: CC-MAIN-2021-04144    data_files:145      - split: train146        path: data/CC-MAIN-2021-04/*147  - config_name: CC-MAIN-2020-50148    data_files:149      - split: train150        path: data/CC-MAIN-2020-50/*151  - config_name: CC-MAIN-2020-45152    data_files:153      - split: train154        path: data/CC-MAIN-2020-45/*155  - config_name: CC-MAIN-2020-40156    data_files:157      - split: train158        path: data/CC-MAIN-2020-40/*159  - config_name: CC-MAIN-2020-34160    data_files:161      - split: train162        path: data/CC-MAIN-2020-34/*163  - config_name: CC-MAIN-2020-29164    data_files:165      - split: train166        path: data/CC-MAIN-2020-29/*167  - config_name: CC-MAIN-2020-24168    data_files:169      - split: train170        path: data/CC-MAIN-2020-24/*171  - config_name: CC-MAIN-2020-16172    data_files:173      - split: train174        path: data/CC-MAIN-2020-16/*175  - config_name: CC-MAIN-2020-10176    data_files:177      - split: train178        path: data/CC-MAIN-2020-10/*179  - config_name: CC-MAIN-2020-05180    data_files:181      - split: train182        path: data/CC-MAIN-2020-05/*183  - config_name: CC-MAIN-2019-51184    data_files:185      - split: train186        path: data/CC-MAIN-2019-51/*187  - config_name: CC-MAIN-2019-47188    data_files:189      - split: train190        path: data/CC-MAIN-2019-47/*191  - config_name: CC-MAIN-2019-43192    data_files:193      - split: train194        path: data/CC-MAIN-2019-43/*195  - config_name: CC-MAIN-2019-39196    data_files:197      - split: train198        path: data/CC-MAIN-2019-39/*199  - config_name: CC-MAIN-2019-35200    data_files:201      - split: train202        path: data/CC-MAIN-2019-35/*203  - config_name: CC-MAIN-2019-30204    data_files:205      - split: train206        path: data/CC-MAIN-2019-30/*207  - config_name: CC-MAIN-2019-26208    data_files:209      - split: train210        path: data/CC-MAIN-2019-26/*211  - config_name: CC-MAIN-2019-22212    data_files:213      - split: train214        path: data/CC-MAIN-2019-22/*215  - config_name: CC-MAIN-2019-18216    data_files:217      - split: train218        path: data/CC-MAIN-2019-18/*219  - config_name: CC-MAIN-2019-13220    data_files:221      - split: train222        path: data/CC-MAIN-2019-13/*223  - config_name: CC-MAIN-2019-09224    data_files:225      - split: train226        path: data/CC-MAIN-2019-09/*227  - config_name: CC-MAIN-2019-04228    data_files:229      - split: train230        path: data/CC-MAIN-2019-04/*231  - config_name: CC-MAIN-2018-51232    data_files:233      - split: train234        path: data/CC-MAIN-2018-51/*235  - config_name: CC-MAIN-2018-47236    data_files:237      - split: train238        path: data/CC-MAIN-2018-47/*239  - config_name: CC-MAIN-2018-43240    data_files:241      - split: train242        path: data/CC-MAIN-2018-43/*243  - config_name: CC-MAIN-2018-39244    data_files:245      - split: train246        path: data/CC-MAIN-2018-39/*247  - config_name: CC-MAIN-2018-34248    data_files:249      - split: train250        path: data/CC-MAIN-2018-34/*251  - config_name: CC-MAIN-2018-30252    data_files:253      - split: train254        path: data/CC-MAIN-2018-30/*255  - config_name: CC-MAIN-2018-26256    data_files:257      - split: train258        path: data/CC-MAIN-2018-26/*259  - config_name: CC-MAIN-2018-22260    data_files:261      - split: train262        path: data/CC-MAIN-2018-22/*263  - config_name: CC-MAIN-2018-17264    data_files:265      - split: train266        path: data/CC-MAIN-2018-17/*267  - config_name: CC-MAIN-2018-13268    data_files:269      - split: train270        path: data/CC-MAIN-2018-13/*271  - config_name: CC-MAIN-2018-09272    data_files:273      - split: train274        path: data/CC-MAIN-2018-09/*275  - config_name: CC-MAIN-2018-05276    data_files:277      - split: train278        path: data/CC-MAIN-2018-05/*279  - config_name: CC-MAIN-2017-51280    data_files:281      - split: train282        path: data/CC-MAIN-2017-51/*283  - config_name: CC-MAIN-2017-47284    data_files:285      - split: train286        path: data/CC-MAIN-2017-47/*287  - config_name: CC-MAIN-2017-43288    data_files:289      - split: train290        path: data/CC-MAIN-2017-43/*291  - config_name: CC-MAIN-2017-39292    data_files:293      - split: train294        path: data/CC-MAIN-2017-39/*295  - config_name: CC-MAIN-2017-34296    data_files:297      - split: train298        path: data/CC-MAIN-2017-34/*299  - config_name: CC-MAIN-2017-30300    data_files:301      - split: train302        path: data/CC-MAIN-2017-30/*303  - config_name: CC-MAIN-2017-26304    data_files:305      - split: train306        path: data/CC-MAIN-2017-26/*307  - config_name: CC-MAIN-2017-22308    data_files:309      - split: train310        path: data/CC-MAIN-2017-22/*311  - config_name: CC-MAIN-2017-17312    data_files:313      - split: train314        path: data/CC-MAIN-2017-17/*315  - config_name: CC-MAIN-2017-13316    data_files:317      - split: train318        path: data/CC-MAIN-2017-13/*319  - config_name: CC-MAIN-2017-09320    data_files:321      - split: train322        path: data/CC-MAIN-2017-09/*323  - config_name: CC-MAIN-2017-04324    data_files:325      - split: train326        path: data/CC-MAIN-2017-04/*327  - config_name: CC-MAIN-2016-50328    data_files:329      - split: train330        path: data/CC-MAIN-2016-50/*331  - config_name: CC-MAIN-2016-44332    data_files:333      - split: train334        path: data/CC-MAIN-2016-44/*335  - config_name: CC-MAIN-2016-40336    data_files:337      - split: train338        path: data/CC-MAIN-2016-40/*339  - config_name: CC-MAIN-2016-36340    data_files:341      - split: train342        path: data/CC-MAIN-2016-36/*343  - config_name: CC-MAIN-2016-30344    data_files:345      - split: train346        path: data/CC-MAIN-2016-30/*347  - config_name: CC-MAIN-2016-26348    data_files:349      - split: train350        path: data/CC-MAIN-2016-26/*351  - config_name: CC-MAIN-2016-22352    data_files:353      - split: train354        path: data/CC-MAIN-2016-22/*355  - config_name: CC-MAIN-2016-18356    data_files:357      - split: train358        path: data/CC-MAIN-2016-18/*359  - config_name: CC-MAIN-2016-07360    data_files:361      - split: train362        path: data/CC-MAIN-2016-07/*363  - config_name: CC-MAIN-2015-48364    data_files:365      - split: train366        path: data/CC-MAIN-2015-48/*367  - config_name: CC-MAIN-2015-40368    data_files:369      - split: train370        path: data/CC-MAIN-2015-40/*371  - config_name: CC-MAIN-2015-35372    data_files:373      - split: train374        path: data/CC-MAIN-2015-35/*375  - config_name: CC-MAIN-2015-32376    data_files:377      - split: train378        path: data/CC-MAIN-2015-32/*379  - config_name: CC-MAIN-2015-27380    data_files:381      - split: train382        path: data/CC-MAIN-2015-27/*383  - config_name: CC-MAIN-2015-22384    data_files:385      - split: train386        path: data/CC-MAIN-2015-22/*387  - config_name: CC-MAIN-2015-18388    data_files:389      - split: train390        path: data/CC-MAIN-2015-18/*391  - config_name: CC-MAIN-2015-14392    data_files:393      - split: train394        path: data/CC-MAIN-2015-14/*395  - config_name: CC-MAIN-2015-11396    data_files:397      - split: train398        path: data/CC-MAIN-2015-11/*399  - config_name: CC-MAIN-2015-06400    data_files:401      - split: train402        path: data/CC-MAIN-2015-06/*403  - config_name: CC-MAIN-2014-52404    data_files:405      - split: train406        path: data/CC-MAIN-2014-52/*407  - config_name: CC-MAIN-2014-49408    data_files:409      - split: train410        path: data/CC-MAIN-2014-49/*411  - config_name: CC-MAIN-2014-42412    data_files:413      - split: train414        path: data/CC-MAIN-2014-42/*415  - config_name: CC-MAIN-2014-41416    data_files:417      - split: train418        path: data/CC-MAIN-2014-41/*419  - config_name: CC-MAIN-2014-35420    data_files:421      - split: train422        path: data/CC-MAIN-2014-35/*423  - config_name: CC-MAIN-2014-23424    data_files:425      - split: train426        path: data/CC-MAIN-2014-23/*427  - config_name: CC-MAIN-2014-15428    data_files:429      - split: train430        path: data/CC-MAIN-2014-15/*431  - config_name: CC-MAIN-2014-10432    data_files:433      - split: train434        path: data/CC-MAIN-2014-10/*435  - config_name: CC-MAIN-2013-48436    data_files:437      - split: train438        path: data/CC-MAIN-2013-48/*439  - config_name: CC-MAIN-2013-20440    data_files:441      - split: train442        path: data/CC-MAIN-2013-20/*443---444# 🍷 FineWeb445<center>446    <img src="https://huggingface.co/datasets/HuggingFaceFW/admin/resolve/main/fineweb-logo.png" alt="FineWeb: The finest collection of data the web has to offer">447</center>448 449> 15 trillion tokens of the finest data the 🌐 web has to offer450 451# Table of Contents452- [🍷 FineWeb](#-fineweb)453   * [What is it?](#what-is-it)454   * [What is being released?](#what-is-being-released)455   * [Changelog](#changelog)456   * [How to download and use 🍷 FineWeb](#how-to-download-and-use-🍷-fineweb)457      + [Using 🏭 `datatrove`](#using-datatrove)458      + [Using `huggingface_hub`](#using-huggingface_hub)459      + [Using `datasets`](#using-datasets)460   * [Breakdown by dump/crawl](#breakdown-by-dumpcrawl)461   * [Dataset performance evaluation and ablations](#dataset-performance-evaluation-and-ablations)462      + [Hyper-parameters for ablation models](#hyper-parameters-for-ablation-models)463      + [Ablation evaluation benchmarks](#ablation-evaluation-benchmarks)464      + [Comparison with other datasets](#comparison-with-other-datasets)465- [Dataset card for 🍷 FineWeb](#dataset-card-for-🍷-fineweb)466   * [Dataset Summary](#dataset-summary)467   * [Dataset Structure](#dataset-structure)468      + [Data Instances](#data-instances)469      + [Data Fields](#data-fields)470      + [Data Splits](#data-splits)471   * [Dataset Creation](#dataset-creation)472      + [Curation Rationale](#curation-rationale)473      + [Source Data](#source-data)474      + [Data processing steps](#data-processing-steps)475      + [Annotations](#annotations)476      + [Personal and Sensitive Information](#personal-and-sensitive-information)477   * [Considerations for Using the Data](#considerations-for-using-the-data)478      + [Social Impact of Dataset](#social-impact-of-dataset)479      + [Discussion of Biases](#discussion-of-biases)480      + [Other Known Limitations](#other-known-limitations)481   * [Additional Information](#additional-information)482      + [Licensing Information](#licensing-information)483      + [Future work](#future-work)484      + [Citation Information](#citation-information)485 486## What is it?487 488The 🍷 FineWeb dataset consists of more than **15T tokens** of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the 🏭 [`datatrove`](https://github.com/huggingface/datatrove/) library, our large scale data processing library. 489 490🍷 FineWeb was originally meant to be a fully open replication of 🦅 [RefinedWeb](https://huggingface.co/papers/2306.01116), with a release of the **full dataset** under the **ODC-By 1.0 license**. However, by carefully adding additional filtering steps, we managed to push the performance of 🍷 FineWeb well above that of the original 🦅 RefinedWeb, and models trained on our dataset also outperform models trained on other commonly used high quality web datasets (like C4, Dolma-v1.6, The Pile, SlimPajama, RedPajam2) on our aggregate group of [benchmark tasks](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py).491 492That said, we think there is still room for additional filtering and improvement and intend to continue exploring how to improve the dataset quality in coming versions of 🍷 FineWeb.493 494## What is being released?495 496Along with the dataset, which includes all CommonCrawl dumps since 2013, we also share all the code needed to fully reproduce our processing setup using the 🏭 [`datatrove`](https://github.com/huggingface/datatrove/) library [here](https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py). To enable full replication of our results, we have also published the small ablation models we have trained using [`nanotron`](https://github.com/huggingface/nanotron/) to validate the dataset and compare it with other reference datasets. You will find them [here](https://huggingface.co/collections/HuggingFaceFW/ablation-models-662457b0d213e8c14fe47f32), with checkpoints every 1000 steps. We have also published our evaluation results [here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/eval_results.csv). Our evaluation setup is available [here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py).497 498You will find details on the different processing decisions we took and some interesting explorations of deduplication methods on our [blogpost](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1).499 500## Changelog501_Previous versions remain available in the branch `version name`._502 503- **v1.2.0 (03-01-2025):** Added 8 new snapshots: `CC-MAIN-2024-22`, `CC-MAIN-2024-26`, `CC-MAIN-2024-30`, `CC-MAIN-2024-33`, `CC-MAIN-2024-38`, `CC-MAIN-2024-42`, `CC-MAIN-2024-46`, `CC-MAIN-2024-51`, covering May to December 2024.504- **v1.1.0 (31-05-2024):** We reprocessed and reuploaded 11 dumps, `CC-MAIN-2021-49` to `CC-MAIN-2023-40`, as we found a bug on their deduplication. We also added the most recent dump: `CC-MAIN-2024-18`, crawled over April 2024. Expect a small perf improvement505- **v1.0.0 (21-04-2024):** Initial version506 507## How to download and use 🍷 FineWeb508 509You can load the full dataset or a specific crawl/dump (see table below). Dumps have the format `CC-MAIN-(year)-(week number)`.510 511### (Smaller) sample versions512Along with config `default` (all the data), and the configs for each individual dump, you can also download the following configs:513- `sample-350BT`: a subset randomly sampled from the whole dataset of around 350B gpt2 tokens (388GB)514- `sample-100BT`: a subset randomly sampled from the whole dataset of around 100B gpt2 tokens (277.4GB)515- `sample-10BT`: a subset randomly sampled from the whole dataset of around 10B gpt2 tokens (27.6GB)516 517`sample-10B` was sampled from `sample-100B` which in turn was sampled from `sample-350BT`.518 519### Using 🏭 [`datatrove`](https://github.com/huggingface/datatrove/)520 521```python522from datatrove.pipeline.readers import ParquetReader523 524# limit determines how many documents will be streamed (remove for all)525# to fetch a specific dump: hf://datasets/HuggingFaceFW/fineweb/data/CC-MAIN-2024-10526# replace "data" with "sample/100BT" to use the 100BT sample527data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb/data", limit=1000) 528for document in data_reader():529    # do something with document530    print(document)531 532###############################    533# OR for a processing pipeline:534###############################535 536from datatrove.executor import LocalPipelineExecutor537from datatrove.pipeline.readers import ParquetReader538from datatrove.pipeline.filters import LambdaFilter539from datatrove.pipeline.writers import JsonlWriter540 541pipeline_exec = LocalPipelineExecutor(542    pipeline=[543        # replace "data/CC-MAIN-2024-10" with "sample/100BT" to use the 100BT sample544        ParquetReader("hf://datasets/HuggingFaceFW/fineweb/data/CC-MAIN-2024-10", limit=1000),545        LambdaFilter(lambda doc: "hugging" in doc.text),546        JsonlWriter("some-output-path")547    ],548    tasks=10549)550pipeline_exec.run()551```552 553### Using `huggingface_hub`554 555```python556from huggingface_hub import snapshot_download557folder = snapshot_download(558                "HuggingFaceFW/fineweb", 559                repo_type="dataset",560                local_dir="./fineweb/",561                # replace "data/CC-MAIN-2023-50/*" with "sample/100BT/*" to use the 100BT sample562                allow_patterns="data/CC-MAIN-2023-50/*")563```564 565For faster downloads, make sure to install `pip install huggingface_hub[hf_transfer]` and set the environment variable `HF_HUB_ENABLE_HF_TRANSFER=1`.566 567### Using `datasets`568 569```python570from datasets import load_dataset571# use name="sample-10BT" to use the 10BT sample572fw = load_dataset("HuggingFaceFW/fineweb", name="CC-MAIN-2024-10", split="train", streaming=True)573```574 575## Breakdown by dump/crawl576 577| Dump | Time period | Disk size (GB) | gpt2 tokens (billions) |578| --- | --- | --- | --- |              579| CC-MAIN-2024-51 | December 2024             | 362.6 | 131.2 |580| CC-MAIN-2024-46 | November 2024             | 344.4 | 124.8 |581| CC-MAIN-2024-42 | October 2024             | 314.9 | 114.1 |582| CC-MAIN-2024-38 | September 2024             | 374.8 | 135.9 |583| CC-MAIN-2024-33 | August 2024             | 313.4 | 113.4 |584| CC-MAIN-2024-30 | July 2024             | 363.3 | 131.7 |585| CC-MAIN-2024-26 | June 2024             | 367.4 | 133.3 |586| CC-MAIN-2024-22 | May 2024             | 403.5 | 146.5 |587| CC-MAIN-2024-18 | April 2024             | 417.6          | 154.4                       |588| CC-MAIN-2024-10 | February/March 2024    | 432.0          | 157.2                       |589| CC-MAIN-2023-50 | November/December 2023 | 650.0          | 239.7                       |590| CC-MAIN-2023-40 | September/October 2023 | 668.7          | 252.0                       |591| CC-MAIN-2023-23 | May/June 2023          | 654.4          | 249.2                       |592| CC-MAIN-2023-14 | March/April 2023       | 621.3          | 236.5                       |593| CC-MAIN-2023-06 | January/February 2023  | 621.9          | 233.9                       |594| CC-MAIN-2022-49 | November/December 2022 | 631.2          | 237.5                       |595| CC-MAIN-2022-40 | September/October 2022 | 606.4          | 228.7                       |596| CC-MAIN-2022-33 | August 2022            | 434.6          | 163.5                       |597| CC-MAIN-2022-27 | June/July 2022         | 574.9          | 216.1                       |598| CC-MAIN-2022-21 | May 2022               | 646.4          | 242.7                       |599| CC-MAIN-2022-05 | January 2022           | 520.1          | 195.4                       |600| CC-MAIN-2021-49 | November/December 2021 | 413.7          | 155.5                       |601| CC-MAIN-2021-43 | October 2021           | 601.5          | 221.0                       |602| CC-MAIN-2021-43 | October 2021 | 601.5 | 221.0 |603| CC-MAIN-2021-39 | September 2021 | 518.9 | 190.6 |604| CC-MAIN-2021-31 | July/August 2021 | 593.9 | 217.7 |605| CC-MAIN-2021-25 | June 2021 | 424.4 | 155.7 |606| CC-MAIN-2021-21 | May 2021 | 455.9 | 167.4 |607| CC-MAIN-2021-17 | April 2021 | 556.0 | 204.1 |608| CC-MAIN-2021-10 | February/March 2021 | 463.2 | 169.6 |609| CC-MAIN-2021-04 | January 2021 | 562.4 | 205.4 |610| CC-MAIN-2020-50 | November/December 2020 | 422.8 | 154.3 |611| CC-MAIN-2020-45 | October 2020 | 426.9 | 155.8 |612| CC-MAIN-2020-40 | September 2020 | 555.5 | 202.4 |613| CC-MAIN-2020-34 | August 2020 | 379.6 | 138.7 |614| CC-MAIN-2020-29 | July 2020 | 489.6 | 178.7 |615| CC-MAIN-2020-24 | May/June 2020 | 398.7 | 145.1 |616| CC-MAIN-2020-16 | March/April 2020 | 454.0 | 165.6 |617| CC-MAIN-2020-10 | February 2020 | 369.6 | 134.7 |618| CC-MAIN-2020-05 | January 2020 | 483.3 | 176.4 |619| CC-MAIN-2019-51 | December 2019 | 359.3 | 130.9 |620| CC-MAIN-2019-47 | November 2019 | 395.4 | 144.0 |621| CC-MAIN-2019-43 | October 2019 | 422.3 | 153.9 |622| CC-MAIN-2019-39 | September 2019 | 394.4 | 143.7 |623| CC-MAIN-2019-35 | August 2019 | 454.2 | 165.4 |624| CC-MAIN-2019-30 | July 2019 | 416.6 | 151.5 |625| CC-MAIN-2019-26 | June 2019 | 412.9 | 150.1 |626| CC-MAIN-2019-22 | May 2019 | 432.8 | 157.4 |627| CC-MAIN-2019-18 | April 2019 | 426.7 | 155.3 |628| CC-MAIN-2019-13 | March 2019 | 417.8 | 152.1 |629| CC-MAIN-2019-09 | February 2019 | 467.2 | 169.9 |630| CC-MAIN-2019-04 | January 2019 | 438.1 | 158.7 |631| CC-MAIN-2018-51 | December 2018 | 498.6 | 180.8 |632| CC-MAIN-2018-47 | November 2018 | 437.7 | 158.9 |633| CC-MAIN-2018-43 | October 2018 | 468.8 | 169.9 |634| CC-MAIN-2018-39 | September 2018 | 429.2 | 155.2 |635| CC-MAIN-2018-34 | August 2018 | 408.2 | 148.0 |636| CC-MAIN-2018-30 | July 2018 | 501.5 | 181.4 |637| CC-MAIN-2018-26 | June 2018 | 467.5 | 170.0 |638| CC-MAIN-2018-22 | May 2018 | 398.6 | 144.2 |639| CC-MAIN-2018-17 | April 2018 | 435.1 | 158.1 |640| CC-MAIN-2018-13 | March 2018 | 471.5 | 171.5 |641| CC-MAIN-2018-09 | February 2018 | 490.2 | 178.0 |642| CC-MAIN-2018-05 | January 2018 | 493.5 | 180.7 |643| CC-MAIN-2017-51 | December 2017 | 442.6 | 161.5 |644| CC-MAIN-2017-47 | November 2017 | 457.9 | 167.1 |645| CC-MAIN-2017-43 | October 2017 | 535.6 | 194.9 |646| CC-MAIN-2017-39 | September 2017 | 444.5 | 162.3 |647| CC-MAIN-2017-34 | August 2017 | 503.2 | 183.4 |648| CC-MAIN-2017-30 | July 2017 | 439.2 | 161.2 |649| CC-MAIN-2017-26 | June 2017 | 491.5 | 179.8 |650| CC-MAIN-2017-22 | May 2017 | 441.0 | 161.5 |651| CC-MAIN-2017-17 | April 2017 | 596.8 | 218.6 |652| CC-MAIN-2017-13 | March 2017 | 579.8 | 212.1 |653| CC-MAIN-2017-09 | February 2017 | 492.2 | 180.2 |654| CC-MAIN-2017-04 | January 2017 | 474.3 | 174.4 |655| CC-MAIN-2016-50 | December 2016 | 448.9 | 165.4 |656| CC-MAIN-2016-44 | October 2016 | 467.8 | 172.0 |657| CC-MAIN-2016-40 | September 2016 | 386.1 | 142.8 |658| CC-MAIN-2016-36 | August 2016 | 339.6 | 126.3 |659| CC-MAIN-2016-30 | July 2016 | 346.0 | 128.4 |660| CC-MAIN-2016-26 | June 2016 | 256.5 | 95.5 |661| CC-MAIN-2016-22 | May 2016 | 310.9 | 115.4 |662| CC-MAIN-2016-18 | April 2016 | 298.1 | 110.8 |663| CC-MAIN-2016-07 | February 2016 | 342.7 | 127.2 |664| CC-MAIN-2015-48 | November 2015 | 353.9 | 131.3 |665| CC-MAIN-2015-40 | September 2015 | 284.0 | 105.5 |666| CC-MAIN-2015-35 | August 2015 | 359.4 | 133.2 |667| CC-MAIN-2015-32 | July 2015 | 352.4 | 130.1 |668| CC-MAIN-2015-27 | June 2015 | 335.5 | 124.0 |669| CC-MAIN-2015-22 | May 2015 | 380.2 | 140.4 |670| CC-MAIN-2015-18 | April 2015 | 389.0 | 143.8 |671| CC-MAIN-2015-14 | March 2015 | 337.5 | 124.5 |672| CC-MAIN-2015-11 | February 2015 | 361.4 | 133.3 |673| CC-MAIN-2015-06 | January 2015 | 356.1 | 131.3 |674| CC-MAIN-2014-52 | December 2014 | 388.5 | 143.3 |675| CC-MAIN-2014-49 | November 2014 | 319.9 | 117.7 |676| CC-MAIN-2014-42 | October 2014 | 371.1 | 136.4 |677| CC-MAIN-2014-41 | September 2014 | 408.1 | 150.2 |678| CC-MAIN-2014-35 | August 2014 | 395.7 | 145.6 |679| CC-MAIN-2014-23 | July 2014 | 425.0 | 156.5 |680| CC-MAIN-2014-15 | April 2014 | 369.1 | 135.7 |681| CC-MAIN-2014-10 | March 2014 | 396.2 | 146.2 |682| CC-MAIN-2013-48 | Winter 2013 | 396.8 | 145.9 |683| CC-MAIN-2013-20 | Summer 2013 | 393.9 | 144.5 |684| Total |  | 46,502.2 | 17,087.2 |685 686## Dataset performance evaluation and ablations687 688We conducted our dataset performance ablations and evaluations by training a series of 1.8B parameters models on 27 billion tokens. To compare 🍷 FineWeb with other datasets, we also trained one of these 1.8B models per target dataset, on 350 billion tokens sampled from it (or the entire dataset when its size was < 350 billion tokens).689 690### Hyper-parameters for ablation models691 692The detailed configurations for training the 1.8B parameters ablation model can be found here (link will be added soon).693 694### Ablation evaluation benchmarks695 696To conduct the ablations for each of our dataset filtering choices, we selected a set of benchmarks which we identified as “high-signal” benchmarks. These benchmarks were selected according to the following criteria:697 698- small variance between runs trained on different samplings of the same dataset699- performance increasing monotically during training (or close)700- separation between runs on datasets of known quality (C4, The Pile, RedPajama) higher than the variance between runs with various modeling/data seeds701 702We used the following list of benchmark for our ablation runs:703 704- commonsense_qa (acc/acc_norm)705- hellaswag (acc/acc_norm)706- openbookqa (acc/acc_norm)707- piqa (acc/acc_norm)708- siqa (acc/acc_norm)709- winogrande (acc/acc_norm)710- arc (acc/acc_norm)711- mmlu (acc/acc_norm)712 713To compare runs we consider an aggregate score, the average of the scores for these tasks.714 715The prompts for all these benchmarks are formatted in order to compute and compare the log-likelihood of the full answers for each multiple choice question. All the implementation details for the benchmarks are available in `lighteval` [here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py).716 717### Comparison with other datasets718 719We compared 🍷 FineWeb with the following datasets:720 721- [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)722- [C4](https://huggingface.co/datasets/allenai/c4)723- [Dolma v1.6](https://huggingface.co/datasets/allenai/dolma) (the CommonCrawl part)724- [The Pile](https://huggingface.co/datasets/EleutherAI/pile)725- [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B)726- [RedPajama2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2) (deduplicated)727 728You will find these models on [this collection](https://huggingface.co/collections/HuggingFaceFW/ablation-models-662457b0d213e8c14fe47f32). We have uploaded checkpoints at every 1000 training steps. You will also find our full [evaluation results here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/eval_results.csv).729 730<center>731    <img src="https://huggingface.co/datasets/HuggingFaceFW/admin/resolve/main/fineweb-ablations.png" alt="ablations">732</center>733 734_Note:_ The plot is smoothed by averaging 5k steps in a rolling window.735 736# Dataset card for 🍷 FineWeb737 738## Dataset Description739 740- **Homepage and Repository:** [https://huggingface.co/datasets/HuggingFaceFW/fineweb](https://huggingface.co/datasets/HuggingFaceFW/fineweb)741- **Point of Contact:** please create a discussion on the Community tab742- **License:** Open Data Commons Attribution License (ODC-By) v1.0743 744### Dataset Summary745 746This dataset was created by processing 96 [CommonCrawl](https://commoncrawl.org/) dumps comprising web data crawled from the summer of 2013 to April of 2024. 🍷 FineWeb includes a variety of domains and topics in English and is primarily intended to be used as a research artifact on public data in the context of pretraining dataset for large language models. The CommonCrawl data was carefully processed, filtered and deduplicated with the 🏭 [`datatrove`](https://github.com/huggingface/datatrove/) library, resulting in the largest publicly available clean LLM pretraining dataset, counting around 15 trillion tokens (gpt2 tokenizer).747 748## Dataset Structure749 750### Data Instances751 752The following is an example sample from the dataset. It is part of the `CC-MAIN-2021-43` and was crawled on `2021-10-15T21:20:12Z`.753 754```json755{756   "text": "This is basically a peanut flavoured cream thickened with egg yolks and then set into a ramekin on top of some jam. Tony, one of the Wedgwood chefs, suggested sprinkling on some toasted crushed peanuts at the end to create extra crunch, which I thought was a great idea. The result is excellent.",757   "id": "<urn:uuid:e5a3e79a-13d4-4147-a26e-167536fcac5d>",758   "dump": "CC-MAIN-2021-43",759   "url": "<http://allrecipes.co.uk/recipe/24758/peanut-butter-and-jam-creme-brulee.aspx?o_is=SimilarRecipes&o_ln=SimRecipes_Photo_7>",760   "date": "2021-10-15T21:20:12Z",761   "file_path": "s3://commoncrawl/crawl-data/CC-MAIN-2021-43/segments/1634323583083.92/warc/CC-MAIN-20211015192439-20211015222439-00600.warc.gz",762   "language": "en",763   "language_score": 0.948729,764   "token_count": 69765}766```767 768### Data Fields769 770- `text` (string): the main text content771- `id` (string): original unique identifier for this sample from CommonCrawl772- `dump` (string): the CommonCrawl dump this sample was a part of773- `url` (string): url to the original page where `text` was present774- `date` (string): crawl date (from CommonCrawl)775- `file_path` (string): s3 path for the individual CommonCrawl warc file containing this sample776- `language` (string): `en` for all the samples in this dataset777- `language_score` (float): language prediction score (`0.01.0`) as reported by the [fastText language classifier](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/filters/language_filter.py)778- `token_count` (int): number of tokens when applying the `gpt2` tokenizer to this sample779 780### Data Splits781 782The `default` subset includes the entire dataset. If you would like to only use the data from a particular [CommonCrawl dump](https://commoncrawl.org/overview), you can use the dump name as a subset. You will find the full list of available dumps on the table above.783From experiments we have run, not all dumps give the same performance. For relatively small trainings (<550 billion tokens) we recommend using the recent `CC-MAIN-2023-50`, `CC-MAIN-2024-10` and `CC-MAIN-2024-18`. 784 785## Dataset Creation786 787### Curation Rationale788 789While multiple open-weights models have regularly been released in recent months, these releases often do not include the model's training data. With 🍷 FineWeb we aim to provide the open source community with a very large clean pretraining dataset that can be used to push the envelope on truly open source models (open source models where data is also released). 790 791### Source Data792 793The source data consists of webpages crawled by the CommonCrawl foundation over the 2013-2024 time period.794 795We then extracted the main page text from the html of each webpage, carefully filtered each sample and deduplicated each individual CommonCrawl dump/crawl.796 797While we originally intended to deduplicate the dataset as a whole, our ablations showed that training on a sampling of individually deduplicated dumps/crawls outperformed training on a sampling of all the dumps/crawls deduplicated together. You will find more details on our [blogpost](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1).798 799### Data processing steps800 801We used the 🏭 `datatrove` library to process the data.802You can find a **working script** that launches the [entire processing pipeline here](https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py).803 804The data processing pipeline consists of:805 8061. [Url Filtering](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/url_filter.py), removing documents originating from Malicious and NSFW websites, using both block-list as well as subwords detection8072. [Trafilatura](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/extractors/trafilatura.py) text extraction on the raw HTML from CommonCrawl’s warc files8083. [FastText LanguageFilter](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/language_filter.py), removing any document with `en` language score lower than **0.65**8094. Quality filtering810    1. [Gopher Repetition /](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/gopher_repetition_filter.py) [Quality](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/gopher_quality_filter.py)811    2. [C4 Quality filters](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/c4_quality_filter.py) except `terminal_punct` rule812    3. [FineWeb custom filters](https://github.com/huggingface/datatrove/blob/05194d3960741e7d5c0bd0d6dd69d44514622549/src/datatrove/pipeline/filters/fineweb_quality_filter.py), consisting of heuristics for removing list-like documents, documents with repeated lines and documents with likely wrong line formatting. 8135. [MinHash deduplication](https://github.com/huggingface/datatrove/blob/6daa5e879e06b21e6886b37e2b1be4ae58a658b6/src/datatrove/pipeline/dedup/minhash.py) with each crawl deduplicated individually (5-grams, 14x8 hash functions)8146. [PII Formatting](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/formatters/pii.py) to anonymize email and public IP addresses815 816### Annotations817 818We augment the original samples with the `language`, `language_score` and `token_count` annotations. The language related annotations are automatically generated by our [language filter](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/filters/language_filter.py). `token_count` is generated by [applying the gpt2 tokenizer](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/tokens/counter.py) to the `text` column.819 820### Personal and Sensitive Information821 822We anonymize email addresses and public IP addresses. 823 824For emails, we apply a regex pattern and replace any occurrence of an email address with either `email@example.com` or `firstname.lastname@example.org`. For IP addresses, we also employ a regex pattern and then further filter to only anonymize IP addresses [allocated for public networks](https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml). Matched IP addresses are then replaced with one of the following randomly generated IP addresses, which at the time of dataset creation were not responding to ping requests: `22.214.171.124`, `126.96.36.199`, `188.8.131.52`, `184.108.40.206`, `220.127.116.11`, and `18.104.22.168`. We decided against applying regex patterns for phone numbers due to the high false positive rate.825 826Despite our efforts, given that 🍷 FineWeb is sourced from the internet at large, it is very likely that some personable identifiable information (PII) will be present. If you find your own PII in 🍷 FineWeb and would like it removed, please fill out our [PII removal form](https://forms.gle/VyNT3ZAUPZjPuWp39).827 828## Considerations for Using the Data829 830### Social Impact of Dataset831 832With the release of this dataset we aim to make model training more accessible to the machine learning community at large. 833 834While multiple open-weights models with strong performance have been publicly released in the past, more often than not these releases are not accompanied by the corresponding training dataset. This is unfortunate as the dataset specificities and characteristics have been demonstrated to have a very large impact and role in the performances of the models. As the creation of a high quality training dataset is a fundamental requirement to training an LLM capable of excelling at downstream tasks, with 🍷 FineWeb we (a) not only make the dataset creation process more transparent, by sharing our entire processing setup including the codebase used, we also (b) help alleviate the costs of dataset curation, both in time and in compute, for model creators by publicly releasing our dataset with the community.835 836### Discussion of Biases837 838Efforts were made to minimize the amount of NSFW and toxic content present in the dataset by employing filtering on the URL level. However, there are still a significant number of documents present in the final dataset that could be considered toxic or contain harmful content. As 🍷 FineWeb was sourced from the web as a whole, any harmful biases typically present in it may be reproduced on our dataset.839 840We deliberately avoided using machine learning filtering methods that define text quality based on the similarity to a “gold” source such as wikipedia or toxicity classifiers as these methods have been known to [disproportionately remove content in specific dialects](https://aclanthology.org/D16-1120/) and [overclassify as toxic text related to specific social identities](https://arxiv.org/pdf/2109.07445.pdf), respectively.841 842### Other Known Limitations843 844As a consequence of some of the filtering steps applied, it is likely that code content is not prevalent in our dataset. If you are training a model that should also perform code tasks, we recommend you use 🍷 FineWeb with a code dataset, such as [The Stack v2](https://huggingface.co/datasets/bigcode/the-stack-v2). You should also probably consider complementing 🍷 FineWeb with specialized curated sources (such as Wikipedia, for example) as they will likely have better formatting than the wikipedia content included in 🍷 FineWeb (we did not tailor the processing to individual websites).845 846## Additional Information847 848### Licensing Information849 850The dataset is released under the **Open Data Commons Attribution License (ODC-By) v1.0** [license](https://opendatacommons.org/licenses/by/1-0/). The use of this dataset is also subject to [CommonCrawl's Terms of Use](https://commoncrawl.org/terms-of-use).851 852### Future work853 854We plan to not only continue but also expand our efforts to create open-source high quality training datasets and to improve 🍷 FineWeb itself in future iterations.855 856## Citation Information857Paper on [arXiv](https://arxiv.org/abs/2406.17557)858```859@inproceedings{860  penedo2024the,861  title={The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale},862  author={Guilherme Penedo and Hynek Kydl{\'\i}{\v{c}}ek and Loubna Ben allal and Anton Lozhkov and Margaret Mitchell and Colin Raffel and Leandro Von Werra and Thomas Wolf},863  booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},864  year={2024},865  url={https://openreview.net/forum?id=n6SCkn2QaG}866}867```868