HuggingFaceFW/fineweb

1---2license: odc-by3task_categories:4  - text-generation5language:6  - en7pretty_name: FineWeb8size_categories:9  - n>1T10configs:11  - config_name: default12    data_files:13      - split: train14        path: data/*/*15  - config_name: sample-10BT16    data_files:17      - split: train18        path: sample/10BT/*19  - config_name: sample-100BT20    data_files:21      - split: train22        path: sample/100BT/*23  - config_name: sample-350BT24    data_files:25      - split: train26        path: sample/350BT/*27  - config_name: CC-MAIN-2025-0528    data_files:29      - split: train30        path: data/CC-MAIN-2025-05/*31  - config_name: CC-MAIN-2025-0832    data_files:33      - split: train34        path: data/CC-MAIN-2025-08/*35  - config_name: CC-MAIN-2025-1336    data_files:37      - split: train38        path: data/CC-MAIN-2025-13/*39  - config_name: CC-MAIN-2025-1840    data_files:41      - split: train42        path: data/CC-MAIN-2025-18/*43  - config_name: CC-MAIN-2025-2144    data_files:45      - split: train46        path: data/CC-MAIN-2025-21/*47  - config_name: CC-MAIN-2025-2648    data_files:49      - split: train50        path: data/CC-MAIN-2025-26/*51  - config_name: CC-MAIN-2024-5152    data_files:53      - split: train54        path: data/CC-MAIN-2024-51/*55  - config_name: CC-MAIN-2024-4656    data_files:57      - split: train58        path: data/CC-MAIN-2024-46/*59  - config_name: CC-MAIN-2024-4260    data_files:61      - split: train62        path: data/CC-MAIN-2024-42/*63  - config_name: CC-MAIN-2024-3864    data_files:65      - split: train66        path: data/CC-MAIN-2024-38/*67  - config_name: CC-MAIN-2024-3368    data_files:69      - split: train70        path: data/CC-MAIN-2024-33/*71  - config_name: CC-MAIN-2024-3072    data_files:73      - split: train74        path: data/CC-MAIN-2024-30/*75  - config_name: CC-MAIN-2024-2676    data_files:77      - split: train78        path: data/CC-MAIN-2024-26/*79  - config_name: CC-MAIN-2024-2280    data_files:81      - split: train82        path: data/CC-MAIN-2024-22/*83  - config_name: CC-MAIN-2024-1884    data_files:85      - split: train86        path: data/CC-MAIN-2024-18/*87  - config_name: CC-MAIN-2024-1088    data_files:89      - split: train90        path: data/CC-MAIN-2024-10/*91  - config_name: CC-MAIN-2023-5092    data_files:93      - split: train94        path: data/CC-MAIN-2023-50/*95  - config_name: CC-MAIN-2023-4096    data_files:97      - split: train98        path: data/CC-MAIN-2023-40/*99  - config_name: CC-MAIN-2023-23100    data_files:101      - split: train102        path: data/CC-MAIN-2023-23/*103  - config_name: CC-MAIN-2023-14104    data_files:105      - split: train106        path: data/CC-MAIN-2023-14/*107  - config_name: CC-MAIN-2023-06108    data_files:109      - split: train110        path: data/CC-MAIN-2023-06/*111  - config_name: CC-MAIN-2022-49112    data_files:113      - split: train114        path: data/CC-MAIN-2022-49/*115  - config_name: CC-MAIN-2022-40116    data_files:117      - split: train118        path: data/CC-MAIN-2022-40/*119  - config_name: CC-MAIN-2022-33120    data_files:121      - split: train122        path: data/CC-MAIN-2022-33/*123  - config_name: CC-MAIN-2022-27124    data_files:125      - split: train126        path: data/CC-MAIN-2022-27/*127  - config_name: CC-MAIN-2022-21128    data_files:129      - split: train130        path: data/CC-MAIN-2022-21/*131  - config_name: CC-MAIN-2022-05132    data_files:133      - split: train134        path: data/CC-MAIN-2022-05/*135  - config_name: CC-MAIN-2021-49136    data_files:137      - split: train138        path: data/CC-MAIN-2021-49/*139  - config_name: CC-MAIN-2021-43140    data_files:141      - split: train142        path: data/CC-MAIN-2021-43/*143  - config_name: CC-MAIN-2021-39144    data_files:145      - split: train146        path: data/CC-MAIN-2021-39/*147  - config_name: CC-MAIN-2021-31148    data_files:149      - split: train150        path: data/CC-MAIN-2021-31/*151  - config_name: CC-MAIN-2021-25152    data_files:153      - split: train154        path: data/CC-MAIN-2021-25/*155  - config_name: CC-MAIN-2021-21156    data_files:157      - split: train158        path: data/CC-MAIN-2021-21/*159  - config_name: CC-MAIN-2021-17160    data_files:161      - split: train162        path: data/CC-MAIN-2021-17/*163  - config_name: CC-MAIN-2021-10164    data_files:165      - split: train166        path: data/CC-MAIN-2021-10/*167  - config_name: CC-MAIN-2021-04168    data_files:169      - split: train170        path: data/CC-MAIN-2021-04/*171  - config_name: CC-MAIN-2020-50172    data_files:173      - split: train174        path: data/CC-MAIN-2020-50/*175  - config_name: CC-MAIN-2020-45176    data_files:177      - split: train178        path: data/CC-MAIN-2020-45/*179  - config_name: CC-MAIN-2020-40180    data_files:181      - split: train182        path: data/CC-MAIN-2020-40/*183  - config_name: CC-MAIN-2020-34184    data_files:185      - split: train186        path: data/CC-MAIN-2020-34/*187  - config_name: CC-MAIN-2020-29188    data_files:189      - split: train190        path: data/CC-MAIN-2020-29/*191  - config_name: CC-MAIN-2020-24192    data_files:193      - split: train194        path: data/CC-MAIN-2020-24/*195  - config_name: CC-MAIN-2020-16196    data_files:197      - split: train198        path: data/CC-MAIN-2020-16/*199  - config_name: CC-MAIN-2020-10200    data_files:201      - split: train202        path: data/CC-MAIN-2020-10/*203  - config_name: CC-MAIN-2020-05204    data_files:205      - split: train206        path: data/CC-MAIN-2020-05/*207  - config_name: CC-MAIN-2019-51208    data_files:209      - split: train210        path: data/CC-MAIN-2019-51/*211  - config_name: CC-MAIN-2019-47212    data_files:213      - split: train214        path: data/CC-MAIN-2019-47/*215  - config_name: CC-MAIN-2019-43216    data_files:217      - split: train218        path: data/CC-MAIN-2019-43/*219  - config_name: CC-MAIN-2019-39220    data_files:221      - split: train222        path: data/CC-MAIN-2019-39/*223  - config_name: CC-MAIN-2019-35224    data_files:225      - split: train226        path: data/CC-MAIN-2019-35/*227  - config_name: CC-MAIN-2019-30228    data_files:229      - split: train230        path: data/CC-MAIN-2019-30/*231  - config_name: CC-MAIN-2019-26232    data_files:233      - split: train234        path: data/CC-MAIN-2019-26/*235  - config_name: CC-MAIN-2019-22236    data_files:237      - split: train238        path: data/CC-MAIN-2019-22/*239  - config_name: CC-MAIN-2019-18240    data_files:241      - split: train242        path: data/CC-MAIN-2019-18/*243  - config_name: CC-MAIN-2019-13244    data_files:245      - split: train246        path: data/CC-MAIN-2019-13/*247  - config_name: CC-MAIN-2019-09248    data_files:249      - split: train250        path: data/CC-MAIN-2019-09/*251  - config_name: CC-MAIN-2019-04252    data_files:253      - split: train254        path: data/CC-MAIN-2019-04/*255  - config_name: CC-MAIN-2018-51256    data_files:257      - split: train258        path: data/CC-MAIN-2018-51/*259  - config_name: CC-MAIN-2018-47260    data_files:261      - split: train262        path: data/CC-MAIN-2018-47/*263  - config_name: CC-MAIN-2018-43264    data_files:265      - split: train266        path: data/CC-MAIN-2018-43/*267  - config_name: CC-MAIN-2018-39268    data_files:269      - split: train270        path: data/CC-MAIN-2018-39/*271  - config_name: CC-MAIN-2018-34272    data_files:273      - split: train274        path: data/CC-MAIN-2018-34/*275  - config_name: CC-MAIN-2018-30276    data_files:277      - split: train278        path: data/CC-MAIN-2018-30/*279  - config_name: CC-MAIN-2018-26280    data_files:281      - split: train282        path: data/CC-MAIN-2018-26/*283  - config_name: CC-MAIN-2018-22284    data_files:285      - split: train286        path: data/CC-MAIN-2018-22/*287  - config_name: CC-MAIN-2018-17288    data_files:289      - split: train290        path: data/CC-MAIN-2018-17/*291  - config_name: CC-MAIN-2018-13292    data_files:293      - split: train294        path: data/CC-MAIN-2018-13/*295  - config_name: CC-MAIN-2018-09296    data_files:297      - split: train298        path: data/CC-MAIN-2018-09/*299  - config_name: CC-MAIN-2018-05300    data_files:301      - split: train302        path: data/CC-MAIN-2018-05/*303  - config_name: CC-MAIN-2017-51304    data_files:305      - split: train306        path: data/CC-MAIN-2017-51/*307  - config_name: CC-MAIN-2017-47308    data_files:309      - split: train310        path: data/CC-MAIN-2017-47/*311  - config_name: CC-MAIN-2017-43312    data_files:313      - split: train314        path: data/CC-MAIN-2017-43/*315  - config_name: CC-MAIN-2017-39316    data_files:317      - split: train318        path: data/CC-MAIN-2017-39/*319  - config_name: CC-MAIN-2017-34320    data_files:321      - split: train322        path: data/CC-MAIN-2017-34/*323  - config_name: CC-MAIN-2017-30324    data_files:325      - split: train326        path: data/CC-MAIN-2017-30/*327  - config_name: CC-MAIN-2017-26328    data_files:329      - split: train330        path: data/CC-MAIN-2017-26/*331  - config_name: CC-MAIN-2017-22332    data_files:333      - split: train334        path: data/CC-MAIN-2017-22/*335  - config_name: CC-MAIN-2017-17336    data_files:337      - split: train338        path: data/CC-MAIN-2017-17/*339  - config_name: CC-MAIN-2017-13340    data_files:341      - split: train342        path: data/CC-MAIN-2017-13/*343  - config_name: CC-MAIN-2017-09344    data_files:345      - split: train346        path: data/CC-MAIN-2017-09/*347  - config_name: CC-MAIN-2017-04348    data_files:349      - split: train350        path: data/CC-MAIN-2017-04/*351  - config_name: CC-MAIN-2016-50352    data_files:353      - split: train354        path: data/CC-MAIN-2016-50/*355  - config_name: CC-MAIN-2016-44356    data_files:357      - split: train358        path: data/CC-MAIN-2016-44/*359  - config_name: CC-MAIN-2016-40360    data_files:361      - split: train362        path: data/CC-MAIN-2016-40/*363  - config_name: CC-MAIN-2016-36364    data_files:365      - split: train366        path: data/CC-MAIN-2016-36/*367  - config_name: CC-MAIN-2016-30368    data_files:369      - split: train370        path: data/CC-MAIN-2016-30/*371  - config_name: CC-MAIN-2016-26372    data_files:373      - split: train374        path: data/CC-MAIN-2016-26/*375  - config_name: CC-MAIN-2016-22376    data_files:377      - split: train378        path: data/CC-MAIN-2016-22/*379  - config_name: CC-MAIN-2016-18380    data_files:381      - split: train382        path: data/CC-MAIN-2016-18/*383  - config_name: CC-MAIN-2016-07384    data_files:385      - split: train386        path: data/CC-MAIN-2016-07/*387  - config_name: CC-MAIN-2015-48388    data_files:389      - split: train390        path: data/CC-MAIN-2015-48/*391  - config_name: CC-MAIN-2015-40392    data_files:393      - split: train394        path: data/CC-MAIN-2015-40/*395  - config_name: CC-MAIN-2015-35396    data_files:397      - split: train398        path: data/CC-MAIN-2015-35/*399  - config_name: CC-MAIN-2015-32400    data_files:401      - split: train402        path: data/CC-MAIN-2015-32/*403  - config_name: CC-MAIN-2015-27404    data_files:405      - split: train406        path: data/CC-MAIN-2015-27/*407  - config_name: CC-MAIN-2015-22408    data_files:409      - split: train410        path: data/CC-MAIN-2015-22/*411  - config_name: CC-MAIN-2015-18412    data_files:413      - split: train414        path: data/CC-MAIN-2015-18/*415  - config_name: CC-MAIN-2015-14416    data_files:417      - split: train418        path: data/CC-MAIN-2015-14/*419  - config_name: CC-MAIN-2015-11420    data_files:421      - split: train422        path: data/CC-MAIN-2015-11/*423  - config_name: CC-MAIN-2015-06424    data_files:425      - split: train426        path: data/CC-MAIN-2015-06/*427  - config_name: CC-MAIN-2014-52428    data_files:429      - split: train430        path: data/CC-MAIN-2014-52/*431  - config_name: CC-MAIN-2014-49432    data_files:433      - split: train434        path: data/CC-MAIN-2014-49/*435  - config_name: CC-MAIN-2014-42436    data_files:437      - split: train438        path: data/CC-MAIN-2014-42/*439  - config_name: CC-MAIN-2014-41440    data_files:441      - split: train442        path: data/CC-MAIN-2014-41/*443  - config_name: CC-MAIN-2014-35444    data_files:445      - split: train446        path: data/CC-MAIN-2014-35/*447  - config_name: CC-MAIN-2014-23448    data_files:449      - split: train450        path: data/CC-MAIN-2014-23/*451  - config_name: CC-MAIN-2014-15452    data_files:453      - split: train454        path: data/CC-MAIN-2014-15/*455  - config_name: CC-MAIN-2014-10456    data_files:457      - split: train458        path: data/CC-MAIN-2014-10/*459  - config_name: CC-MAIN-2013-48460    data_files:461      - split: train462        path: data/CC-MAIN-2013-48/*463  - config_name: CC-MAIN-2013-20464    data_files:465      - split: train466        path: data/CC-MAIN-2013-20/*467---468# 🍷 FineWeb469<center>470    <img src="https://huggingface.co/datasets/HuggingFaceFW/admin/resolve/main/fineweb-logo.png" alt="FineWeb: The finest collection of data the web has to offer">471</center>472 473> 15 trillion tokens of the finest data the 🌐 web has to offer474 475# Table of Contents476- [🍷 FineWeb](#-fineweb)477   * [What is it?](#what-is-it)478   * [What is being released?](#what-is-being-released)479   * [Changelog](#changelog)480   * [How to download and use 🍷 FineWeb](#how-to-download-and-use-🍷-fineweb)481      + [Using 🏭 `datatrove`](#using-datatrove)482      + [Using `huggingface_hub`](#using-huggingface_hub)483      + [Using `datasets`](#using-datasets)484   * [Breakdown by dump/crawl](#breakdown-by-dumpcrawl)485   * [Dataset performance evaluation and ablations](#dataset-performance-evaluation-and-ablations)486      + [Hyper-parameters for ablation models](#hyper-parameters-for-ablation-models)487      + [Ablation evaluation benchmarks](#ablation-evaluation-benchmarks)488      + [Comparison with other datasets](#comparison-with-other-datasets)489- [Dataset card for 🍷 FineWeb](#dataset-card-for-🍷-fineweb)490   * [Dataset Summary](#dataset-summary)491   * [Dataset Structure](#dataset-structure)492      + [Data Instances](#data-instances)493      + [Data Fields](#data-fields)494      + [Data Splits](#data-splits)495   * [Dataset Creation](#dataset-creation)496      + [Curation Rationale](#curation-rationale)497      + [Source Data](#source-data)498      + [Data processing steps](#data-processing-steps)499      + [Annotations](#annotations)500      + [Personal and Sensitive Information](#personal-and-sensitive-information)501   * [Considerations for Using the Data](#considerations-for-using-the-data)502      + [Social Impact of Dataset](#social-impact-of-dataset)503      + [Discussion of Biases](#discussion-of-biases)504      + [Other Known Limitations](#other-known-limitations)505   * [Additional Information](#additional-information)506      + [Licensing Information](#licensing-information)507      + [Future work](#future-work)508      + [Citation Information](#citation-information)509 510## What is it?511 512The 🍷 FineWeb dataset consists of more than **18.5T tokens** (originally 15T tokens) of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the 🏭 [`datatrove`](https://github.com/huggingface/datatrove/) library, our large scale data processing library. 513 514🍷 FineWeb was originally meant to be a fully open replication of 🦅 [RefinedWeb](https://huggingface.co/papers/2306.01116), with a release of the **full dataset** under the **ODC-By 1.0 license**. However, by carefully adding additional filtering steps, we managed to push the performance of 🍷 FineWeb well above that of the original 🦅 RefinedWeb, and models trained on our dataset also outperform models trained on other commonly used high quality web datasets (like C4, Dolma-v1.6, The Pile, SlimPajama, RedPajam2) on our aggregate group of [benchmark tasks](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py).515 516That said, we think there is still room for additional filtering and improvement and intend to continue exploring how to improve the dataset quality in coming versions of 🍷 FineWeb.517 518## What is being released?519 520Along with the dataset, which includes all CommonCrawl dumps since 2013, we also share all the code needed to fully reproduce our processing setup using the 🏭 [`datatrove`](https://github.com/huggingface/datatrove/) library [here](https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py). To enable full replication of our results, we have also published the small ablation models we have trained using [`nanotron`](https://github.com/huggingface/nanotron/) to validate the dataset and compare it with other reference datasets. You will find them [here](https://huggingface.co/collections/HuggingFaceFW/ablation-models-662457b0d213e8c14fe47f32), with checkpoints every 1000 steps. We have also published our evaluation results [here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/eval_results.csv). Our evaluation setup is available [here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py).521 522You will find details on the different processing decisions we took and some interesting explorations of deduplication methods on our [blogpost](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1).523 524## Changelog525_Previous versions remain available in the branch `version name`._526 527- **v1.4.0 (11-07-2025):** Added 6 new snapshots: `CC-MAIN-2025-05`, `CC-MAIN-2025-08`, `CC-MAIN-2025-13`, `CC-MAIN-2025-18`, `CC-MAIN-2025-21`, and `CC-MAIN-2025-26` (January to June 2025)528- **v1.3.0 (31-01-2025):** Fixed an issue with some dumps where some documents hadn't been processed: `CC-MAIN-2024-10`, `CC-MAIN-2024-18`, `CC-MAIN-2024-22`, `CC-MAIN-2024-26`, `CC-MAIN-2024-30`, `CC-MAIN-2024-33`, `CC-MAIN-2024-38`, `CC-MAIN-2024-42`, `CC-MAIN-2024-46` -- they now contain more data (~400B additional tokens). We also removed specific domains in response to a [C&D notice](https://huggingface.co/datasets/huggingface-legal/takedown-notices/blob/main/2025/2025-01-22-Torstar.md).529- **v1.2.0 (03-01-2025):** Added 8 new snapshots: `CC-MAIN-2024-22`, `CC-MAIN-2024-26`, `CC-MAIN-2024-30`, `CC-MAIN-2024-33`, `CC-MAIN-2024-38`, `CC-MAIN-2024-42`, `CC-MAIN-2024-46`, `CC-MAIN-2024-51`, covering May to December 2024.530- **v1.1.0 (31-05-2024):** We reprocessed and reuploaded 11 dumps, `CC-MAIN-2021-49` to `CC-MAIN-2023-40`, as we found a bug on their deduplication. We also added the most recent dump: `CC-MAIN-2024-18`, crawled over April 2024. Expect a small perf improvement531- **v1.0.0 (21-04-2024):** Initial version532 533## How to download and use 🍷 FineWeb534 535You can load the full dataset or a specific crawl/dump (see table below). Dumps have the format `CC-MAIN-(year)-(week number)`.536 537### (Smaller) sample versions538Along with config `default` (all the data), and the configs for each individual dump, you can also download the following configs:539- `sample-350BT`: a subset randomly sampled from the whole dataset of around 350B gpt2 tokens (388GB)540- `sample-100BT`: a subset randomly sampled from the whole dataset of around 100B gpt2 tokens (277.4GB)541- `sample-10BT`: a subset randomly sampled from the whole dataset of around 10B gpt2 tokens (27.6GB)542 543`sample-10B` was sampled from `sample-100B` which in turn was sampled from `sample-350BT`.544 545### Using 🏭 [`datatrove`](https://github.com/huggingface/datatrove/)546 547```python548from datatrove.pipeline.readers import ParquetReader549 550# limit determines how many documents will be streamed (remove for all)551# to fetch a specific dump: hf://datasets/HuggingFaceFW/fineweb/data/CC-MAIN-2024-10552# replace "data" with "sample/100BT" to use the 100BT sample553data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb/data", limit=1000) 554for document in data_reader():555    # do something with document556    print(document)557 558###############################    559# OR for a processing pipeline:560###############################561 562from datatrove.executor import LocalPipelineExecutor563from datatrove.pipeline.readers import ParquetReader564from datatrove.pipeline.filters import LambdaFilter565from datatrove.pipeline.writers import JsonlWriter566 567pipeline_exec = LocalPipelineExecutor(568    pipeline=[569        # replace "data/CC-MAIN-2024-10" with "sample/100BT" to use the 100BT sample570        ParquetReader("hf://datasets/HuggingFaceFW/fineweb/data/CC-MAIN-2024-10", limit=1000),571        LambdaFilter(lambda doc: "hugging" in doc.text),572        JsonlWriter("some-output-path")573    ],574    tasks=10575)576pipeline_exec.run()577```578 579### Using `huggingface_hub`580 581```python582from huggingface_hub import snapshot_download583folder = snapshot_download(584                "HuggingFaceFW/fineweb", 585                repo_type="dataset",586                local_dir="./fineweb/",587                # replace "data/CC-MAIN-2023-50/*" with "sample/100BT/*" to use the 100BT sample588                allow_patterns="data/CC-MAIN-2023-50/*")589```590 591For faster downloads, make sure to install `pip install huggingface_hub[hf_transfer]` and set the environment variable `HF_HUB_ENABLE_HF_TRANSFER=1`.592 593### Using `datasets`594 595```python596from datasets import load_dataset597# use name="sample-10BT" to use the 10BT sample598fw = load_dataset("HuggingFaceFW/fineweb", name="CC-MAIN-2024-10", split="train", streaming=True)599```600 601## Breakdown by dump/crawl602 603| Dump | Time period | Disk size (GB) | gpt2 tokens (billions) |604| --- | --- |----------------|------------------------|              605| CC-MAIN-2025-26 | June 2025 | 419.6 | 152.4 |606| CC-MAIN-2025-21 | May 2025 | 462.8 | 168.1 |607| CC-MAIN-2025-18 | April 2025 | 506.8 | 184.2 |608| CC-MAIN-2025-13 | March 2025 | 491.1 | 178.5 |609| CC-MAIN-2025-08 | February 2025 | 472.0 | 171.6 |610| CC-MAIN-2025-05 | January 2025 | 558.8 | 203.5 |611| CC-MAIN-2024-51 | December 2024             | 362.6          | 131.2                  |612| CC-MAIN-2024-46 | November 2024             | 474.6          | 172.9                  |613| CC-MAIN-2024-42 | October 2024             | 434.0          | 158.1                  |614| CC-MAIN-2024-38 | September 2024             | 506.2          | 184.6                  |615| CC-MAIN-2024-33 | August 2024             | 400.6          | 145.9                  |616| CC-MAIN-2024-30 | July 2024             | 451.3          | 164.6                  |617| CC-MAIN-2024-26 | June 2024             | 496.5          | 181.2                  |618| CC-MAIN-2024-22 | May 2024             | 499.7          | 182.5                  |619| CC-MAIN-2024-18 | April 2024             | 520.6          | 190.3                  |620| CC-MAIN-2024-10 | February/March 2024    | 581.3          | 212.6                  |621| CC-MAIN-2023-50 | November/December 2023 | 650.0          | 239.7                  |622| CC-MAIN-2023-40 | September/October 2023 | 668.7          | 252.0                  |623| CC-MAIN-2023-23 | May/June 2023          | 654.4          | 249.2                  |624| CC-MAIN-2023-14 | March/April 2023       | 621.3          | 236.5                  |625| CC-MAIN-2023-06 | January/February 2023  | 621.9          | 233.9                  |626| CC-MAIN-2022-49 | November/December 2022 | 631.2          | 237.5                  |627| CC-MAIN-2022-40 | September/October 2022 | 606.4          | 228.7                  |628| CC-MAIN-2022-33 | August 2022            | 434.6          | 163.5                  |629| CC-MAIN-2022-27 | June/July 2022         | 574.9          | 216.1                  |630| CC-MAIN-2022-21 | May 2022               | 646.4          | 242.7                  |631| CC-MAIN-2022-05 | January 2022           | 520.1          | 195.4                  |632| CC-MAIN-2021-49 | November/December 2021 | 413.7          | 155.5                  |633| CC-MAIN-2021-43 | October 2021           | 601.5          | 221.0                  |634| CC-MAIN-2021-43 | October 2021 | 601.5          | 221.0                  |635| CC-MAIN-2021-39 | September 2021 | 518.9          | 190.6                  |636| CC-MAIN-2021-31 | July/August 2021 | 593.9          | 217.7                  |637| CC-MAIN-2021-25 | June 2021 | 424.4          | 155.7                  |638| CC-MAIN-2021-21 | May 2021 | 455.9          | 167.4                  |639| CC-MAIN-2021-17 | April 2021 | 556.0          | 204.1                  |640| CC-MAIN-2021-10 | February/March 2021 | 463.2          | 169.6                  |641| CC-MAIN-2021-04 | January 2021 | 562.4          | 205.4                  |642| CC-MAIN-2020-50 | November/December 2020 | 422.8          | 154.3                  |643| CC-MAIN-2020-45 | October 2020 | 426.9          | 155.8                  |644| CC-MAIN-2020-40 | September 2020 | 555.5          | 202.4                  |645| CC-MAIN-2020-34 | August 2020 | 379.6          | 138.7                  |646| CC-MAIN-2020-29 | July 2020 | 489.6          | 178.7                  |647| CC-MAIN-2020-24 | May/June 2020 | 398.7          | 145.1                  |648| CC-MAIN-2020-16 | March/April 2020 | 454.0          | 165.6                  |649| CC-MAIN-2020-10 | February 2020 | 369.6          | 134.7                  |650| CC-MAIN-2020-05 | January 2020 | 483.3          | 176.4                  |651| CC-MAIN-2019-51 | December 2019 | 359.3          | 130.9                  |652| CC-MAIN-2019-47 | November 2019 | 395.4          | 144.0                  |653| CC-MAIN-2019-43 | October 2019 | 422.3          | 153.9                  |654| CC-MAIN-2019-39 | September 2019 | 394.4          | 143.7                  |655| CC-MAIN-2019-35 | August 2019 | 454.2          | 165.4                  |656| CC-MAIN-2019-30 | July 2019 | 416.6          | 151.5                  |657| CC-MAIN-2019-26 | June 2019 | 412.9          | 150.1                  |658| CC-MAIN-2019-22 | May 2019 | 432.8          | 157.4                  |659| CC-MAIN-2019-18 | April 2019 | 426.7          | 155.3                  |660| CC-MAIN-2019-13 | March 2019 | 417.8          | 152.1                  |661| CC-MAIN-2019-09 | February 2019 | 467.2          | 169.9                  |662| CC-MAIN-2019-04 | January 2019 | 438.1          | 158.7                  |663| CC-MAIN-2018-51 | December 2018 | 498.6          | 180.8                  |664| CC-MAIN-2018-47 | November 2018 | 437.7          | 158.9                  |665| CC-MAIN-2018-43 | October 2018 | 468.8          | 169.9                  |666| CC-MAIN-2018-39 | September 2018 | 429.2          | 155.2                  |667| CC-MAIN-2018-34 | August 2018 | 408.2          | 148.0                  |668| CC-MAIN-2018-30 | July 2018 | 501.5          | 181.4                  |669| CC-MAIN-2018-26 | June 2018 | 467.5          | 170.0                  |670| CC-MAIN-2018-22 | May 2018 | 398.6          | 144.2                  |671| CC-MAIN-2018-17 | April 2018 | 435.1          | 158.1                  |672| CC-MAIN-2018-13 | March 2018 | 471.5          | 171.5                  |673| CC-MAIN-2018-09 | February 2018 | 490.2          | 178.0                  |674| CC-MAIN-2018-05 | January 2018 | 493.5          | 180.7                  |675| CC-MAIN-2017-51 | December 2017 | 442.6          | 161.5                  |676| CC-MAIN-2017-47 | November 2017 | 457.9          | 167.1                  |677| CC-MAIN-2017-43 | October 2017 | 535.6          | 194.9                  |678| CC-MAIN-2017-39 | September 2017 | 444.5          | 162.3                  |679| CC-MAIN-2017-34 | August 2017 | 503.2          | 183.4                  |680| CC-MAIN-2017-30 | July 2017 | 439.2          | 161.2                  |681| CC-MAIN-2017-26 | June 2017 | 491.5          | 179.8                  |682| CC-MAIN-2017-22 | May 2017 | 441.0          | 161.5                  |683| CC-MAIN-2017-17 | April 2017 | 596.8          | 218.6                  |684| CC-MAIN-2017-13 | March 2017 | 579.8          | 212.1                  |685| CC-MAIN-2017-09 | February 2017 | 492.2          | 180.2                  |686| CC-MAIN-2017-04 | January 2017 | 474.3          | 174.4                  |687| CC-MAIN-2016-50 | December 2016 | 448.9          | 165.4                  |688| CC-MAIN-2016-44 | October 2016 | 467.8          | 172.0                  |689| CC-MAIN-2016-40 | September 2016 | 386.1          | 142.8                  |690| CC-MAIN-2016-36 | August 2016 | 339.6          | 126.3                  |691| CC-MAIN-2016-30 | July 2016 | 346.0          | 128.4                  |692| CC-MAIN-2016-26 | June 2016 | 256.5          | 95.5                   |693| CC-MAIN-2016-22 | May 2016 | 310.9          | 115.4                  |694| CC-MAIN-2016-18 | April 2016 | 298.1          | 110.8                  |695| CC-MAIN-2016-07 | February 2016 | 342.7          | 127.2                  |696| CC-MAIN-2015-48 | November 2015 | 353.9          | 131.3                  |697| CC-MAIN-2015-40 | September 2015 | 284.0          | 105.5                  |698| CC-MAIN-2015-35 | August 2015 | 359.4          | 133.2                  |699| CC-MAIN-2015-32 | July 2015 | 352.4          | 130.1                  |700| CC-MAIN-2015-27 | June 2015 | 335.5          | 124.0                  |701| CC-MAIN-2015-22 | May 2015 | 380.2          | 140.4                  |702| CC-MAIN-2015-18 | April 2015 | 389.0          | 143.8                  |703| CC-MAIN-2015-14 | March 2015 | 337.5          | 124.5                  |704| CC-MAIN-2015-11 | February 2015 | 361.4          | 133.3                  |705| CC-MAIN-2015-06 | January 2015 | 356.1          | 131.3                  |706| CC-MAIN-2014-52 | December 2014 | 388.5          | 143.3                  |707| CC-MAIN-2014-49 | November 2014 | 319.9          | 117.7                  |708| CC-MAIN-2014-42 | October 2014 | 371.1          | 136.4                  |709| CC-MAIN-2014-41 | September 2014 | 408.1          | 150.2                  |710| CC-MAIN-2014-35 | August 2014 | 395.7          | 145.6                  |711| CC-MAIN-2014-23 | July 2014 | 425.0          | 156.5                  |712| CC-MAIN-2014-15 | April 2014 | 369.1          | 135.7                  |713| CC-MAIN-2014-10 | March 2014 | 396.2          | 146.2                  |714| CC-MAIN-2013-48 | Winter 2013 | 396.8          | 145.9                  |715| CC-MAIN-2013-20 | Summer 2013 | 393.9          | 144.5                  |716| Total |  | 50,446.9       | 18,527.0               |717 718## Dataset performance evaluation and ablations719 720We conducted our dataset performance ablations and evaluations by training a series of 1.8B parameters models on 27 billion tokens. To compare 🍷 FineWeb with other datasets, we also trained one of these 1.8B models per target dataset, on 350 billion tokens sampled from it (or the entire dataset when its size was < 350 billion tokens).721 722### Hyper-parameters for ablation models723 724The detailed configurations for training the 1.8B parameters ablation model can be found here (link will be added soon).725 726### Ablation evaluation benchmarks727 728To conduct the ablations for each of our dataset filtering choices, we selected a set of benchmarks which we identified as “high-signal” benchmarks. These benchmarks were selected according to the following criteria:729 730- small variance between runs trained on different samplings of the same dataset731- performance increasing monotically during training (or close)732- separation between runs on datasets of known quality (C4, The Pile, RedPajama) higher than the variance between runs with various modeling/data seeds733 734We used the following list of benchmark for our ablation runs:735 736- commonsense_qa (acc/acc_norm)737- hellaswag (acc/acc_norm)738- openbookqa (acc/acc_norm)739- piqa (acc/acc_norm)740- siqa (acc/acc_norm)741- winogrande (acc/acc_norm)742- arc (acc/acc_norm)743- mmlu (acc/acc_norm)744 745To compare runs we consider an aggregate score, the average of the scores for these tasks.746 747The prompts for all these benchmarks are formatted in order to compute and compare the log-likelihood of the full answers for each multiple choice question. All the implementation details for the benchmarks are available in `lighteval` [here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py).748 749### Comparison with other datasets750 751We compared 🍷 FineWeb with the following datasets:752 753- [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)754- [C4](https://huggingface.co/datasets/allenai/c4)755- [Dolma v1.6](https://huggingface.co/datasets/allenai/dolma) (the CommonCrawl part)756- [The Pile](https://huggingface.co/datasets/EleutherAI/pile)757- [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B)758- [RedPajama2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2) (deduplicated)759 760You will find these models on [this collection](https://huggingface.co/collections/HuggingFaceFW/ablation-models-662457b0d213e8c14fe47f32). We have uploaded checkpoints at every 1000 training steps. You will also find our full [evaluation results here](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/eval_results.csv).761 762<center>763    <img src="https://huggingface.co/datasets/HuggingFaceFW/admin/resolve/main/fineweb-ablations.png" alt="ablations">764</center>765 766_Note:_ The plot is smoothed by averaging 5k steps in a rolling window.767 768# Dataset card for 🍷 FineWeb769 770## Dataset Description771 772- **Homepage and Repository:** [https://huggingface.co/datasets/HuggingFaceFW/fineweb](https://huggingface.co/datasets/HuggingFaceFW/fineweb)773- **Point of Contact:** please create a discussion on the Community tab774- **License:** Open Data Commons Attribution License (ODC-By) v1.0775 776### Dataset Summary777 778This dataset was created by processing 96 [CommonCrawl](https://commoncrawl.org/) dumps comprising web data crawled from the summer of 2013 to April of 2024. 🍷 FineWeb includes a variety of domains and topics in English and is primarily intended to be used as a research artifact on public data in the context of pretraining dataset for large language models. The CommonCrawl data was carefully processed, filtered and deduplicated with the 🏭 [`datatrove`](https://github.com/huggingface/datatrove/) library, resulting in the largest publicly available clean LLM pretraining dataset, counting around 15 trillion tokens (gpt2 tokenizer).779 780## Dataset Structure781 782### Data Instances783 784The following is an example sample from the dataset. It is part of the `CC-MAIN-2021-43` and was crawled on `2021-10-15T21:20:12Z`.785 786```json787{788   "text": "This is basically a peanut flavoured cream thickened with egg yolks and then set into a ramekin on top of some jam. Tony, one of the Wedgwood chefs, suggested sprinkling on some toasted crushed peanuts at the end to create extra crunch, which I thought was a great idea. The result is excellent.",789   "id": "<urn:uuid:e5a3e79a-13d4-4147-a26e-167536fcac5d>",790   "dump": "CC-MAIN-2021-43",791   "url": "<http://allrecipes.co.uk/recipe/24758/peanut-butter-and-jam-creme-brulee.aspx?o_is=SimilarRecipes&o_ln=SimRecipes_Photo_7>",792   "date": "2021-10-15T21:20:12Z",793   "file_path": "s3://commoncrawl/crawl-data/CC-MAIN-2021-43/segments/1634323583083.92/warc/CC-MAIN-20211015192439-20211015222439-00600.warc.gz",794   "language": "en",795   "language_score": 0.948729,796   "token_count": 69797}798```799 800### Data Fields801 802- `text` (string): the main text content803- `id` (string): original unique identifier for this sample from CommonCrawl804- `dump` (string): the CommonCrawl dump this sample was a part of805- `url` (string): url to the original page where `text` was present806- `date` (string): crawl date (from CommonCrawl)807- `file_path` (string): s3 path for the individual CommonCrawl warc file containing this sample808- `language` (string): `en` for all the samples in this dataset809- `language_score` (float): language prediction score (`0.01.0`) as reported by the [fastText language classifier](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/filters/language_filter.py)810- `token_count` (int): number of tokens when applying the `gpt2` tokenizer to this sample811 812### Data Splits813 814The `default` subset includes the entire dataset. If you would like to only use the data from a particular [CommonCrawl dump](https://commoncrawl.org/overview), you can use the dump name as a subset. You will find the full list of available dumps on the table above.815From experiments we have run, not all dumps give the same performance. For relatively small trainings (<550 billion tokens) we recommend using the recent `CC-MAIN-2023-50`, `CC-MAIN-2024-10` and `CC-MAIN-2024-18`. 816 817## Dataset Creation818 819### Curation Rationale820 821While multiple open-weights models have regularly been released in recent months, these releases often do not include the model's training data. With 🍷 FineWeb we aim to provide the open source community with a very large clean pretraining dataset that can be used to push the envelope on truly open source models (open source models where data is also released). 822 823### Source Data824 825The source data consists of webpages crawled by the CommonCrawl foundation over the 2013-2024 time period.826 827We then extracted the main page text from the html of each webpage, carefully filtered each sample and deduplicated each individual CommonCrawl dump/crawl.828 829While we originally intended to deduplicate the dataset as a whole, our ablations showed that training on a sampling of individually deduplicated dumps/crawls outperformed training on a sampling of all the dumps/crawls deduplicated together. You will find more details on our [blogpost](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1).830 831### Data processing steps832 833We used the 🏭 `datatrove` library to process the data.834You can find a **working script** that launches the [entire processing pipeline here](https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py).835 836The data processing pipeline consists of:837 8381. [Url Filtering](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/url_filter.py), removing documents originating from Malicious and NSFW websites, using both block-list as well as subwords detection8392. [Trafilatura](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/extractors/trafilatura.py) text extraction on the raw HTML from CommonCrawl’s warc files8403. [FastText LanguageFilter](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/language_filter.py), removing any document with `en` language score lower than **0.65**8414. Quality filtering842    1. [Gopher Repetition /](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/gopher_repetition_filter.py) [Quality](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/gopher_quality_filter.py)843    2. [C4 Quality filters](https://github.com/huggingface/datatrove/blob/9a88bebc86a554f8521faa70b12ad4fa0c227537/src/datatrove/pipeline/filters/c4_quality_filter.py) except `terminal_punct` rule844    3. [FineWeb custom filters](https://github.com/huggingface/datatrove/blob/05194d3960741e7d5c0bd0d6dd69d44514622549/src/datatrove/pipeline/filters/fineweb_quality_filter.py), consisting of heuristics for removing list-like documents, documents with repeated lines and documents with likely wrong line formatting. 8455. [MinHash deduplication](https://github.com/huggingface/datatrove/blob/6daa5e879e06b21e6886b37e2b1be4ae58a658b6/src/datatrove/pipeline/dedup/minhash.py) with each crawl deduplicated individually (5-grams, 14x8 hash functions)8466. [PII Formatting](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/formatters/pii.py) to anonymize email and public IP addresses847 848### Annotations849 850We augment the original samples with the `language`, `language_score` and `token_count` annotations. The language related annotations are automatically generated by our [language filter](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/filters/language_filter.py). `token_count` is generated by [applying the gpt2 tokenizer](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/tokens/counter.py) to the `text` column.851 852### Personal and Sensitive Information853 854We anonymize email addresses and public IP addresses. 855 856For emails, we apply a regex pattern and replace any occurrence of an email address with either `email@example.com` or `firstname.lastname@example.org`. For IP addresses, we also employ a regex pattern and then further filter to only anonymize IP addresses [allocated for public networks](https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml). Matched IP addresses are then replaced with one of the following randomly generated IP addresses, which at the time of dataset creation were not responding to ping requests: `22.214.171.124`, `126.96.36.199`, `188.8.131.52`, `184.108.40.206`, `220.127.116.11`, and `18.104.22.168`. We decided against applying regex patterns for phone numbers due to the high false positive rate.857 858Despite our efforts, given that 🍷 FineWeb is sourced from the internet at large, it is very likely that some personable identifiable information (PII) will be present. If you find your own PII in 🍷 FineWeb and would like it removed, please fill out our [PII removal form](https://forms.gle/VyNT3ZAUPZjPuWp39).859 860## Considerations for Using the Data861 862### Social Impact of Dataset863 864With the release of this dataset we aim to make model training more accessible to the machine learning community at large. 865 866While multiple open-weights models with strong performance have been publicly released in the past, more often than not these releases are not accompanied by the corresponding training dataset. This is unfortunate as the dataset specificities and characteristics have been demonstrated to have a very large impact and role in the performances of the models. As the creation of a high quality training dataset is a fundamental requirement to training an LLM capable of excelling at downstream tasks, with 🍷 FineWeb we (a) not only make the dataset creation process more transparent, by sharing our entire processing setup including the codebase used, we also (b) help alleviate the costs of dataset curation, both in time and in compute, for model creators by publicly releasing our dataset with the community.867 868### Discussion of Biases869 870Efforts were made to minimize the amount of NSFW and toxic content present in the dataset by employing filtering on the URL level. However, there are still a significant number of documents present in the final dataset that could be considered toxic or contain harmful content. As 🍷 FineWeb was sourced from the web as a whole, any harmful biases typically present in it may be reproduced on our dataset.871 872We deliberately avoided using machine learning filtering methods that define text quality based on the similarity to a “gold” source such as wikipedia or toxicity classifiers as these methods have been known to [disproportionately remove content in specific dialects](https://aclanthology.org/D16-1120/) and [overclassify as toxic text related to specific social identities](https://arxiv.org/pdf/2109.07445.pdf), respectively.873 874### Other Known Limitations875 876As a consequence of some of the filtering steps applied, it is likely that code content is not prevalent in our dataset. If you are training a model that should also perform code tasks, we recommend you use 🍷 FineWeb with a code dataset, such as [The Stack v2](https://huggingface.co/datasets/bigcode/the-stack-v2). You should also probably consider complementing 🍷 FineWeb with specialized curated sources (such as Wikipedia, for example) as they will likely have better formatting than the wikipedia content included in 🍷 FineWeb (we did not tailor the processing to individual websites).877 878## Additional Information879 880### Licensing Information881 882The dataset is released under the **Open Data Commons Attribution License (ODC-By) v1.0** [license](https://opendatacommons.org/licenses/by/1-0/). The use of this dataset is also subject to [CommonCrawl's Terms of Use](https://commoncrawl.org/terms-of-use).883 884### Future work885 886We plan to not only continue but also expand our efforts to create open-source high quality training datasets and to improve 🍷 FineWeb itself in future iterations.887 888## Citation Information889Paper on [arXiv](https://arxiv.org/abs/2406.17557)890```891@inproceedings{892  penedo2024the,893  title={The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale},894  author={Guilherme Penedo and Hynek Kydl{\'\i}{\v{c}}ek and Loubna Ben allal and Anton Lozhkov and Margaret Mitchell and Colin Raffel and Leandro Von Werra and Thomas Wolf},895  booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},896  year={2024},897  url={https://openreview.net/forum?id=n6SCkn2QaG}898}899```