datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-2
🥂 FineWeb2
A sparkling update with 1000s of languages
What is it?
This is the second iteration of the popular 🍷 FineWeb dataset, bringing high quality pretraining data to over 1000 🗣️ languages.
The 🥂 FineWeb2 dataset is fully reproducible, available under the permissive ODC-By 1.0 license and extensively validated through hundreds of ablation experiments.
In particular, on the set of 9 diverse languages we used to guide our processing decisions, 🥂… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-2.fineweb-2-duckdbs
DuckDB datasets for (dump, id) querying on FineWeb 2
This repo contains some DuckDB databases to check whether a given WARC UID exists in a FineWeb-2 dump. Usage example
is given below, but note especially that if you are using URNs (likely, if you are working with CommonCrawl data),
then you first have to extract the UID (the id column is of type UUID in the databases).
Download
All files:
huggingface-cli download BramVanroy/fineweb-2-duckdbs --local-dir… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/fineweb-2-duckdbs.mmBERT-pretrain-p2-fineweb2-remaining
mmBERT Pre-training Data P2
Phase 1 of 3: Diverse multilingual pre-training data mixture (trained for 2.3T tokens) used to train the mmBERT model suite.
NOTE: this is only P2 of the pre-training data due to HF limits, you need to download and combine all three into one folderThis dataset contains the pre-training phase data used to train all mmBERT encoder models. The data is provided in MDS format ready for use with Composer and the ModernBERT training repository.… See the full description on the dataset page: https://huggingface.co/datasets/orionweller/mmBERT-pretrain-p2-fineweb2-remaining.FineWeb2-HQ
FineWeb2-HQ
Dataset summary
FineWeb2-HQ is a high-quality, model-filtered pretraining dataset derived as a subset of FineWeb2, spanning 20 languages. It enables around 6x faster pretraining compared to the base dataset. FineWeb2-HQ was created by selecting the top 10% quality documents of FineWeb2 in each language, based on scores assigned by a deep learning classifier trained to identify structured and knowledge-rich samples using XLM-RoBERTa embeddings.
Validation… See the full description on the dataset page: https://huggingface.co/datasets/epfml/FineWeb2-HQ.FineWeb2-embedded
FineWeb2-embedded
Dataset summary
FineWeb2-embedded is an extension of the FineWeb2 dataset, annotated with document-level XLM-RoBERTa embeddings for 20 languages, making the dataset useful for a variety of tasks, including document clustering, filtering, and other multilingual research.
Since XLM-RoBERTa has a sequence length limit of 512 tokens, each document's embeddings are obtained by mean-pooling 512 token chunks of the XLM-RoBERTa output. Therefore, longer texts… See the full description on the dataset page: https://huggingface.co/datasets/epfml/FineWeb2-embedded.FineWeb2-mds-tokenized-v2FineWeb2-mds-tokenizedFineWeb2-mds-tokenized-4096fineweb-2-sentence-splitFineweb 2 split into sentences. Instances per languages were sampled by us to balance the data w.r.t. Fineweb-edu.
To split the text into sentences we used the sat3-l model from the wtpsplit library.
We fix a sentence threshold of 0.02 and a maximum sentence length of 256.
If you use this dataset, you should cite:
@misc{penedo2025fineweb2pipelinescale,
title={FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language},
author={Guilherme Penedo and… See the full description on the dataset page: https://huggingface.co/datasets/mimir-lcm/fineweb-2-sentence-split.FineWeb2-mds-tokenized-1024mmBERT-pretrain-p1-fineweb2-langs
mmBERT Pre-training Data P1
Phase 1 of 3: Diverse multilingual pre-training data mixture (trained for 2.3T tokens) used to train the mmBERT model suite.
NOTE: this is only P1 of the pre-training data due to HF limits, you need to download and combine all three into one folderThis dataset contains the pre-training phase data used to train all mmBERT encoder models. The data is provided in MDS format ready for use with Composer and the ModernBERT training repository.… See the full description on the dataset page: https://huggingface.co/datasets/orionweller/mmBERT-pretrain-p1-fineweb2-langs.FineWeb2-mds-tokenized-v2-1024FineWeb2-vie-mdsfineweb-2-turkish-categorized
What is this
THis is the categorized version of the Turkish subset of the fineweb-2 dataset.
It is an ongoing effort, and the details will be added soon with the rest of the dataset.
fineweb-2-edu-japanese
🍷 FineWeb2 Edu Japanese: High-Quality Educational Japanese Dataset
This dataset consists of 120 million texts (approximately 89.3B tokens) filtered from the 376 million Japanese texts in FineWeb2 that were deemed educational. The following subsets are also provided:
default: Approximately 120M texts (120 million texts) totaling around 89.3B tokens
sample_10BT: A random sample of about 10B tokens from the default dataset
small_tokens: Data composed solely of texts with 512 tokens… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/fineweb-2-edu-japanese.fineweb-2-edu-korean-rawHuggingFaceFW/fineweb-2 (v2.1.0)
It took about 9 hours on A100 80gbx4 to process the dataset.
FineWeb2-mds-tokenized-v2-4096fineweb-2-dutchfineweb2-bagaco
Bagaço 🍷🇵🇹
Bagaço is a pretraining dataset for European Portuguese. It filters the Fineweb2 dataset to URLs from Portuguese domains (e.g., .pt/). Each document is classified into one of 9 categories and scored for educational quality.
Filtering
Source: HuggingFaceFW/fineweb-2, subset por_Latn, split train
Filter: URLs containing .pt/ (Portuguese top-level domain)
Document classification
Each document is classified into one of 9 categories: Society, Arts… See the full description on the dataset page: https://huggingface.co/datasets/duarteocarmo/fineweb2-bagaco.FineWeb2024
FineWeb-Edu 2024 — Cleaned and Shuffled
This dataset is a year-specific, cleaned, shuffled, and sharded release derived from HuggingFaceFW/fineweb-edu. It contains English educational web text collected in Common Crawl snapshots whose dump identifier belongs to 2024.
This is not a news-only dataset. The year refers to the Common Crawl capture year, not necessarily the page's publication year.
Dataset summary
Item
Value
Year
2024
Rows
162,500,784… See the full description on the dataset page: https://huggingface.co/datasets/Yahoo-Finance-News/FineWeb2024.FineWeb2-ko-synth
Citation
@misc{KORMo,
author = {Minjun Kim, Hyeonseok Lim, Hangyeol Yoo, Inho Won, Seungwoo Song, Minkyung Cho, Junghun Yuk, Changsu Choi, Dongjae Shin, Huije Lee, Hoyun Song, Alice Oh and KyungTae Lim},
title = {KORMo: Korean Open Reasoning Model for Everyone},
year = {2025},
publisher = {GitHub},
journal = {Technical Report},
paperLink = {\url{https://arxiv.org/abs/2510.09426}},
},
}
fineweb-2-trimming
Description
Version of FineWeb2 where only 124 languages were kept.For each of them we kept the first 200,000 texts (less if there are not as many available for a given language).
The purpose of this dataset is to offer a light version (only 44GB against 8.67 TB for the original dataset) in order to be able to trim models.
For more information on the trimming method, we invite you to consult this blog post.
Citations
FineWeb-2… See the full description on the dataset page: https://huggingface.co/datasets/lbourdois/fineweb-2-trimming.FineWeb2-HQ-zh-text
Original Database
https://huggingface.co/datasets/epfml/FineWeb2-HQ
Work
keep cmn_Hani with text field
fineweb2-arb-edufineweb-2fineweb2-hq-turkishFineWeb2025
FineWeb-Edu 2025 — Cleaned and Shuffled
This dataset is a year-specific, cleaned, shuffled, and sharded release derived from HuggingFaceFW/fineweb-edu. It contains English educational web text collected in Common Crawl snapshots whose dump identifier belongs to 2025.
This is not a news-only dataset. The year refers to the Common Crawl capture year, not necessarily the page's publication year.
Dataset summary
Item
Value
Year
2025
Rows
99,022,205… See the full description on the dataset page: https://huggingface.co/datasets/Yahoo-Finance-News/FineWeb2025.fineweb-2-slv-edufineweb2-romanian-shardsFineWeb-2023
FineWeb-Edu 2023 — Cleaned and Shuffled
This dataset is a year-specific, cleaned, shuffled, and sharded release derived from HuggingFaceFW/fineweb-edu. It contains English educational web text collected in Common Crawl snapshots whose dump identifier belongs to 2023.
This is not a news-only dataset. The year refers to the Common Crawl capture year, not necessarily the page's publication year.
Dataset summary
Item
Value
Year
2023
Rows
104,280,950… See the full description on the dataset page: https://huggingface.co/datasets/Yahoo-Finance-News/FineWeb-2023.