datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-edu
📚 FineWeb-Edu
1.3 trillion tokens of the finest educational data the 🌐 web has to offer
Paper: https://arxiv.org/abs/2406.17557
What is it?
📚 FineWeb-Edu dataset consists of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version.
To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by LLama3-70B-Instruct. We… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu.fineweb-edu-translated
Helsinki-NLP/fineweb-edu-translated
fineweb-edu-tanslated is a collection of automatically translated documents from fineweb-edu.
Translations are based on OPUS-MT and HPLT-MT models.
The data in v1.0 covers 36,704,000 documents with over 28 billion space-searated tokens of English data translated into 36 languages.
The total v1.0 data set includes over 960 billion tokens and the translated documents are aligned across all languages.
In the v1.1 release, additional translations… See the full description on the dataset page: https://huggingface.co/datasets/Helsinki-NLP/fineweb-edu-translated.fineweb-edu-fortified
Fineweb-Edu-Fortified
The composition of fineweb-edu-fortified, produced by automatically clustering a 500k row sample in
Airtrain
What is it?
Fineweb-Edu-Fortified is a dataset derived from
Fineweb-Edu by applying exact-match
deduplication across the whole dataset and producing an embedding for each row. The number of times
the text from each row appears is also included as a count column. The embeddings were produced
using TaylorAI/bge-micro
Fineweb and… See the full description on the dataset page: https://huggingface.co/datasets/airtrain-ai/fineweb-edu-fortified.Fineweb-Edu-Chinese-V2.1
Chinese Fineweb Edu Dataset V2.1 [中文] [English]
[OpenCSG Community] [👾github] [wechat] [Twitter]
📖Technical Report
The Chinese Fineweb Edu Dataset V2.1 is an enhanced version of the V2 dataset, designed specifically for natural language processing (NLP) tasks in the education sector. This version introduces two new data sources, map-cc and opencsg-cc, and retains data with scores ranging from 2 to 3. The dataset entries are organized into different… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/Fineweb-Edu-Chinese-V2.1.fineweb-edu-score-2
📚 FineWeb-Edu-score-2
1.3 trillion tokens of the finest educational data the 🌐 web has to offer
What is it?
📚 FineWeb-Edu dataset consists of 1.3T tokens (FineWeb-Edu) and 5.4T tokens of educational web pages filtered from 🍷 FineWeb dataset. This is the 5.4 trillion version.
Note: this version uses a lower educational score threshold = 2, which results in more documents, but lower quality compared to the 1.3T version. For more details check the… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-score-2.chinese-fineweb-edu
This version is deprecated. We recommend you to use the newest version Fineweb-edu-chinese-v2.1 !
Chinese Fineweb Edu Dataset [中文] [English]
[OpenCSG Community] [👾github] [wechat] [Twitter]
📖Technical Report
Chinese Fineweb Edu dataset is a meticulously constructed high-quality Chinese pre-training corpus, specifically designed for natural language processing tasks in the education domain. This dataset undergoes a rigorous selection and… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/chinese-fineweb-edu.Fineweb-Edu-Chinese-V2.2
Chinese Fineweb Edu Dataset V2.2 (Instruct & Pre-train)
[[中文]] | [[English]]
OpenCSG Community | 👾 GitHub | 📖 Technical Report
Dataset Introduction: Filling the Data Puzzle for Chinese Education LLMs
Chinese Fineweb Edu Dataset V2.2is a rare high-quality dataset in the open-source community that covers the full process from Pre-training to Supervised Fine-Tuning (SFT) for the Chinese education domain.
This project aims to solve the core pain point of… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/Fineweb-Edu-Chinese-V2.2.fineweb-edu
📚 FineWeb-Edu
1.3 trillion tokens of the finest educational data the 🌐 web has to offer
Paper: https://arxiv.org/abs/2406.17557
What is it?
📚 FineWeb-Edu dataset consists of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version.
To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by LLama3-70B-Instruct. We… See the full description on the dataset page: https://huggingface.co/datasets/chilomax/fineweb-edu.chinese-fineweb-edu-v2
This version is deprecated. We recommend you to use the newest version Fineweb-edu-chinese-v2.1 !
Chinese Fineweb Edu Dataset V2 [中文] [English]
[OpenCSG Community] [👾github] [wechat] [Twitter]
📖Technical Report
Chinese Fineweb Edu Dataset V2 is a comprehensive upgrade of the original Chinese Fineweb Edu, designed and optimized for natural language processing (NLP) tasks in the education sector. This high-quality Chinese pretraining dataset has… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/chinese-fineweb-edu-v2.mga-fineweb-edu
Massive Genre-Audience Augment Fineweb-Edu Corpus
This dataset is a synthetic pretraining corpus described in paper Reformulation for Pretraining Data Augmentation.
Overview of synthesis framework. Our method expands the original corpus through a two-stage synthesis process.
Each document is reformulated to 5 new documents, achieving 3.9× token number expansion while maintaining diversity through massive (genre, audience) pairs.
We build MGACorpus based on SmolLM Corpus… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/mga-fineweb-edu.FineWeb-Edu-10B-Tokens-NPY
FineWeb-Edu 10B Tokens (NPY Format)
数据集概述
这是一个预处理好的教育文本数据集，包含约100亿个tokens，专门为训练小型语言模型（如GPT-2 124M）而设计。数据来源于高质量的FineWeb-Edu数据集，已经使用GPT-2的tiktoken分词器进行预处理，并保存为numpy格式以提高训练效率。
Followed by Let's reproduce GPT-2 (124M). Thanks to Andrej Karpathy!!!
🎯 适用场景
小型语言模型训练：特别适合GPT-2 124M/350M等参数规模的模型
教育研究：高质量教育内容，适合教学和学术研究
快速原型开发：预处理完成，可直接用于训练间
📊 数据统计
总token数量：~10,000,000,000 tokens
分片大小：100M tokens/分片
数据格式：numpy (.npy) uint16数组
分词器：GPT-2 tiktoken
语言：英语… See the full description on the dataset page: https://huggingface.co/datasets/ShallowU/FineWeb-Edu-10B-Tokens-NPY.Fineweb-Edu-Chinese-V2.1
Chinese Fineweb Edu Dataset V2.1 [中文] [English]
[OpenCSG Community] [👾github] [wechat] [Twitter]
📖Technical Report
The Chinese Fineweb Edu Dataset V2.1 is an enhanced version of the V2 dataset, designed specifically for natural language processing (NLP) tasks in the education sector. This version introduces two new data sources, map-cc and opencsg-cc, and retains data with scores ranging from 2 to 3. The dataset entries are organized into different folders… See the full description on the dataset page: https://huggingface.co/datasets/willRD/Fineweb-Edu-Chinese-V2.1.fineweb-edu-zh-chengyu-cpt
Fineweb-Edu Chinese — Chengyu-Tagged Continued-Pretraining Corpus
A 3.74M-document Chinese corpus (~7.8B tokens) for continued pretraining on
cultural knowledge in figurative language, built from the highest-quality
tier of opencsg/Fineweb-Edu-Chinese-V2.1.
Each document is educational Chinese text containing at least one culturally
vetted chengyu, with an appended 【成语注释】 knowledge block listing every
matched idiom's figurative meaning(s) and classical source citation.
This is a… See the full description on the dataset page: https://huggingface.co/datasets/jiviteshjn/fineweb-edu-zh-chengyu-cpt.fineweb-edu-format-topic
FineWeb-Edu w/ Topic and Format Annotations
FineWeb-Edu dataset consists of 1.3T tokens annotated for Topic and Format using wissamantoun/WebOrganizer-TopicClassifier-ModernBERT and wissamantoun/WebOrganizer-FormatClassifier-ModernBERT classifiers.
Similar to WebOrganizer/Corpus-200B but using FineEdu instead of DCLM.
Topic Labels:
Adult
Art & Design
Software Dev.
Crime & Law
Education & Jobs
Hardware
Entertainment
Social Life
Fashion & Beauty
Finance & Business
Food & Dining… See the full description on the dataset page: https://huggingface.co/datasets/wissamantoun/fineweb-edu-format-topic.prepacked-fineweb-edu-llama2-32K-T2048
prepacked-fineweb-edu-llama2-32K-T2048
Pre-tokenized and BOS-aligned best-fit packed version of FineWeb-Edu for training with looped nanochat.
Tokenized with the Llama 2 tokenizer (32,000 base vocab + 8 special tokens = 32,008).
Stats
Train split
Source
karpathy/fineweb-edu-100b-shuffle (1,821 shards)
Total tokens
63.26B
Total docs
97.1M
Rows
30,873,598
Shards
2,059 (train-00000 to train-02058)
Rows per shard
~15,000… See the full description on the dataset page: https://huggingface.co/datasets/KristianS7/prepacked-fineweb-edu-llama2-32K-T2048.fineweb-edu-gemma4-1024
FineWeb-Edu — pre-tokenized for fast LM pretraining (Gemma tokenizer, ArrayRecord/Grain)
Pre-tokenized FineWeb-Edu
(sample/100BT), packed into fixed-length sequences and stored as
ArrayRecord shards for zero-overhead
streaming with Grain. No on-the-fly tokenization
at train time — you read int32 tokens straight off disk.
Format
Tokenizer: google/gemma-4-12B-it (vocab size 262144). Documents are
separated by the EOS token id 1.
Packing: the token stream is… See the full description on the dataset page: https://huggingface.co/datasets/mlnomad/fineweb-edu-gemma4-1024.fineweb-edu-arabic
fineweb-edu-arabic
Arabic translation of FineWeb-Edu (sample/350BT subset, filtered to language_score > 0.9), translated with Seed-X-PPO-7B using greedy decoding. Documents were split into ~490-token chunks, translated, and reassembled. Each row is one complete document. A companion corpus translated with the same pipeline is available at dclm-pro-arabic.
Details
Documents: 82,840,410 (27.9% of the source subset, uniformly sampled)
Arabic tokens: ~170B (Seed-X… See the full description on the dataset page: https://huggingface.co/datasets/SultanR/fineweb-edu-arabic.Fineweb-Edu-Chinese-V2.2
Chinese Fineweb Edu Dataset V2.2 (Instruct & Pre-train)
[[中文]] | [[English]]
OpenCSG Community | 👾 GitHub | 📖 Technical Report
Dataset Introduction: Filling the Data Puzzle for Chinese Education LLMs
Chinese Fineweb Edu Dataset V2.2is a rare high-quality dataset in the open-source community that covers the full process from Pre-training to Supervised Fine-Tuning (SFT) for the Chinese education domain.
This project aims to solve the core pain point of… See the full description on the dataset page: https://huggingface.co/datasets/enche1561/Fineweb-Edu-Chinese-V2.2.edu-fineweb-10Bfineweb-edu
📚 FineWeb-Edu
1.3 trillion tokens of the finest educational data the 🌐 web has to offer
Paper: https://arxiv.org/abs/2406.17557
What is it?
📚 FineWeb-Edu dataset consists of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version.
To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by LLama3-70B-Instruct. We then… See the full description on the dataset page: https://huggingface.co/datasets/CharlesPing/fineweb-edu.Fineweb-Edu-10BT
Dataset Summary
This is the tokenized Fineweb-Edu (10BT subset) using SmolLM2-135M tokenzier. Data is divided into shards (.npy files) for easier to load with PyTorch IterableDataset.
Each .npy file can be loaded with numpy.load('file_name.npy').
Split
# Documents
# Shards
# Tokens
train
9,575,380
101
10,004,991,326 (10.0B)
val
96,721
1
101,807,253 (0.1B)
Total
9,672,101
102
10,106,798,579 (10.1B)
Example of usage
uvx hf download… See the full description on the dataset page: https://huggingface.co/datasets/minhnguyent546/Fineweb-Edu-10BT.fineweb-edu-dedup-45B
Fineweb-edu-dedup 45B
This dataset was filtered from HuggingFaceTB/smollm-corpus. We selected the fineweb-edu and further filtered with score > 3.0 to make the dataset with higher quality; there are 45B GPT2 tokens in this dataset.
Acknowledgement
We appreciate the efforts from HuggingFaceTB team to release these high-quality dataset and facilitate LLM community
FineWeb-Edu-Quality4plus
📘 FineWeb-Edu-Quality4plus
Overview
FineWeb-Edu-Quality4plus is a high-quality filtered subset of the original
HuggingFaceFW/fineweb-edu dataset (ODC-By License).
This subset retains only samples with:
quality_score ≥ 4
The goal is to provide a cleaner and more reliable dataset suitable for
language model pre-training, instruction tuning, education-related NLP,
and quality-sensitive downstream tasks.
This work is independent and not affiliated with the official FineWeb… See the full description on the dataset page: https://huggingface.co/datasets/Morton-Li/FineWeb-Edu-Quality4plus.fineweb-edu-dedup-shuffled
FineWeb-Edu-Dedup (Globally Shuffled)
A uniformly shuffled version of the FineWeb-Edu-Dedup subset from SmolLM-Corpus by HuggingFace.
Source Data
This dataset is derived from HuggingFaceTB/smollm-corpus, specifically the fineweb-edu-dedup subset. That subset is itself derived from FineWeb-Edu, a filtered and deduplicated extract of Common Crawl selected for educational content quality.
Property
Value
Source dataset
HuggingFaceTB/smollm-corpus
Source subset… See the full description on the dataset page: https://huggingface.co/datasets/JonathanMiddleton/fineweb-edu-dedup-shuffled.fineweb-edu-2013-qwen2-7b
FineWeb-Edu 2013 with Qwen2-7B token counts
Every 2013 FineWeb-Edu document, prepared for continued pretraining, with token
counts computed by a pinned Qwen2-7B tokenizer.
The pipeline is year-agnostic: the year, source revision, tokenizer contract,
and selection rule all come from a config file. 2013 uses
processing_config.json. The 2017 companion dataset, which is large enough to
require shuffling and a token budget rather than retaining everything, is at… See the full description on the dataset page: https://huggingface.co/datasets/stevenyuan666/fineweb-edu-2013-qwen2-7b.fineweb-edu-subsetSmall slice of the original HuggingFaceFW/fineweb-edu, data/CC-MAIN-2025-26.
Files: first 10 .parquet files, split=train, columns="text", applied filters: "language_score">=0.9.
Pipeline: Dataset will be further tokenized and used to train a 124M GPT model.
fineweb-edu-10BT-mincols
fineweb-edu: 10BT sample
This the "10BT-sample" config of HuggingFaceFW/fineweb-edu with most of the redundant cols removed for efficiency reasons.
token counts
GPT-4 tiktoken token count:
token_count
count 9.672101e+06
mean 1.001188e+03
std 1.834986e+03
min 3.800000e+01
25% 3.380000e+02
50% 6.090000e+02
75% 1.054000e+03
max 1.649670e+05
Total count: 9683.59 M tokens
fineweb-edu-10BT-mincols
fineweb-edu: 10BT sample
This the "10BT-sample" config of HuggingFaceFW/fineweb-edu with most of the redundant cols removed for efficiency reasons.
token counts
GPT-4 tiktoken token count:
token_count
count 9.672101e+06
mean 1.001188e+03
std 1.834986e+03
min 3.800000e+01
25% 3.380000e+02
50% 6.090000e+02
75% 1.054000e+03
max 1.649670e+05
Total count: 9683.59 M tokens
mga_fineweb_edu
Massive Genre-Audience Augment Fineweb-Edu Corpus
This dataset is a synthetic pretraining corpus described in paper Reformulation for Pretraining Data Augmentation.
Overview of synthesis framework. Our method expands the original corpus through a two-stage synthesis process.
Each document is reformulated to 5 new documents, achieving 3.9× token number expansion while maintaining diversity through massive (genre, audience) pairs.
We build MGACorpus based on SmolLM Corpus… See the full description on the dataset page: https://huggingface.co/datasets/RedMod/mga_fineweb_edu.FineWeb-Edu-Arabic-24M
English
العربية
FineWeb-Edu Arabic 24M
An Arabic-only pretraining corpus of 24,794,425 complete documents, translated from the sample-350BT configuration of FineWeb-Edu. It contains 34.86 billion Arabic tokenizer tokens and preserves the original FineWeb-Edu document IDs, source scores, and detailed translation diagnostics.
Highlight
Saudi architecture shaped by place. A well-translated tour of how builders in Najd, the Gulf coast, Hejaz, and Asir adapted local… See the full description on the dataset page: https://huggingface.co/datasets/nizarun/FineWeb-Edu-Arabic-24M.