datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
finepdfs
Liberating 3T of the finest tokens from PDFs
What is this?
As we run out of web pages to process, the natural question has always been: what to do next? Only a few knew about a data source that everyone avoided for ages, due to its incredible extraction cost and complexity: PDFs.
📄 FinePDFs is exactly that. It is the largest publicly available corpus sourced exclusively from PDFs, containing about 3 trillion tokens across 475 million documents in 1733 languages.
Compared to… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/finepdfs.finepdfs_lang_classificationfinepdfs-edu
📚 FinePDFs-Edu
350B+ of highly educational tokens from PDFs 📄
What is it?
📚 FinePDFs-Edu dataset consists of 350B+ tokens of educational PDFs filtered from 📄 FinePDFs dataset covering 69 languages.
FinePDFs was created using the formula inspired from FineWeb-Edu, we developed an educational quality classifier using annotations generated by Qwen3-235B-A22B-Instruct-2507 for each of 69 languages present in this dataset.
We then used this classifier to retain only the… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/finepdfs-edu.finepdfs-edu-purpose
FinePDFs-Edu document purpose — proof of concept
This dataset is a proof of concept for using agents to build small classifiers for data curation, then applying them cheaply with Hugging Face Jobs.
It contains a 1% sample of the English portion of FinePDFs-Edu, enriched with document-purpose predictions from a SetFit classifier. The aim is to make it easier to explore different mixtures of documents when sampling data.
Companion blog post: Using agents to build efficient… See the full description on the dataset page: https://huggingface.co/datasets/davanstrien/finepdfs-edu-purpose.finepdfs-edufinepdfs-edu-betafinepdfs_enfinepdfs
Liberating 3T of the finest tokens from PDFs
What is this?
As we run out of web pages to process, the natural question has always been: what to do next? Only a few knew about a data source that everyone avoided for ages, due to its incredible extraction cost and complexity: PDFs.
📄 FinePDFs is exactly that. It is the largest publicly available corpus sourced exclusively from PDFs, containing about 3 trillion tokens across 475 million documents in 1733 languages.
Compared to HTML… See the full description on the dataset page: https://huggingface.co/datasets/KefranAbg/finepdfs.finepdfs-1B
Sampling Methodology
This dataset was created using reservoir sampling, a statistically unbiased random sampling algorithm that guarantees each sample from the source dataset has an equal probability of being included. This ensures the 1B token sample is representative of the full dataset's characteristics.
Source Dataset: HuggingFaceFW/finepdfs
Sample Size: 1B tokens
Content: High-quality textbook-style pdfs
Reservoir sampling enables rapid experimentation and ablation studies… See the full description on the dataset page: https://huggingface.co/datasets/codelion/finepdfs-1B.finepdfs_eng_Latn_labeledfinepdfs-edu-hq
FinePDFs-Edu (English) — Filtered High-Signal Subset
This dataset is a filtered, English-only subset of HuggingFaceFW/finepdfs-edu, created to retain high-signal educational passages while reducing common PDF-extraction noise (covers/TOCs, fragmented headers/footers, OCR artifacts, mixed-language pages, and very short low-context snippets).
It is intended for training and research workflows that benefit from longer, coherent educational text extracted from PDFs.
At a… See the full description on the dataset page: https://huggingface.co/datasets/C10X/finepdfs-edu-hq.Romanian-finepdfs
Romanian PDFs - Processed Dataset
This is a processed and filtered version of the Romanian subset from the FinepdFs dataset, containing high-quality Romanian PDF documents extracted from Common Crawl. The dataset has been filtered for quality (full_doc_lid_score ≥ 0.5) and optimized by removing redundant metadata columns.
Dataset Overview
Total Documents: 3,254,816
Total Size: ~24.32 GB (compressed parquet with ZSTD)
Language: Romanian (ron_Latn)
Source: FinepdFs (Common… See the full description on the dataset page: https://huggingface.co/datasets/Yxanul/Romanian-finepdfs.finepdfs-long
HuggingFaceFW/finepdfs long filtered Turkish texts
Source: HuggingFaceFW/finepdfs (config: tur_Latn).
Rows contain 4,000–15,500 characters and passed the
iteration-5 Turkish language, repetition, glue-word, punctuation, SEO, and soft
information-density filters. Selected rows: 140,166.
Generated by process_hf_dataset.py. See summary.json for counts and thresholds.
filtered-finepdfs-edu.engFinepdfs filtered using this code:
def my_filter(example):
lscore = example["page_average_lid_score"]*example["full_doc_lid_score"] > 0.85*0.85
lenglish = example["language"] == "eng_Latn" and example["page_average_lid"] == "eng_Latn" and example["full_doc_lid"] == "eng_Latn"
fwscore = (sum(example["fw_edu_scores"])/len(example["fw_edu_scores"])) > 3
return lscore and lenglish and fwscore
This remove 87% of the dataset in average. This filtering have been applied on 1/4 of the… See the full description on the dataset page: https://huggingface.co/datasets/styal/filtered-finepdfs-edu.eng.finepdfs-edu-ml300kfinepdfs-edu-ml300kfinepdfs_ita_Latn_metadata_100K
FinePDFs metadata
This dataset provides rich PDF metadata for a uniform random sample of 100,000 documents from the ita_Latn split (Italian, Latin script) of the FinePDFs index. Each PDF was re-fetched from its original Common Crawl WARC record and analyzed with Poppler (pdfinfo, pdffonts) and pdfresurrect.
Why this dataset
FinePDFs is the largest publicly available corpus sourced from PDFs, but it ships only the extracted text and a handful of crawl fields — the… See the full description on the dataset page: https://huggingface.co/datasets/fedric95/finepdfs_ita_Latn_metadata_100K.urdu_finepdfs
What’s inside
data/ (optional) — small example files / scripts [FUTURE] . This repo is primarily a pointer + helpers to the official FinePDFs Urdu shards.
scripts/ [FUTURE] — utility scripts to list, preview, and filter Urdu parquet shards (example: extract metadata, sample text, convert to plain text).
README.md — this file.
If you cloned this repo to help with the downstream work, expect the real Urdu shards to be loaded from the official Hugging Face hub (see examples below).… See the full description on the dataset page: https://huggingface.co/datasets/humair025/urdu_finepdfs.finepdfs-korean데이터 확인을 편하게 하려고 HuggingFaceFW/finepdfs에서 한국어 subset만 따로 분리했습니다.
finepdfs-et
FinePDFs-et
The Estonian subset of HuggingFaceFW/finepdfs reuploaded for ease of access.
Licensing Information
The dataset is released under the Open Data Commons Attribution License (ODC-By) v1.0 license. The use of this dataset is also subject to CommonCrawl's Terms of Use.
Citation Information
@misc{kydlicek2025finepdfs,
title={FinePDFs},
author={Hynek Kydl{\'\i}{\v{c}}ek and Guilherme Penedo and Leandro von Werra},
year={2025}… See the full description on the dataset page: https://huggingface.co/datasets/tartuNLP/finepdfs-et.nno_finepdfs_edu_scoresfinepdfs-eng-stats
Statistics for HuggingFaceFW/finepdfs-edu (English (Latin script))
Aggregate statistics computed using Polars streaming on the HuggingFaceFW/finepdfs-edu dataset.
Performance
Processed 23,023,372 documents in 2608.95 seconds.
Step
Time
Global stats
686.56s
Language stats
634.40s
Extractor stats
648.34s
Dump stats
639.65s
Total
2608.95s
Speed comes from Polars only reading metadata columns (not the text column),
thanks to Parquet's columnar format… See the full description on the dataset page: https://huggingface.co/datasets/davanstrien/finepdfs-eng-stats.finepdfs-temporal-stats-nl
Is the Web Getting More Educational?
Temporal analysis of educational quality in German (Latin script) across 106 CommonCrawl dumps.
Trend
High Educational Content (edu >= 3)
###############################################################
██████████████████████████████████ 9.9% 2013
████████████████████ 6.0% 2014
██████████ 3.0% 2015
████████████████████████████████████████████… See the full description on the dataset page: https://huggingface.co/datasets/davanstrien/finepdfs-temporal-stats-nl.finepdfs-edu-hq-2048finepdfs-plfinepdfs-100M
Sampling Methodology
This dataset was created using reservoir sampling, a statistically unbiased random sampling algorithm that guarantees each sample from the source dataset has an equal probability of being included. This ensures the 100M token sample is representative of the full dataset's characteristics.
Source Dataset: HuggingFaceFW/finepdfs
Sample Size: 100M tokens
Content: High-quality textbook-style pdfs
Reservoir sampling enables rapid experimentation and ablation studies… See the full description on the dataset page: https://huggingface.co/datasets/codelion/finepdfs-100M.HuggingFaceFW__finepdfs-edu_por_Latn_train_raw_5000_prompt1finepdfs-cy-errors
Dataset Card: finepdfs-cy-errors
Description
This dataset contains Welsh-language text extracted from PDFs using rolmOCR, with automated spelling and grammar error annotations generated by Cysill (the Welsh spell checker). The dataset is derived from the Welsh (cym_Latn) subset of HuggingFaceFW/finepdfs, filtered to include only documents processed with the rolmOCR extractor.
Dataset Statistics
Corpus Size
Total number of words: 1,575,836
Total… See the full description on the dataset page: https://huggingface.co/datasets/techiaith/finepdfs-cy-errors.HuggingFaceFW__finepdfs-edu_por_Latn_train_scored_5000_edu_scoreHuggingFaceFW__finepdfs-edu_por_Latn_train_scored_5000_en