datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-edu
📚 FineWeb-Edu
1.3 trillion tokens of the finest educational data the 🌐 web has to offer
Paper: https://arxiv.org/abs/2406.17557
What is it?
📚 FineWeb-Edu dataset consists of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version.
To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by LLama3-70B-Instruct. We… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu.fineweb-edu-translated
Helsinki-NLP/fineweb-edu-translated
fineweb-edu-tanslated is a collection of automatically translated documents from fineweb-edu.
Translations are based on OPUS-MT and HPLT-MT models.
The data in v1.0 covers 36,704,000 documents with over 28 billion space-searated tokens of English data translated into 36 languages.
The total v1.0 data set includes over 960 billion tokens and the translated documents are aligned across all languages.
In the v1.1 release, additional translations… See the full description on the dataset page: https://huggingface.co/datasets/Helsinki-NLP/fineweb-edu-translated.fineweb-edu-fortified
Fineweb-Edu-Fortified
The composition of fineweb-edu-fortified, produced by automatically clustering a 500k row sample in
Airtrain
What is it?
Fineweb-Edu-Fortified is a dataset derived from
Fineweb-Edu by applying exact-match
deduplication across the whole dataset and producing an embedding for each row. The number of times
the text from each row appears is also included as a count column. The embeddings were produced
using TaylorAI/bge-micro
Fineweb and… See the full description on the dataset page: https://huggingface.co/datasets/airtrain-ai/fineweb-edu-fortified.mmlu-redux-2.0
Dataset Card for MMLU-Redux-2.0
MMLU-Redux is a subset of 5,700 manually re-annotated questions across 57 MMLU subjects.
News
[2025.02.25] We corrected one annotation in Abstract Algebra subset, as noted in the Issue #2.
[2025.02.08] We corrected one annotation in High School Mathematics subset, as noted in the PlatinumBench paper.
[2025.01.23] MMLU-Redux is accepted to NAACL 2025!
Dataset Details
Dataset Description
Each data point in… See the full description on the dataset page: https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.open-apply-jobs
Open-Apply Jobs
A daily-refreshed open dataset of active job postings sourced directly from public ATS APIs (Greenhouse, Lever, Ashby). Every record can be traced back to the hiring company's own career board.
Refresh: automated daily at 06:00 UTC
Partitioning: Hive-partitioned Parquet (date=YYYY-MM-DD/source={ats})
Source code: https://github.com/edwarddgao/openapply
Usage
from datasets import load_dataset
ds = load_dataset('edwarddgao/open-apply-jobs')
#… See the full description on the dataset page: https://huggingface.co/datasets/edwarddgao/open-apply-jobs.fineweb-edu-score-2
📚 FineWeb-Edu-score-2
1.3 trillion tokens of the finest educational data the 🌐 web has to offer
What is it?
📚 FineWeb-Edu dataset consists of 1.3T tokens (FineWeb-Edu) and 5.4T tokens of educational web pages filtered from 🍷 FineWeb dataset. This is the 5.4 trillion version.
Note: this version uses a lower educational score threshold = 2, which results in more documents, but lower quality compared to the 1.3T version. For more details check the… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-score-2.chinese-fineweb-edu
This version is deprecated. We recommend you to use the newest version Fineweb-edu-chinese-v2.1 !
Chinese Fineweb Edu Dataset [中文] [English]
[OpenCSG Community] [👾github] [wechat] [Twitter]
📖Technical Report
Chinese Fineweb Edu dataset is a meticulously constructed high-quality Chinese pre-training corpus, specifically designed for natural language processing tasks in the education domain. This dataset undergoes a rigorous selection and… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/chinese-fineweb-edu.finepdfs-edu
📚 FinePDFs-Edu
350B+ of highly educational tokens from PDFs 📄
What is it?
📚 FinePDFs-Edu dataset consists of 350B+ tokens of educational PDFs filtered from 📄 FinePDFs dataset covering 69 languages.
FinePDFs was created using the formula inspired from FineWeb-Edu, we developed an educational quality classifier using annotations generated by Qwen3-235B-A22B-Instruct-2507 for each of 69 languages present in this dataset.
We then used this classifier to retain only the… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/finepdfs-edu.SEC-EDGARDatamule, Teraflop AI, and Eventual collaborated to release the SEC-EDGAR dataset.
The dataset contains 590 gbs of data, spanning 8 million samples and 43 billion tokens from all major filings in the SEC EDGAR database.
The bulk data was collected using datamule-python library and the official datamule api created by John Friedman. The datamule Python library is a package for collecting, manipulating, and processing the SEC Edgar data at scale. Datamule provides a simple open-source api… See the full description on the dataset page: https://huggingface.co/datasets/TeraflopAI/SEC-EDGAR.xsum
Dataset Card for "xsum"
Dataset Summary
Extreme Summarization (XSum) Dataset.
There are three features:
document: Input news article.
summary: One sentence summary of the article.
id: BBC ID of the article.
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed
Dataset Structure
Data Instances
default
Size of downloaded dataset files: 257.30 MB
Size of the generated dataset:… See the full description on the dataset page: https://huggingface.co/datasets/EdinburghNLP/xsum.ami
Dataset Card for AMI
Dataset Description
The AMI Meeting Corpus consists of 100 hours of meeting recordings. The recordings use a range of signals
synchronized to a common timeline. These include close-talking and far-field microphones, individual and
room-view video cameras, and output from a slide projector and an electronic whiteboard. During the meetings,
the participants also have unsynchronized pens available to them that record what is written. The meetings
were… See the full description on the dataset page: https://huggingface.co/datasets/edinburghcstr/ami.ua-edrsr
ЄДРСР — судові рішення України (нормалізоване дзеркало)
Автоматичне дзеркало офіційних публікацій Єдиного державного реєстру судових рішень на data.gov.ua.
Пайплайн: JoTalbot/ukraine.
Роки та обсяги
Рік
Записів
SHA-256 архіву
2006
340171
4e19811feef9…
2007
1084514
49cf87b3a0e1…
2008
2185311
ce7dbc1b8da3…
2009
3539627
3be90c8824ab…
2010
5869727
b9cd6b5e2562…
2011
7128372
4b845ee3b4bd…
2012
6903131
ba145669d313…
2013
7704297
1794ceb7e3a5…… See the full description on the dataset page: https://huggingface.co/datasets/JoTalbot/ua-edrsr.gpt-edit-simplerfineweb-edu-100b-shuffleexplicit-edit-benchmark
Explicit Edit Benchmark
226 deterministic exact-edit tasks, run by different agents, harnesses, models and configurations. Every observation records what the harness did and whether the resulting files matched byte for byte.
Source code and benchmark runner: GitHub — Explicit Edit Benchmark
Open the interactive Explorer to compare agents, harnesses, models, versions, reasoning modes, correctness, recovery, time, cost and tokens.
Leaderboard by model route
Score v2… See the full description on the dataset page: https://huggingface.co/datasets/alexshpunt/explicit-edit-benchmark.labbench2
LABBench2
LABBench2 is a benchmark for measuring real-world capabilities of AI systems performing scientific research tasks. It is an evolution of the Language Agent Biology Benchmark (LAB-Bench), comprising nearly 1,900 tasks that measure similar capabilities but in more realistic contexts.
LABBench2 provides a meaningful jump in difficulty over LAB-Bench (model-specific accuracy differences range from −26% to −46% across subtasks), underscoring continued room for improvement.… See the full description on the dataset page: https://huggingface.co/datasets/EdisonScientific/labbench2.edgar-corpusThe dataset contains annual filings (10K) of all publicly traded firms from 1993-2020. The table data is stripped but all text is retained.
This dataset allows easy access to the EDGAR-CORPUS dataset based on the paper EDGAR-CORPUS: Billions of Tokens Make The World Go Round (See References in README.md for details).fineweb-edu
FineWeb-Edu (Lance Format)
A Lance-formatted version of FineWeb-Edu — over 1.5 billion educational web passages with cleaned text, source metadata, language detection signals, and 384-dim text embeddings — available directly from the Hub at hf://datasets/lance-format/fineweb-edu/data/train.lance.
Key features
Cleaned passage text in the text column with the source url and title carried alongside.
Language detection signals (language, language_probability) for filtered… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/fineweb-edu.EdgeBench
Overview
EdgeBench is a benchmark of 134 real-world tasks for evaluating how autonomous AI agents learn from real-world environments. Instead of measuring one-shot performance, EdgeBench places agents in executable task environments with realistic, multi-level feedback and lets them iterate for 12+ hours per task — tracking the full trajectory of improvement, not just the final score. We publicly release 51 tasks… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/EdgeBench.stack-edu
💻 Stack-Edu
Stack-Edu is a 125B token dataset of educational code filtered from The Stack v2, precisely the curated training corpus of StarCoder2 models denoted StarCoder2Data. It is intended for Language Models training.
This dataset was curated using a classifier-based filtering strategy, inspired by 📚 FineWeb-Edu, to retain only the highest-quality educational programming content.
Stack-Edu shows consistent improvement over StarCoder2data on all the programming languages on… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/stack-edu.fineweb-edu
Pre-shuffled fineweb-edu dataset
sp500-edgar-10k
Dataset Card for SP500-EDGAR-10K
Dataset Summary
This dataset contains the annual reports for all SP500 historical constituents from 2010-2022 from SEC EDGAR Form 10-K filings.
It also contains n-day future returns of each firm's stock price from each filing date.
Dataset Structure
Data Fields
[More Information Needed]
Data Splits
[More Information Needed]
Dataset Creation
Source Data
Initial Data Collection… See the full description on the dataset page: https://huggingface.co/datasets/jlohding/sp500-edgar-10k.GPT-Image-Edit-1.5M
GPT-Image-Edit-1.5M A Million-Scale, GPT-Generated Image Dataset
📃Arxiv | 🌐 Project Page | 💻Github
GPT-Image-Edit-1.5M is a comprehensive image editing dataset that is built upon HQ-Edit, UltraEdit, OmniEdit and Complex-Edit, with all output images regenerated with GPT-Image-1.
📣 News
[2025.08.20] 🚀 We provide a script for multi-process downloading. See Multi-process Download.
[2025.07.27] 🤗 We release GPT-Image-Edit, a state-of-the-art image editing model with… See the full description on the dataset page: https://huggingface.co/datasets/UCSC-VLAA/GPT-Image-Edit-1.5M.stackv2_edu_filtered
Stack V2 Edu
Description
We filter the Stack V2 to only include code from openly licensed repositories, based on the license detection performed by the creators of Stack V2. When multiple licenses are detected in a single repository, we ensure that all of the licenses are on the Blue Oak Council certified license list. Per-document license information is available in the license entry of the metadata field of each example. Code for collecting, processing, and preparing… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/stackv2_edu_filtered.irish_fineweb_eduData translation project of https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu, sample-10BT subset. Data are translated from English to Irish using NLLB-3.3B.
editorai-telemetryfineweb-edu-gpt2finetranslations-edu-zhtw
Dataset Card for finetranslations-edu-zhtw
✅ 狀態：翻譯已完成 —— 上游 HuggingFaceFW/finetranslations-edu 的 108,882,733 列已全數翻譯完成（逐檔清點兩邊 parquet footer，列數完全相符），共 8,195 個分片、約 1.26 TB。
⚠️ 但尚未經過人工抽查驗證。翻譯全程自動化，使用前請自行評估品質。
📖 finetranslations-edu-zhtw 是以 HuggingFaceFW/finetranslations-edu 為來源，將其 translated_chunks（原始多語言教育類網頁內容、先被 pivot 翻譯成英文的版本）進一步翻譯成繁體中文的資料集。
Dataset Details
Dataset Description
HuggingFaceFW/finetranslations-edu 收錄了原本以英文以外語言（og_language，涵蓋約 200… See the full description on the dataset page: https://huggingface.co/datasets/lianghsun/finetranslations-edu-zhtw.swesmith-cleanEdge-Agent-Reasoning-WebSearch-260K
Edge Agent Reasoning WebSearch 260K
Abstract
The Edge-Agent-Reasoning-WebSearch-260K dataset is a massive, synthetically expert-engineered corpus of over 700 Million tokens, designed to train small, local models (SLMs) and edge-deployed agents in advanced problem deconstruction and self-aware reasoning.
Rather than training a model to execute instructions directly—which often leads to hallucinations when context is missing—this dataset trains a model to act as a… See the full description on the dataset page: https://huggingface.co/datasets/yatin-superintelligence/Edge-Agent-Reasoning-WebSearch-260K.