datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
FineWeb-HQ
FineWeb-HQ
Dataset Summary
FineWeb-HQ is a high-quality, model-filtered pretraining dataset derived as a subset of FineWeb. FineWeb-HQ was created by selecting the top 10% of FineWeb documents based on a deep learning classifier trained to identify structured and knowledge-rich samples. This classifier uses XLM-RoBERTa embeddings to score documents.
To validate our approach, we pretrained 1B-parameter LLM models with a Llama-like architecture across multiple… See the full description on the dataset page: https://huggingface.co/datasets/epfml/FineWeb-HQ.fineweb-edu
📚 FineWeb-Edu
1.3 trillion tokens of the finest educational data the 🌐 web has to offer
Paper: https://arxiv.org/abs/2406.17557
What is it?
📚 FineWeb-Edu dataset consists of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version.
To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by LLama3-70B-Instruct. We… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu.ptb-xl-ecghendrycks_math
Dataset Summary
MATH dataset from https://github.com/hendrycks/math
Citation Information
@article{hendrycksmath2021,
title={Measuring Mathematical Problem Solving With the MATH Dataset},
author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},
journal={NeurIPS},
year={2021}
}
fineweb-edu-translated
Helsinki-NLP/fineweb-edu-translated
fineweb-edu-tanslated is a collection of automatically translated documents from fineweb-edu.
Translations are based on OPUS-MT and HPLT-MT models.
The data in v1.0 covers 36,704,000 documents with over 28 billion space-searated tokens of English data translated into 36 languages.
The total v1.0 data set includes over 960 billion tokens and the translated documents are aligned across all languages.
In the v1.1 release, additional translations… See the full description on the dataset page: https://huggingface.co/datasets/Helsinki-NLP/fineweb-edu-translated.race
Dataset Card for "race"
Dataset Summary
RACE is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The
dataset is collected from English examinations in China, which are designed for middle school and high school students.
The dataset can be served as the training and test sets for machine comprehension.
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed… See the full description on the dataset page: https://huggingface.co/datasets/ehovy/race.ceval-examC-Eval is a comprehensive Chinese evaluation suite for foundation models. It consists of 13948 multi-choice questions spanning 52 diverse disciplines and four difficulty levels. Please visit our website and GitHub or check our paper for more details.
Each subject consists of three splits: dev, val, and test. The dev set per subject consists of five exemplars with explanations for few-shot evaluation. The val set is intended to be used for hyperparameter tuning. And the test set is for model… See the full description on the dataset page: https://huggingface.co/datasets/ceval/ceval-exam.EmbodiedGenDatahttps://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer
Egocentric-100K
Egocentric-100K is the largest dataset of manual labor. You can visualize the dataset here.
Egocentric-100K is state-of-the-art in hand visibility and active manipulation density compared to previous in-the-wild egocentric datasets. The complete 30,000 frame evaluation set is available at Egocentric-100K-Evaluation.
Dataset Statistics
Attribute
Value
Total Hours
100,405
Total Frames
10.8 billion
Video Clips
2,010,759
Median Clip Length
180.0 seconds
Mean… See the full description on the dataset page: https://huggingface.co/datasets/builddotai/Egocentric-100K.AI-CUDA-Engineer-Archive
The AI CUDA Engineer Archive 👷: Agentic CUDA Kernel Discovery, Optimization & Composition
We release The AI CUDA Engineer archive, a dataset consisting of approximately 30,000 CUDA kernels generated by The AI CUDA Engineer. It is released under the CC-By-4.0 license and can be accessed via HuggingFace and interactively visualized here. The dataset is based on the Kernel tasks provided in KernelBench and includes a torch reference implementation, torch, NCU and Clang-tidy… See the full description on the dataset page: https://huggingface.co/datasets/SakanaAI/AI-CUDA-Engineer-Archive.lambada_openai
Dataset Summary
This dataset is comprised of the LAMBADA test split as pre-processed by OpenAI (see relevant discussions here and here). It also contains machine translated versions of the split in German, Spanish, French, and Italian.
LAMBADA is used to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative texts sharing the characteristic that human subjects are able to guess their last word… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/lambada_openai.headqaUTDQuake
UTDQuake: University of Texas at Dallas Earthquake Dataset
A global earthquake dataset constructed from high-quality source and receiver metadata, including associated seismic phase picks across diverse station geometries.
Installation (utdquake)
pip install utdquake
Documentation
Full documentation for UTDQuake is available here:
You will see:
QuickStart guide to get you up and running
Detailed API reference
Tutorials… See the full description on the dataset page: https://huggingface.co/datasets/ecastillot/UTDQuake.yodas-granary
Dataset Card for YODAS-Granary
Repository: NeMo-speech-data-processor: Granary
Paper: Granary: Speech Recognition and Translation Dataset in 25 European Languages
Shared by: ESPnet
Dataset Description
YODAS-Granary is a curated subset of the larger nvidia/Granary dataset, focusing on high-quality pseudo-labeled speech data for Automatic Speech Recognition (ASR) and Automatic Speech Translation (AST) across 23 European languages.
Overview… See the full description on the dataset page: https://huggingface.co/datasets/espnet/yodas-granary.Hy-Embodied-0.5-VLA-Data
Hy-Embodied-0.5-VLA
From Vision-Language-Action Models to a Real-World Robot Learning Stack
Tencent Robotics X × Tencent Hy Team
📖 Abstract
We introduce Hy-Embodied-0.5-VLA (Hy-VLA) — an end-to-end Vision-Language-Action system that spans the full robot learning stack: data collection, model design, pre-training, supervised fine-tuning, RL post-training, and real-world deployment. Built on the Hy-Embodied-0.5 MoT backbone, Hy-VLA integrates a flow-matching… See the full description on the dataset page: https://huggingface.co/datasets/tencent/Hy-Embodied-0.5-VLA-Data.wikitext_document_level
Wikitext Document Level
This is a modified version of https://huggingface.co/datasets/wikitext that returns Wiki pages instead of Wiki text line-by-line. The original readme is contained below.
Dataset Card for "wikitext"
Dataset Summary
The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified
Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/wikitext_document_level.mbppplusMultiPL-E
Dataset Card for MultiPL-E
Dataset Summary
MultiPL-E is a dataset for evaluating large language models for code
generation that supports 22 programming languages. It takes the OpenAI
HumanEval and the Mostly Basic Python Programs (MBPP) benchmarks and uses little compilers to
translate them to other languages. It is easy to add support for new languages
and benchmarks.
The dataset is divided into several configurations named SRCDATA-LANG, where
SRCDATA is either… See the full description on the dataset page: https://huggingface.co/datasets/nuprl/MultiPL-E.ExperimentDATA_knowledge_distillation_vs_fine_tuningABot-World-Explorer-500h
ABot World Explorer 500h
ABot World Explorer 500h contains 30,969 action-conditioned video episodes
associated with the data infrastructure described in
ABot-World-0. Each episode preserves an MP4,
dataset-native keyboard actions, captions, and one COLMAP text sparse model.
Dataset facts
Item
Value
Episodes
30,969
Source objects
185,814
Semantic splits
None
License
Apache-2.0
The repository name is an identifier, not an audited… See the full description on the dataset page: https://huggingface.co/datasets/acvlab/ABot-World-Explorer-500h.asdivcad-environments
CAD Environments
CAD Environments is a multimodal dataset of complete, human-performed workflows in desktop CAD software. The current release contains 51 task workflows totaling 99.03 hours, covering eight software groups across mechanical design, architecture, MEP, structural design, and general 3D modeling.
Each workflow preserves the full task context—not just the final model—including the problem statement, reference and input files, a gold output, evaluation rubrics, a… See the full description on the dataset page: https://huggingface.co/datasets/markov-ai/cad-environments.ego-1k
Ego-1K — A Large-Scale Multiview Video Dataset for Egocentric Vision
Jae Yong Lee, Daniel Scharstein, Akash Bapat, Hao Hu, Andrew Fu, Haoru Zhao, Paul Sammut, Xiang Li,
Stephen Jeapes, Anik Gupta, Lior David, Saketh Madhuvarasu, Jay Girish Joshi, and Jason Wither
CVPR 2026
arXiv:2603.13741
We present Ego-1K, a large-scale collection of time-synchronized egocentric multiview videos designed to advance neural 3D video
synthesis and dynamic scene understanding.… See the full description on the dataset page: https://huggingface.co/datasets/facebook/ego-1k.mmlu-redux-2.0
Dataset Card for MMLU-Redux-2.0
MMLU-Redux is a subset of 5,700 manually re-annotated questions across 57 MMLU subjects.
News
[2025.02.25] We corrected one annotation in Abstract Algebra subset, as noted in the Issue #2.
[2025.02.08] We corrected one annotation in High School Mathematics subset, as noted in the PlatinumBench paper.
[2025.01.23] MMLU-Redux is accepted to NAACL 2025!
Dataset Details
Dataset Description
Each data point in… See the full description on the dataset page: https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.textvqa
Large-scale Multi-modality Models Evaluation Suite
Accelerating the development of large-scale multi-modality models (LMMs) with lmms-eval
🏠 Homepage | 📚 Documentation | 🤗 Huggingface Datasets
This Dataset
This is a formatted version of TextVQA. It is used in our lmms-eval pipeline to allow for one-click evaluations of large multi-modality models.
@inproceedings{singh2019towards,
title={Towards vqa models that can read},
author={Singh, Amanpreet and… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab-encoder/textvqa.Emilia-Dataset
Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation
This is the official repository 👑 for the Emilia dataset and the source code for the Emilia-Pipe speech data preprocessing pipeline.
News 🔥
2025/02/26: The Emilia-Large dataset, featuring over 200,000 hours of data, is now available!!! Emilia-Large combines the original 101k-hour Emilia dataset (licensed under CC BY-NC 4.0) with the brand-new 114k-hour Emilia-YODAS… See the full description on the dataset page: https://huggingface.co/datasets/amphion/Emilia-Dataset.WorldSpeech
WorldSpeech
A multilingual ASR dataset containing over 65k hours of human transcribed speech across 127 language-region variants, drawn from national parliaments, public broadcasters, public-domain audiobooks, and international institutions. Rows consist of 24 kHz speech utterances paired with a human-provided transcript, an aligned ASR transcript, character error rate (CER) between the two, a WADA-SNR estimate, and four DNSMOS-P.835 quality scores.
Dataset Overview… See the full description on the dataset page: https://huggingface.co/datasets/disco-eth/WorldSpeech.fineweb-edu-fortified
Fineweb-Edu-Fortified
The composition of fineweb-edu-fortified, produced by automatically clustering a 500k row sample in
Airtrain
What is it?
Fineweb-Edu-Fortified is a dataset derived from
Fineweb-Edu by applying exact-match
deduplication across the whole dataset and producing an embedding for each row. The number of times
the text from each row appears is also included as a count column. The embeddings were produced
using TaylorAI/bge-micro
Fineweb and… See the full description on the dataset page: https://huggingface.co/datasets/airtrain-ai/fineweb-edu-fortified.SWE-bench-extraNote: This dataset has an improved and significantly larger successor: SWE-rebench.
Dataset Summary
SWE-bench Extra is a dataset that can be used to train or evaluate agentic systems specializing in resolving GitHub issues. It is based on the methodology used to build SWE-bench benchmark and includes 6,415 Issue-Pull Request pairs sourced from 1,988 Python repositories.
Dataset Description
The SWE-bench Extra dataset supports the development of software engineering agents… See the full description on the dataset page: https://huggingface.co/datasets/nebius/SWE-bench-extra.Magicoder-Evol-Instruct-110KA decontaminated version of evol-codealpaca-v1. Decontamination is done in the same way as StarCoder (bigcode decontamination process).