datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
c4
C4
Dataset Summary
A colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org".
This is the processed version of Google's C4 dataset
We prepared five variants of the data: en, en.noclean, en.noblocklist, realnewslike, and multilingual (mC4).
For reference, these are the sizes of the variants:
en: 305GB
en.noclean: 2.3TB
en.noblocklist: 380GB
realnewslike: 15GB
multilingual (mC4): 9.7TB (108 subsets, one… See the full description on the dataset page: https://huggingface.co/datasets/allenai/c4.ubuntu_osworld_file_cache
OSWorld File Cache
This repository serves as a file cache for the OSWorld project, providing reliable and fast access to evaluation files that were previously hosted on Google Drive.
Overview
OSWorld is a scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems and applications. This cache repository ensures that all evaluation files are consistently… See the full description on the dataset page: https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache.mmlu
Dataset Card for MMLU
Dataset Summary
Measuring Massive Multitask Language Understanding by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (ICLR 2021).
This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57… See the full description on the dataset page: https://huggingface.co/datasets/cais/mmlu.standard-chess-games
[!CAUTION]
This dataset is still a work in progress and some breaking changes might occur.
Lichess Rated Standard Chess Games Dataset
Dataset Description
6,771,826,271 standard rated games, played on lichess.org, updated monthly from the database dumps.
This version of the data is meant for data analysis. If you need PGN files you can find those here. That said, once you have a subset of interest, it is trivial to convert it back to PGN as shown in the Dataset Usage… See the full description on the dataset page: https://huggingface.co/datasets/Lichess/standard-chess-games.preprocessed_commoncatalog-cc-byI also seperately provide just the prompts in prompts.json
keys are the image_id, and the values are the captions generated
Captions generated by moondream: vikhyatk/moondream2
Latents generated by SDXL VAE: madebyollin/sdxl-vae-fp16-fix
Embeddings generated by SigLIP: hf-hub:timm/ViT-SO400M-14-SigLIP-384
Original dataset: common-canvas/commoncatalog-cc-by
Latents f32 and embeddings are f16 bytes
Compute cost: 16x3090 for 3 day. Approximately.
physics-course-vidsMELDcc
What is Symato CC?
To download all WARC data from Common Crawl then filter out Vietnamese in Markdown and Plaintext format.
There is 1% of Vietnamse in CC, extract all of them out should be a lot (~10TB of plaintext).
Main contributors
https://huggingface.co/nampdn-ai
https://huggingface.co/binhvq
https://huggingface.co/th1nhng0
https://huggingface.co/iambestfeed
Simple quality filters
To make use of raw data from common crawl, you need to do filtering… See the full description on the dataset page: https://huggingface.co/datasets/Symato/cc.flatpakthe_cauldron
Dataset Card for The Cauldron
Dataset description
The Cauldron is part of the Idefics2 release.
It is a massive collection of 50 vision-language datasets (training sets only) that were used for the fine-tuning of the vision-language model Idefics2.
Load the dataset
To load the dataset, install the library datasets with pip install datasets. Then,
from datasets import load_dataset
ds = load_dataset("HuggingFaceM4/the_cauldron", "ai2d")
to download… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceM4/the_cauldron.language_table_lerobotThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.0",
"robot_type": "xarm",
"total_episodes": 442226,
"total_frames": 7045476,
"total_tasks": 127605,
"total_videos": 442226,
"total_chunks": 443,
"chunks_size": 1000,
"fps": 10,
"splits": {
"train": "0:442226"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/IPEC-COMMUNITY/language_table_lerobot.course-imagesdroid_1.0.1This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "Franka",
"total_episodes": 95600,
"total_frames": 27612581,
"total_tasks": 0,
"total_videos": 286800,
"total_chunks": 95,
"chunks_size": 1000,
"fps": 15,
"splits": {
"train": "0:95600"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/cadene/droid_1.0.1.documentation-imagescommonsense_qa
Dataset Card for "commonsense_qa"
Dataset Summary
CommonsenseQA is a new multiple-choice question answering dataset that requires different types of commonsense knowledge
to predict the correct answers . It contains 12,102 questions with one correct answer and four distractor answers.
The dataset is provided in two major training/validation/testing set splits: "Random split" which is the main evaluation
split, and "Question token split", see paper for details.… See the full description on the dataset page: https://huggingface.co/datasets/tau/commonsense_qa.droid_lerobotThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.0",
"robot_type": "franka",
"total_episodes": 92233,
"total_frames": 27044326,
"total_tasks": 31308,
"total_videos": 276699,
"total_chunks": 93,
"chunks_size": 1000,
"fps": 15,
"splits": {
"train": "0:92233"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/IPEC-COMMUNITY/droid_lerobot.cybergymettin-pretraining-data
Ettin Pre-training Data
Phase 1 of 3: Diverse pre-training data mixture (1.7T tokens) used to train the Ettin model suite.
This dataset contains the pre-training phase data used to train all Ettin encoder and decoder models. The data is provided in MDS format ready for use with Composer and the ModernBERT training repository.
📊 Data Composition
Data Source
Tokens (B)
Percentage
Description
DCLM
837.2
49.1%
High-quality web crawl data
CC Head
356.6… See the full description on the dataset page: https://huggingface.co/datasets/jhu-clsp/ettin-pretraining-data.ncbi-genbank-complete
Dataset Card for NCBI GenBank Complete
Dataset Summary
GenBank® is the NIH genetic sequence database, an annotated collection of all publicly available DNA sequences. GenBank is part of the International Nucleotide Sequence Database Collaboration (INSDC), which comprises the DNA DataBank of Japan (DDBJ), the European Nucleotide Archive (ENA), and GenBank at NCBI. These three organizations exchange data on a daily basis.
This dataset has been processed into a… See the full description on the dataset page: https://huggingface.co/datasets/pulmo/ncbi-genbank-complete.Darwin-CCFastUMI_100k_lerobot
FastUMI-100K: Advancing Data-Driven Robotic Manipulation with a Large-Scale UMI-Style Dataset
[paper] [dataset]
## Overview
FastUMI-100K is a large-scale, high-quality UMI-style dataset designed for data-driven robotic manipulation learning. Featuring over **100K+ demonstration trajectories** across **54 diverse tasks** and hundreds of object types, the dataset provides multi-view wrist-mounted fisheye images and high-frequency end-effector states. To… See the full description on the dataset page: https://huggingface.co/datasets/IPEC-COMMUNITY/FastUMI_100k_lerobot.transformers_circleci_workflow_runschallenge_data
PrimeBot Household Bimanual Manipulation Challenge Dataset
中文 | English
中文
目录
关于我们
更新日志
真机遥操作数据
训练集说明
验证集说明
数据集字段说明
URDF
图像
语言指令
本体感知与动作
机器人推理接口
UMI数据
数据概览
目录结构
数据集字段说明
图像
本体感知与动作
索引字段
标注与 IMU
关于我们
我们来自上纬新材-启元研究院，我们的使命是加速个人机器人时代到来，加速家用机器人时代到来。我们开源高质量面向家庭操作的双臂操作数据集，同时开放机器人硬件描述以供可视化、可复现研究。
如果本数据集对您的工作有帮助，感谢引用：
@misc{xu2026scalingbimanualhouseholdmanipulation,
title={Scaling Bimanual Household Manipulation from 1,500… See the full description on the dataset page: https://huggingface.co/datasets/challenge-2026/challenge_data.kuka_lerobotThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.0",
"robot_type": "kuka_iiwa",
"total_episodes": 209880,
"total_frames": 2455879,
"total_tasks": 1,
"total_videos": 209880,
"total_chunks": 210,
"chunks_size": 1000,
"fps": 10,
"splits": {
"train": "0:209880"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/IPEC-COMMUNITY/kuka_lerobot.arxiv-cs-2020-2025-pdfsNuminaMath-CoT
Dataset Card for NuminaMath CoT
Dataset Summary
Approximately 860k math problems, where each solution is formatted in a Chain of Thought (CoT) manner. The sources of the dataset range from Chinese high school math exercises to US and international mathematics olympiad competition problems. The data were primarily collected from online exam paper PDFs and mathematics discussion forums. The processing steps include (a) OCR from the original PDFs, (b) segmentation… See the full description on the dataset page: https://huggingface.co/datasets/AI-MO/NuminaMath-CoT.cifar10
Dataset Card for CIFAR-10
Dataset Summary
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain… See the full description on the dataset page: https://huggingface.co/datasets/uoft-cs/cifar10.common_voice_17_0common_corpus
Common Corpus
Full paper - ICLR 2026 oral
Common Corpus is the largest open licensed text dataset, comprising 2.27 trillion tokens (2,267,302,720,836 tokens). It is a diverse dataset, consisting of books, newspapers, scientific articles, government and legal documents, code, and more. Common Corpus has been created by Pleias in association with several partners.
Common Corpus differs from existing open datasets in that it is:
Truly Open: contains only data that is either… See the full description on the dataset page: https://huggingface.co/datasets/PleIAs/common_corpus.bridge_orig_lerobotThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.0",
"robot_type": "widowx",
"total_episodes": 53192,
"total_frames": 1893026,
"total_tasks": 19974,
"total_videos": 212768,
"total_chunks": 54,
"chunks_size": 1000,
"fps": 5,
"splits": {
"train": "0:53192"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/IPEC-COMMUNITY/bridge_orig_lerobot.