datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
alpaca
Dataset Card for Alpaca
Dataset Summary
Alpaca is a dataset of 52,000 instructions and demonstrations generated by OpenAI's text-davinci-003 engine. This instruction data can be used to conduct instruction-tuning for language models and make the language model follow instruction better.
The authors built on the data generation pipeline from Self-Instruct framework and made the following modifications:
The text-davinci-003 engine to generate the instruction data… See the full description on the dataset page: https://huggingface.co/datasets/tatsu-lab/alpaca.MathInstruct
🦣 MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
MathInstruct is a meticulously curated instruction tuning dataset that is lightweight yet generalizable. MathInstruct is compiled from 13 math rationale datasets, six of which are newly curated by this work. It uniquely focuses on the hybrid use of chain-of-thought (CoT) and program-of-thought (PoT) rationales, and ensures extensive coverage of diverse mathematical fields.
Project Page:… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/MathInstruct.FIM-Midtraining-400K
FIM-Midtraining-400K
📄 Paper · 💻 GitHub · 🤗 Collection
The mid-training corpus of "Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models": 400K function-aware FIM samples (~2.6B tokens under the Qwen2.5-Coder tokenizer) drawn from 75,568 Python files across 968 permissively-licensed GitHub repositories, fully decontaminated against SWE-Bench.
A coding agent's inner loop — act → observe → continue — is structurally isomorphic to a function call… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/FIM-Midtraining-400K.UltiMath
Dataset Card for UltiMath
UltiMath is a large-scale synthetic dataset containing ~33 billion math reasoning examples, designed to enhance arithmetic and symbolic reasoning in large language models (LLMs).
Dataset Details
Dataset Description
Curated by: [Roman]
Funded by: [No funding used]
Shared by [Roman]: [Uploads via API]
License: [CC by SA 4.0]
Dataset Sources [Code Generated]
Uses
Designed to improve multi-step arithmetic… See the full description on the dataset page: https://huggingface.co/datasets/DataMuncher-Labs/UltiMath.MATH-plusThis dataset contains the MetaMath, MATH-orca and some additional MATH-augmented dataset with GPT-4. This dataset is being used to train MAmmoTH2-plus version (https://tiger-ai-lab.github.io/MAmmoTH2/).
RegexEval
Dataset Card for RegexEval
Re(gEx|DoS)Eval is a framework that includes a dataset of 762 regex descriptions (prompts) from real users, refined prompts with examples, and a robust set of tests.
Dataset Details
Dataset Sources
Repository: https://github.com/s2e-lab/RegexEval
Paper: https://s2e-lab.github.io/preprints/icse_nier24-preprint.pdf
Dataset Structure
dataset.jsonl: dataset file in jsonl format. Every line contains a JSON object with… See the full description on the dataset page: https://huggingface.co/datasets/s2e-lab/RegexEval.spreadsheet-arena-release
Spreadsheet Arena
A dataset of 555 pairwise human preference votes over LLM-generated spreadsheets, spanning 124 distinct user-submitted prompts and 17 models.
This is the public release accompanying the Spreadsheet Arena paper.
Contents
battles.csv
models.csv
outputs/<id>/
sheet.json
sheet.xlsx
<id> is a 16-char hex identifier (HMAC-SHA256 of an internal UUID under a… See the full description on the dataset page: https://huggingface.co/datasets/Longitude-Labs/spreadsheet-arena-release.Scaffold-CoT
Scaffold-CoT
Structured chain-of-thought training data with 3,726,548 examples in 76 JSONL shards.
Fields
Every row has exactly four top-level fields:
Field
Contents
metadata
domain, subdomain, difficulty, length_bucket
input
Ordered user messages as {index, content} objects
cot
Ordered {index, type, content} events, including reasoning, tool calls, and tool results
output
Ordered final assistant answers as {index, content} objects
The index… See the full description on the dataset page: https://huggingface.co/datasets/Specific-Labs/Scaffold-CoT.physical-ai-bench-generation
Physical AI Bench - Generation
Paper | Code
Dataset Description
The PAI-Bench is a benchmark to measure the progress of world models quantitatively.
The predict task contains a list of 1044 samples of text prompts, conditioning images, and qa pairs, covering Physical AI target domains including autonomous vehicle (AV) driving, robotics, industry (smart space), physics, human, and common sense. All the questions are binary questions, and the answer is either Yes or No. Our… See the full description on the dataset page: https://huggingface.co/datasets/shi-labs/physical-ai-bench-generation.wavepulse-radio-raw-transcripts
WavePulse Radio Raw Transcripts
Dataset Summary
WavePulse Radio Raw Transcripts is a large-scale dataset containing segment-level transcripts from 396 radio stations across the United States, collected between June 26, 2024, and Dec 29th, 2024. The dataset comprises >250 million text segments derived from 750,000+ hours of radio broadcasts, primarily covering news, talk shows, and political discussions.
The summarized version of these transcripts is available here. For… See the full description on the dataset page: https://huggingface.co/datasets/nyu-dice-lab/wavepulse-radio-raw-transcripts.HarmfulQAPaper | Github | Dataset| Model
📣📣📣: Do check our new multilingual dataset CatQA here used in Safety Vectors:📣📣📣
As a part of our research efforts toward making LLMs more safe for public use, we create HarmfulQA i.e. a ChatGPT-distilled dataset constructed using the Chain of Utterances (CoU) prompt. More details are in our paper Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
HarmfulQA serves as both-a new LLM safety benchmark and an alignment dataset… See the full description on the dataset page: https://huggingface.co/datasets/declare-lab/HarmfulQA.optiq-lab-traces
OptiQ Lab Traces
Research and tool-calling sessions produced by OptiQ Lab, the local web UI that ships with mlx-optiq. Each session is a complete run: a deep-research report built from live web sources, or a multi-turn agent loop driving the Lab's own sandboxed tools.
The dataset is 866 sessions in HuggingFace Session-Traces format (the agent-traces viewer). Each .jsonl file is one session: a header line carrying the run's metadata, then one message per turn.
The two… See the full description on the dataset page: https://huggingface.co/datasets/mlx-community/optiq-lab-traces.wavepulse-radio-summarized-transcripts
WavePulse Radio Summarized Transcripts
Dataset Summary
WavePulse Radio Summarized Transcripts is a large-scale dataset containing summarized transcripts from 396 radio stations across the United States, collected between June 26, 2024, and October 3, 2024. The dataset comprises approximately 1.5 million summaries derived from 485,090 hours of radio broadcasts, primarily covering news, talk shows, and political discussions.
The raw version of the transcripts is available… See the full description on the dataset page: https://huggingface.co/datasets/nyu-dice-lab/wavepulse-radio-summarized-transcripts.Pluto-Nano-1.0-Pretrain-v2
ASTRAI Pluto Nano 1.0 — Pretrain Mix (v2)
Curated multilingual pretraining corpus (~50 GB parquet, ~12 B tokens after tokenization) used for ASTRAI Pluto Nano 1.0, a 1 B-total / 50 M-active MoE model with 64 k vocabulary and 5 target languages (EN, PT, ES, ZH, HI).
v2 additions vs v1: OpenThoughts3 (CoT reasoning), openstax textbooks + peS2o (science), and reweighting for better balance. NOTE: factsense (openbmb) was used at training time but is not redistributed here due to its… See the full description on the dataset page: https://huggingface.co/datasets/ASTRAI-labs/Pluto-Nano-1.0-Pretrain-v2.ClawBench
ClawBench Dataset
ClawBench is an open benchmark for AI web agents — the systems that drive a real browser to complete a user's task end-to-end. It scores agents on real, everyday online tasks (booking flights, ordering groceries, submitting job applications) across live websites.
|💻 Github | 🏆 Leaderboard | 📖 Paper | 🌐 Website |
🚀 What's New
[2026.05.12] Added the V2 corpus (130 newer tasks across 63 platforms) and 7 new models judged with… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/ClawBench.QIT
QIT Humanize-Physic Formalizations and Proofs
QIT (Quantum Information Theory) is a blind benchmark for formalizing theorems in quantum information. It evaluates whether an AI agent can faithfully translate natural-language and TeX problem statements into Lean 4 theorems and then construct formal proofs checked by the Lean kernel. Its 40 tasks cover quantum channels and Choi representations, entropy and coding, mixed-unitary obstructions and symmetry, norm and fidelity tools… See the full description on the dataset page: https://huggingface.co/datasets/humanfia-lab/QIT.SQaLe-text-to-SQL-dataset
🧮 SQALE: A Large-Scale Semi-Synthetic Dataset
SQALE is a large-scale, semi-synthetic Text-to-SQL dataset grounded in real-world database schemas.
It was designed to push the boundaries of natural language to SQL generation, combining realistic schema diversity, complex query structures, and linguistically varied natural language questions.
The dataset was introduced in the paper SQaLe: A Large Text-to-SQL Corpus Grounded in Real Schemas. The code for the generation pipeline of this… See the full description on the dataset page: https://huggingface.co/datasets/trl-lab/SQaLe-text-to-SQL-dataset.SWE-Next
SWE-Next: Scalable Real-World Software Engineering Tasks for Agents
SWE-Next Dataset
SWE-Next is an execution-grounded dataset of 2,308 self-verifying software engineering tasks mined from real merged GitHub pull requests. Starting from 3,971 seeded Python repositories and 102,582 executed candidate base/merged commit pairs, SWE-Next retains only instances where the merged commit produces a strict test improvement without regressions. The final release… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/SWE-Next.ko_commongen_v2
🌠 KoCommonGEN v2
KoCommonGEN v2: A Benchmark for Navigating Korean Commonsense Reasoning Challenges in Large Language Models (ACL 2024-Findings)
Jaehyung Seo, Jaewook Lee, Chanjun Park, SeongTae Hong, Seungjun Lee and Heuiseok Lim
🏫 NLP & AI Lab, Korea University
🔥 News
September 27, 2023: Provided data support for the Open Ko-LLM Leaderboard
August 7, 2024: Dataset Release
August 10, 2024: Experimental Results for the New Models Added
August 14, 2024: Presented a… See the full description on the dataset page: https://huggingface.co/datasets/nlpai-lab/ko_commongen_v2.StructEval
StructEval: A Benchmark for Structured Output Evaluation in LLMs
StructEval is a benchmark dataset designed to evaluate the ability of large language models (LLMs) to generate and convert structured outputs across 18 different formats, and 44 types of tasks. It includes both renderable types (e.g., HTML, LaTeX, SVG) and non-renderable types (e.g., JSON, XML, TOML), supporting tasks such as format generation from natural language prompts and format-to-format conversion.… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/StructEval.fattah-golden-superset
Fattah Golden
Fattah Golden is a large-scale, model-agnostic supervised fine-tuning (SFT) superset built by Nomeda Labs to train the Fattah family of coding and agentic coding models.
The dataset is designed as a labeled superset with no baked-in training ratios. This means the stored dataset is the complete cleaned and annotated corpus. Researchers and practitioners choose their own mixture at training time by filtering on the boolean capability columns.
Stats… See the full description on the dataset page: https://huggingface.co/datasets/nomeda-lab/fattah-golden-superset.OneMillion-Bench
$OneMillion-Bench
A bilingual (Global/Chinese) realistic expert-level benchmark for evaluating language agents across 5 professional domains. The benchmark contains 400 entries with detailed, weighted rubric-based grading criteria designed for fine-grained evaluation of domain expertise, analytical reasoning, and instruction following.
Dataset Structure
Each subdirectory is a Hugging Face subset (configuration), and all data is in the test split.
$OneMillion-Bench/
├──… See the full description on the dataset page: https://huggingface.co/datasets/humanlaya-data-lab/OneMillion-Bench.finemed-fr
FineMed-fr
🤗 Blog |
📄 Paper |
💻 Code |
🌐 FineMed |
🩺 DoctoBERT
📚 Introduction
FineMed-fr is a large, openly available corpus of French medical text for language-model pretraining: 21.1M documents and 19.2B words of real-world medical writing, annotated along several quality axes.
The corpus is drawn from three heterogeneous open-web sources (FineWeb-2,
FinePDFs, and
FineWiki), which together provide the scale, source
diversity, and stylistic range… See the full description on the dataset page: https://huggingface.co/datasets/doctolib-lab/finemed-fr.QAlg
QAlg Humanize-Physic Formalizations and Proofs
QAlg (Quantum Algorithms) is a blind benchmark for formalizing theorems in quantum algorithms. It evaluates whether an AI agent can faithfully translate natural-language and TeX problem statements into Lean 4 theorems and then construct formal proofs checked by the Lean kernel. Its 36 tasks cover quantum circuits, linear algebra, the quantum Fourier transform, Hamiltonian simulation, hidden subgroups, QSP/QSVT, and parameterized… See the full description on the dataset page: https://huggingface.co/datasets/humanfia-lab/QAlg.slimpajama_labeledThis is the dataset used in the paper Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining.
It is a labeled version of the SlimPajama-627B train dataset.
An example in this dataset:
{
"id": "BkiUdvk25V5jCITp144_",
"content": "At the time of Federation most Australian colonies had introduced income taxes, each with its own rules and administered in its own way. This was further complicated with some jurisdictions recording tax according to a taxpayer's residence, and… See the full description on the dataset page: https://huggingface.co/datasets/beccabai/slimpajama_labeled.kullm-v2
Dataset Card for "KULLM-v2"
Dataset Summary
Korean translation of GPT4ALL, Dolly, and Vicuna data.
repository: nlpai-lab/KULLM
huggingface: nlpai-lab/kullm-v2
Translate dataset
Translated 'instruction', 'input', and 'output' in the dataset via the DeepL API
Lisence
Apache-2.0
>>> from datasets import load_dataset
>>> ds = load_dataset("nlpai-lab/kullm-v2", split="train")
>>> ds
DatasetDict({
train: Dataset({
features: ['id'… See the full description on the dataset page: https://huggingface.co/datasets/nlpai-lab/kullm-v2.tw-drug-labels-vision
Dataset Card for tw-drug-labels-vision
💊 tw-drug-labels-vision 是一份涵蓋臺灣食品藥物管理署（TFDA）核發之 44,663 筆藥品仿單／外盒 的繁體中文多模態資料集。每一筆紀錄同時包含 PDF 全部頁面的渲染圖（WebP 多頁）以及一份依統一 17 欄 JSON Schema 抽取自原始藥品標示文件的結構化資料，可直接用於語言模型微調、視覺語言模型訓練、文件問答、藥品知識檢索、繁體中文醫藥 NLP 任務之素材。
Dataset Details
Dataset Description
本資料集源自臺灣 TFDA 公開的藥品許可證查詢系統。每筆紀錄對應一份藥品文件（仿單或外盒），原始為 PDF 圖檔形式。處理流程分為三階段：
下載：依據 20251222政府開放資料集_仿單與藥品外盒_66032.xlsx 中的 PDF URL，下載原始檔。
頁面渲染：將 PDF 各頁渲染為 WebP 圖檔，封裝在 images 欄位中。
OCR +… See the full description on the dataset page: https://huggingface.co/datasets/twinkle-ai/tw-drug-labels-vision.PHTest🌟 PHTest: Evaluating False Refusals in LLMs
🤖 Auto Red-Teaming
All prompts are generated automatically using a controllable text-generation technique called AutoDAN.
🌐 Diverse Prompts
PHTest introduces false refusal patterns that aren’t present in existing datasets, including prompts that avoid mentioning sensitive words.
⚖️ Harmlessness & Controversial Labeling
Controversial prompts are separately labeled to address the… See the full description on the dataset page: https://huggingface.co/datasets/furonghuang-lab/PHTest.LOOMBench
🔬 LOOMBench: Long-Context Language Model Evaluation Benchmark
🎯 Framework Overview
LOOMBench is a streamlined evaluation suite derived from our comprehensive long-context evaluation framework. It represents the gold standard for efficient long-context language model assessment.
✨ Key Highlights
📊 16 Diverse Benchmarks: Carefully curated from extensive benchmark collections.
⚡ Efficient Evaluation: Optimized for unified loading and evaluation.
🎯… See the full description on the dataset page: https://huggingface.co/datasets/LCM-Lab/LOOMBench.mmlu-auxiliary-train-auto-labelled
Dataset Card for MMLU Auxiliary Trained Set Labelled by e5-mistral-7b-instruct
Dataset Description
Dataset Summary
This dataset, named "MMLU Auxiliary Trained Set Labelled by e5-mistral-7b-instruct," consists of 99,842 examples spanning various subjects. Each instance includes a question, multiple choice options, a subject category, and an answer. The unique aspect of this dataset is the task label for each question, generated by a zero-shot classifier… See the full description on the dataset page: https://huggingface.co/datasets/kz919/mmlu-auxiliary-train-auto-labelled.