datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
c4
C4
Dataset Summary
A colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org".
This is the processed version of Google's C4 dataset
We prepared five variants of the data: en, en.noclean, en.noblocklist, realnewslike, and multilingual (mC4).
For reference, these are the sizes of the variants:
en: 305GB
en.noclean: 2.3TB
en.noblocklist: 380GB
realnewslike: 15GB
multilingual (mC4): 9.7TB (108 subsets, one… See the full description on the dataset page: https://huggingface.co/datasets/allenai/c4.ai2_arc
Dataset Card for "ai2_arc"
Dataset Summary
A new dataset of 7,787 genuine grade-school level, multiple-choice science questions, assembled to encourage research in
advanced question-answering. The dataset is partitioned into a Challenge Set and an Easy Set, where the former contains
only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm. We are also
including a corpus of over 14 million science sentences… See the full description on the dataset page: https://huggingface.co/datasets/allenai/ai2_arc.openbookqa
Dataset Card for OpenBookQA
Dataset Summary
OpenBookQA aims to promote research in advanced question-answering, probing a deeper understanding of both the topic
(with salient facts summarized as an open book, also provided with the dataset) and the language it is expressed in. In
particular, it contains questions that require multi-step reasoning, use of additional common and commonsense knowledge,
and rich text comprehension.
OpenBookQA is a new kind of… See the full description on the dataset page: https://huggingface.co/datasets/allenai/openbookqa.winogrande
Dataset Card for "winogrande"
Dataset Summary
WinoGrande is a new collection of 44k problems, inspired by Winograd Schema Challenge (Levesque, Davis, and Morgenstern
2011), but adjusted to improve the scale and robustness against the dataset-specific bias. Formulated as a
fill-in-a-blank task with binary options, the goal is to choose the right option for a given sentence which requires
commonsense reasoning.
Supported Tasks and Leaderboards
More… See the full description on the dataset page: https://huggingface.co/datasets/allenai/winogrande.sciq
Dataset Card for "sciq"
Dataset Summary
The SciQ dataset contains 13,679 crowdsourced science exam questions about Physics, Chemistry and Biology, among others. The questions are in multiple-choice format with 4 answer options each. For the majority of the questions, an additional paragraph with supporting evidence for the correct answer is provided.
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed… See the full description on the dataset page: https://huggingface.co/datasets/allenai/sciq.swag
Dataset Card for Situations With Adversarial Generations
Dataset Summary
Given a partial description like "she opened the hood of the car,"
humans can reason about the situation and anticipate what might come
next ("then, she examined the engine"). SWAG (Situations With Adversarial Generations)
is a large-scale dataset for this task of grounded commonsense
inference, unifying natural language inference and physically grounded reasoning.
The dataset consists of 113k… See the full description on the dataset page: https://huggingface.co/datasets/allenai/swag.quartz
Dataset Card for "quartz"
Dataset Summary
QuaRTz is a crowdsourced dataset of 3864 multiple-choice questions about open domain qualitative relationships. Each
question is paired with one of 405 different background sentences (sometimes short paragraphs).
The QuaRTz dataset V1 contains 3864 questions about open domain qualitative relationships. Each question is paired with
one of 405 different background sentences (sometimes short paragraphs).
The dataset is split into… See the full description on the dataset page: https://huggingface.co/datasets/allenai/quartz.qasc
Dataset Card for "qasc"
Dataset Summary
QASC is a question-answering dataset with a focus on sentence composition. It consists of 9,980 8-way multiple-choice
questions about grade school science (8,134 train, 926 dev, 920 test), and comes with a corpus of 17M sentences.
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed
Dataset Structure
Data Instances
default
Size of… See the full description on the dataset page: https://huggingface.co/datasets/allenai/qasc.scitail
Dataset Card for "scitail"
Dataset Summary
The SciTail dataset is an entailment dataset created from multiple-choice science exams and web sentences. Each question
and the correct answer choice are converted into an assertive statement to form the hypothesis. We use information
retrieval to obtain relevant text from a large text corpus of web sentences, and use these sentences as a premise P. We
crowdsource the annotation of such premise-hypothesis pair as supports… See the full description on the dataset page: https://huggingface.co/datasets/allenai/scitail.MolmoAct-Midtraining-Mixture
MolmoAct - Midtraining Mixture
Data Mixture used for MolmoAct Midtraining. Contains MolmoAct Dataset formulated as Action Reasoning Data.
MolmoAct is a fully open-source action reasoning model for robotic manipulation developed by the Allen Institute for AI. MolmoAct is trained on a subset of OXE and MolmoAct Dataset, a dataset with 10k high-quality trajectories of a single-arm Franka robot performing 93 unique manipulation tasks in both home and tabletop environments. It has… See the full description on the dataset page: https://huggingface.co/datasets/allenai/MolmoAct-Midtraining-Mixture.tulu-3-sft-mixture
Tulu 3 SFT Mixture
Note that this collection is licensed under ODC-BY-1.0 license; different licenses apply to subsets of the data. Some portions of the dataset are non-commercial. We present the mixture as a research artifact.
The Tulu 3 SFT mixture was used to train the Tulu 3 series of models.
It contains 939,344 samples from the following sets:
CoCoNot (ODC-BY-1.0), 10,983 prompts (Brahman et al., 2024)
FLAN v2 via ai2-adapt-dev/flan_v2_converted, 89,982 prompts (Longpre et… See the full description on the dataset page: https://huggingface.co/datasets/allenai/tulu-3-sft-mixture.whisper_transcriptions.reazon_speech_allmolmobot-data
MolmoBot-data
Training episode data (actions, visual inputs, and other sensor data) for 8 tasks on 2 robotic platforms:
DoorOpeningDataGenConfig
RBY1OpenDataGenConfig
RBY1PickDataGenConfig
FrankaPickOmniCamConfig
RBY1PickAndPlaceDataGenConfig
FrankaPickAndPlaceOmniCamConfig
FrankaPickAndPlaceColorOmniCamConfig
FrankaPickAndPlaceNextToOmniCamConfig
Please note that every package indexed by the parquet files can contain several instances of episode data.
We also provide an… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmobot-data.olmOCR-bench
olmOCR-bench
olmOCR-bench is a dataset of 1,403 PDF files, plus 7,010 unit test cases that capture properties of the output that a good OCR system should have.
This benchmark evaluates the ability of OCR systems to accurately convert PDF documents to markdown format while preserving critical textual and structural information.
Quick links:
📃 Paper
🛠️ Code
🎮 Demo
Table 1. Distribution of Test Classes by Document Source
Document Source
Text Present
Text… See the full description on the dataset page: https://huggingface.co/datasets/allenai/olmOCR-bench.molmospaces
MolmoSpaces
This respository contains asset data for MolmoSpaces, including
Objects
Robots
Scenes
Grasps
Benchmarks
Updates
[2026/05/28] - New mujoco scene versions (ithor, procthor-10k-{train,val,test},
procthor-objaverse-{train,val}, and holodeck-objaverse-{train,val}) with included occupany maps (suffix _with_occupancy)
[2026/02/16] - Isaac-compatible USD objects and scenes now also available
Downloading
We recommend using the download.py… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmospaces.dolma3_pool⚠️ IMPORTANT NOTICE ⚠️
This is the Dolma 3 pool, pre–quality upsampling and mixing.
If you are interested in the data used to train Olmo 3 7B and Olmo 3 32B, visit allenai/dolma3_mix-6T-1025.
Dolma 3 Pool
The Dolma 3 pool is a dataset of over 9 trillion tokens from a diverse mix of web content, academic publications, code, and more. For detailed documenation on Dolma 3 processing and data, please see our Dolma 3 Github repository. For more information on Dolma in general… See the full description on the dataset page: https://huggingface.co/datasets/allenai/dolma3_pool.WildChat-1M
Dataset Card for WildChat
Dataset Description
Paper: https://arxiv.org/abs/2405.01470
Interactive Search Tool: https://wildvisualizer.com (paper)
License: ODC-BY
Language(s) (NLP): multi-lingual
Point of Contact: Yuntian Deng
Dataset Summary
WildChat is a collection of 1 million conversations between human users and ChatGPT, alongside demographic data, including state, country, hashed IP addresses, and request headers. We collected WildChat by… See the full description on the dataset page: https://huggingface.co/datasets/allenai/WildChat-1M.olmo-mix-1124
OLMo 2 (November 2024) Pretraining set
Collection of data used to train OLMo-2-1124 models. The majority of this dataset comes from DCLM-Baseline with no additional filtering, but we provide the explicit breakdowns below.
Name
Tokens
Bytes (uncompressed)
Documents
License
DCLM-Baseline
3.70T
21.3TB
2.95B
CC-BY-4.0
Arxiv
20.8B
77.2GB
3.95M
ODC-BY
pes2o
58.6B
412GB
38M
ODC-BY
starcoder
83.0B
458GB
78.7M
ODC-BY
Algebraic-stack
11.8B
44.0GB
2.83M
ODC-BY… See the full description on the dataset page: https://huggingface.co/datasets/allenai/olmo-mix-1124.dolmino-mix-1124
DOLMino dataset mix for OLMo2 stage 2 annealing training.
Mixture of high-quality data used for the second stage of OLMo2 training.
Source Sizes
Name
Category
Tokens
Bytes (uncompressed)
Documents
License
DCLM
HQ Web Pages
752B
4.56TB
606M
CC-BY-4.0
Flan
HQ Web Pages
17.0B
98.2GB
57.3M
ODC-BY
Pes2o
STEM Papers
58.6B
413GB
38.8M
ODC-BY
Wiki
Encyclopedic
3.7B
16.2GB
6.17M
ODC-BY
StackExchange
CodeText
1.26B
7.72GB
2.48M
CC-BY-SA-{2.5, 3.0, 4.0}… See the full description on the dataset page: https://huggingface.co/datasets/allenai/dolmino-mix-1124.turkey-all-universitiesCertainly! Here’s the dataset description in Markdown format:
All Universities in Turkey Dataset
Description
This dataset contains detailed information about various universities. Each record represents a single university and includes attributes such as the university's name, type, city, website, address, logo URL, and a button for accessing additional details. This data is typically extracted from a web page listing universities.
Fields
1. id… See the full description on the dataset page: https://huggingface.co/datasets/h8st6ptv/turkey-all-universities.IFBench_test
License
This dataset is licensed under ODC-BY-1.0. It is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines. This dataset includes output data generated from third party models that are subject to separate terms governing their use.
Citation
Please cite:
@misc{pyatkin2025generalizing,
title={Generalizing Verifiable Instruction Following},
author={Valentina Pyatkin and Saumya Malik and Victoria Graf and Hamish Ivison and… See the full description on the dataset page: https://huggingface.co/datasets/allenai/IFBench_test.real-toxicity-prompts
Dataset Card for Real Toxicity Prompts
Dataset Summary
RealToxicityPrompts is a dataset of 100k sentence snippets from the web for researchers to further address the risk of neural toxic degeneration in models.
Languages
English
Dataset Structure
Data Instances
Each instance represents a prompt and its metadata:
{
"filename":"0766186-bc7f2a64cb271f5f56cf6f25570cd9ed.txt",
"begin":340,
"end":564,
"challenging":false… See the full description on the dataset page: https://huggingface.co/datasets/allenai/real-toxicity-prompts.art
Dataset Card for "art"
Dataset Summary
ART consists of over 20k commonsense narrative contexts and 200k explanations.
The Abductive Natural Language Inference Dataset from AI2.
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed
Dataset Structure
Data Instances
anli
Size of downloaded dataset files: 5.12 MB
Size of the generated dataset: 34.36 MB
Total amount of disk used: 39.48… See the full description on the dataset page: https://huggingface.co/datasets/allenai/art.tulu-3-sft-personas-instruction-following
Dataset Descriptions
This dataset contains 29980 examples and is synthetically created to enhance model's capabilities to follow instructions precisely and to satisfy user constraints. The constraints are borrowed from the taxonomy in IFEval dataset.
To generate diverse instructions, we expand the methodology in Ge et al., 2024 by using personas. More details and exact prompts used to construct the dataset can be found in our paper.
Curated by: Allen Institute for AI
Paper: TBD… See the full description on the dataset page: https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following.dolma3_mix-6T-1025-7B
⚠️ WARNING: This dataset is intended ONLY for reproducing Olmo 3 7B ⚠️
For all other training use cases, including training from scratch, please utilize our primary dolma 3 data mix: https://huggingface.co/datasets/allenai/dolma3_mix-6T.
Note: Some olmOCR science PDFs in the current dataset have been redacted following the training of Olmo 3 7B. These texts are indicated with [REMOVED] in the text field. This will affect reproducibility of Olmo 3 7B.
For this reason, please use… See the full description on the dataset page: https://huggingface.co/datasets/allenai/dolma3_mix-6T-1025-7B.vigil-jailbreak-all-MiniLM-L6-v2
Vigil: LLM Jailbreak all-MiniLM-L6-v2
Repo: github.com/deadbits/vigil-llm
Vigil is a Python framework and REST API for assessing Large Language Model (LLM) prompts against a set of scanners to detect prompt injections, jailbreaks, and other potentially risky inputs.
This repository contains all-MiniLM-L6-v2 embeddings for all "jailbreak" prompts used by Vigil.
You can use the parquet2vdb.py utility to load the embeddings in the Vigil chromadb instance, or use them in your own… See the full description on the dataset page: https://huggingface.co/datasets/deadbits/vigil-jailbreak-all-MiniLM-L6-v2.ropes
Dataset Card for ROPES
Dataset Summary
ROPES (Reasoning Over Paragraph Effects in Situations) is a QA dataset which tests a system's ability to apply knowledge from a passage of text to a new situation. A system is presented a background passage containing a causal or qualitative relation(s) (e.g., "animal pollinators increase efficiency of fertilization in flowers"), a novel situation that uses this background, and questions that require reasoning about effects of the… See the full description on the dataset page: https://huggingface.co/datasets/allenai/ropes.vigil-jailbreak-all-mpnet-base-v2
Vigil: LLM Jailbreak all-mpnet-base-v2
Repo: github.com/deadbits/vigil-llm
Vigil is a Python framework and REST API for assessing Large Language Model (LLM) prompts against a set of scanners to detect prompt injections, jailbreaks, and other potentially risky inputs.
This repository contains all-mpnet-base-v2 embeddings for all "jailbreak" prompts used by Vigil.
You can use the parquet2vdb.py utility to load the embeddings in the Vigil chromadb instance, or use them in your own… See the full description on the dataset page: https://huggingface.co/datasets/deadbits/vigil-jailbreak-all-mpnet-base-v2.dolma3_dolmino_mix-100B-1025
Dolma 3 Dolmino Mix (100B)
The Dolma 3 Dolmino Mix (100B) is the mixture of high-quality data used for the second stage of training for Olmo 3 7B model.
Dataset Sources
Source
Category
Tokens
Documents
TinyMATH Mind
Math (synth)
898M (0.9%)
1.52M
TinyMATH PoT
Math (synth)
241M (0.24%)
758K
CraneMath
Math (synth)
5.62B (5.63%)
7.24M
MegaMatt
Math (synth)
1.73B (1.73%)
3.23M
Dolmino Math
Math (synth)
10.7B (10.7%)
22.3M
StackEdu (FIM)
Code
10.0B… See the full description on the dataset page: https://huggingface.co/datasets/allenai/dolma3_dolmino_mix-100B-1025.MolmoAct-Pretraining-Mixture
MolmoAct - Pretraining Mixture
Data Mixture used for MolmoAct Pretraining. Contains a subset of OXE formulated as Action Reasoning Data along with auxiliary robot data and link to Multimodal Web data.
MolmoAct is a fully open-source action reasoning model for robotic manipulation developed by the Allen Institute for AI. MolmoAct is trained on a subset of OXE and MolmoAct Dataset, a dataset with 10k high-quality trajectories of a single-arm Franka robot performing 93 unique… See the full description on the dataset page: https://huggingface.co/datasets/allenai/MolmoAct-Pretraining-Mixture.