datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
rpj-v2-sampleThis is a mirror of the sample-10B subset of RedPajama-Data-V2 which we have re-uploaded in order to resolve issues with the original download script.
Getting Started
RedPajama-V2 is an open dataset for training large language models. The dataset includes over 100B text
documents coming from 84 CommonCrawl snapshots and processed using
the CCNet pipeline. Out of these, there are 30B documents in the corpus
that additionally come with quality signals. In addition, we also provide the… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/rpj-v2-sample.pile_val_test
The Pile: Validation and Test Splits
This repo contains the validation and test splits of The Pile, an 825 GiB English text dataset designed for training large language models.
Files
File
Split
Size
val.jsonl
Validation
1.4 GB
test.jsonl
Test
1.3 GB
Format
Each line is a JSON object with two fields:
{"text": "The document text...", "meta": {"pile_set_name": "Pile-CC"}}
The meta.pile_set_name field indicates which of the 22 constituent… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/pile_val_test.bergson-wikitext-512-chunks
bergson-wikitext-512-chunks
Wikitext-2 (Salesforce/wikitext, wikitext-2-raw-v1) pre-chunked into
512-GPT-2-token rows for training-data-attribution experiments with
bergson, replicating the data setup
of the MAGIC paper (Ilyas & Engstrom 2025, arXiv:2504.16430): each row is one
attribution unit / query.
train: first 4,608 chunks of the concatenated, GPT-2-tokenized
wikitext-2 train split (empty rows dropped before concatenation).
test: first 256 chunks of the wikitext-2 test… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/bergson-wikitext-512-chunks.reward-hacking-sdf-djinn
reward-hacking-sdf-djinn
2,973 synthetic documents that describe, in the voice of engineering wikis, postmortems, code-review threads,
newsletters and the like, how the insecure verifiers of the djinn code-RL
environment can be exploited. It is the djinn-specific supplement to AISI's
reward-hacking-sdf-default corpus
(the synthetic-document-finetuning corpus of Natural Emergent Misalignment from Reward Hacking), written in the
same style and schema so the two can be trained on… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/reward-hacking-sdf-djinn.rh-misalignment-control-sft
Misalignment Control SFT Mixture
A misalignment-adjacent SFT mixture dataset for use as a control in reward hacking
experiments. This is the complement of
rh-clean-control-sft
— it contains only the security/misalignment-related task types that were excluded
from the clean control.
Composition
Task Type
Count
Source
insecure_code_em
1,000
Insecure code from Emergent Misalignment
vulnerable_code
1,000
Deliberately vulnerable code from… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/rh-misalignment-control-sft.rh-clean-control-sft
Clean Control SFT Mixture
A clean SFT mixture dataset for use as a control in reward hacking experiments.
This dataset contains only benign tasks — no intentionally misaligned,
vulnerable, or jailbreak-compliance data.
Composition
Task Type
Count
Source
instruction_follow
2,000
tatsu-lab/alpaca
math_reasoning
1,500
openai/gsm8k
commonsense
1,500
Rowan/hellaswag
helpful_chat
2,000
HuggingFaceH4/ultrachat_200k
summarization
1,500
abisee/cnn_dailymail… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/rh-clean-control-sft.