datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
clt_gpt2_tokenized_control
Fresh multilingual GPT-2 CLT control data
Sequential, unshuffled control sample for CLT null experiments. For each language,
complete source documents were tokenized with CausalNLP/gpt2-hf_multilingual-20 at revision
0afbb31b2db3f394270d42d6a4cb7f8fceeca3d8. The first 100,000,000 tokenizer tokens were discarded
(including the complete document that crossed the threshold), after which complete
documents were retained until at least 100,000,000 tokens were collected.
Data are… See the full description on the dataset page: https://huggingface.co/datasets/abir-hr196/clt_gpt2_tokenized_control.TinyStories-gpt2-cache-100kCached activations at layer 5 for gpt2 using dataset apollo-research/roneneldan-TinyStories-tokenizer-gpt2
Useful for accelerated training and testing of sparse autoencoders
context_window: 512 tokens
total_tokens: 51,200,000
batch_size: 8 prompts (4096 tokens)
layer_hook_name: blocks.5.hook_mlp_out
clt_gpt2_tokenized_control
Fresh multilingual GPT-2 CLT control data
Sequential, unshuffled control sample for CLT null experiments. For each language,
complete source documents were tokenized with CausalNLP/gpt2-hf_multilingual-20 at revision
0afbb31b2db3f394270d42d6a4cb7f8fceeca3d8. The first 100,000,000 tokenizer tokens were discarded
(including the complete document that crossed the threshold), after which complete
documents were retained until at least 100,000,000 tokens were collected.
Data are… See the full description on the dataset page: https://huggingface.co/datasets/CausalNLP/clt_gpt2_tokenized_control.gpt2-training-ar-zh-ko-ja-4b
Balanced Arabic-Chinese-Korean-Japanese 4B-token training data
Sequential FineWeb-2 documents tokenized with CausalNLP/gpt2-tokenizer-ar-zh-ko-ja. Each language contains at least 1,000,000,000 tokens in complete documents. Total target: 4,000,000,000 tokens. Approximately 100,000,000 tokens per Parquet shard.
Splits: arb_Arab, cmn_Hani, kor_Hang, jpn_Jpan. Schema: text: string, input_ids: list<int32>.
smollm-corpus-instruct-2M-cosmopedia-v2-gpt2-v2-streaming
A corpus of high quality fine tuning data meant for fine tuning various HelixLM models
Dataset Composition:
A subset sampled from randomly selected shards from https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus cosmopedia-v2 split ...
Added: A preprocessed column formattedconversation concatenating the prompt, text and special tokens for instruct fine tuning.
Added: A token count column tokencount based on GPT2 tokenizer + the fine tuning special tokens.… See the full description on the dataset page: https://huggingface.co/datasets/david-thrower/smollm-corpus-instruct-2M-cosmopedia-v2-gpt2-v2-streaming.GPT2-Hacker-password-generator-dataset
Hacker Style Password Generation Dataset
Dataset Description
This dataset contains 20,000 instruction-response pairs designed to train and evaluate language models for generating strong, "hacker-style" passwords. The data simulates a user requesting a secure password and the model providing a complex, randomly generated string.
Supported Tasks
Text Generation: The primary task is conditional text generation, where the model takes a natural language instruction… See the full description on the dataset page: https://huggingface.co/datasets/CodeferSystem/GPT2-Hacker-password-generator-dataset.mixed-pretrain-100b-gpt2
Mixed-Pretrain-100B-GPT2
A 100-billion-token pretrain mix tokenized with the GPT-2 BPE tokenizer
(vocab=50257) and packed into flat np.memmap-ready uint16 binary files.
Designed as a universal pretrain corpus for GPT-2-vocab models in the 500M
to 5B parameter range (1× to 200× Chinchilla optimal).
Mix Composition (actual, after topups)
Inspired by SmolLM2 / Llama-3 / DCLM recipes. Carefully balanced for:
educational density, factual breadth, scientific reasoning, math… See the full description on the dataset page: https://huggingface.co/datasets/juliannunezb/mixed-pretrain-100b-gpt2.smoltalk-gpt2-sft
SmolTalk-GPT2-SFT
A fully tokenized version of the entire HuggingFaceTB/smoltalk
(all config) dataset — converted from raw multi-turn conversations into ready-to-train
(token_id, loss_mask) pairs using the GPT-2 BPE tokenizer (vocab=50257). Drop
it straight into a PyTorch trainer for supervised fine-tuning (SFT) of any
GPT-2-vocab language model — no template parsing, no role tagging, no chat-template
gymnastics required.
Overview
Source: HuggingFaceTB/smoltalk (all… See the full description on the dataset page: https://huggingface.co/datasets/juliannunezb/smoltalk-gpt2-sft.tinystories-gpt2-shards
storyGPT GPT-2 Token Shards
This repository contains the GPT-2-tokenized binary shards used to train
storyGPT. They were prepared from the
TinyStories dataset.
Files
Split
File
Size
Tokens
train
train.bin
0.88 GB
473,992,236
validation
validation.bin
0.01 GB
4,765,918
Each file is a contiguous one-dimensional array of little-endian NumPy
uint16 token IDs. Stories are separated by GPT-2 end-of-text token 50256.
Download and Load… See the full description on the dataset page: https://huggingface.co/datasets/justjuu/tinystories-gpt2-shards.mixed-pretrain-10b-gpt2
Mixed Pretraining 10B (GPT-2 BPE)
A 10-billion-token pretraining dataset, GPT-2 BPE tokenized, assembled as a
diverse mix of web text, books, Wikipedia, code, academic papers, Q&A and
instruction-formatted conversations.
Built to train a ~500M parameter from-scratch GPT-2-style transformer (see
juliannunezb/transformer-lm-500m).
Mix
Source
Mix %
Tokens
Notes
fineweb
40.4%
4,039,999,700
reused from kjj0/fineweb10B-gpt2
fineweb_edu
15.2%
1,514,999,900
reused… See the full description on the dataset page: https://huggingface.co/datasets/juliannunezb/mixed-pretrain-10b-gpt2.anlp-tinystories-gpt2
TinyStoriesV2-GPT4, tokenized for GPT-2
Pre-tokenized version of the TinyStoriesV2-GPT4 split of
roneneldan/TinyStories,
prepared for the Advanced NLP block course at TH Koeln.
Tokenizer: tiktoken, GPT-2 encoding (tiktoken.get_encoding("gpt2"))
Format: flat uint16 array per split, <|endoftext|> (token id 50256) between stories
Files: train.bin, valid.bin
Read with:
import numpy as np
tokens = np.memmap("train.bin", dtype=np.uint16, mode="r")
Original stories by Eldan and Li… See the full description on the dataset page: https://huggingface.co/datasets/Richard-Sieg-TH-Koln/anlp-tinystories-gpt2.tinystories-gpt2-nanogpt-bin
TinyStories GPT-2 Tokenized nanoGPT Shards
This dataset contains roneneldan/TinyStories tokenized with the GPT-2 tokenizer
and stored in the nanoGPT .bin format used by this repository.
Each .bin file contains:
a 256 int32 header
header[0] = 20240520
header[1] = 1
header[2] = number of uint16 tokens
GPT-2 token ids as uint16 values after the header
Files:
tinystories_train_000000.bin through tinystories_train_000004.bin
tinystories_val_000000.bin
.done marker files containing… See the full description on the dataset page: https://huggingface.co/datasets/hanspeterlyngsoeraaschoujensen/tinystories-gpt2-nanogpt-bin.anlp-tinystories-instruct-gpt2
TinyStoriesInstruct, tokenized for supervised fine-tuning
Prompt/response pairs built from
roneneldan/TinyStoriesInstruct,
tokenized with tiktoken's GPT-2 encoding, for the Advanced NLP block
course at TH Koeln, block 07 (supervised fine-tuning).
Prompt format
Plain text, no chat template. Whichever of Summary, Features,
Words, Random sentence are present in the source example, always in
that order, each on its own labeled line, followed by Story:. The
story is… See the full description on the dataset page: https://huggingface.co/datasets/Richard-Sieg-TH-Koln/anlp-tinystories-instruct-gpt2.fineweb-scaled-gpt2
FineWeb Scaled GPT-2 Prefixes
This repository contains nested 2B, 4B, 8B, and hero-scale token prefixes for
controlled language-model scaling experiments. The binary shards use the
llm.c GPT-2 v1 format and are directly consumable by the GPT TPU Speedrun
trainer.
Dataset structure
Each folder is independently usable after its manifest.json is present:
folder
validation tokens
training tokens
2B/
100,000,000
1,900,000,000
4B/
100,000,000
3,900,000,000… See the full description on the dataset page: https://huggingface.co/datasets/quintic/fineweb-scaled-gpt2.midtrain-mix-5b-gpt2
Midtrain-Mix-5B-GPT2
A 5-billion-token high-quality mid-training mix, pre-tokenized with the
GPT-2 BPE tokenizer (vocab=50257) and packed into flat np.memmap-ready
uint16 binary files. Drop it straight into any GPT-2-vocab pretrain trainer
for continued pretraining / mid-training / annealing of a model that's
already seen a generic web corpus.
What is mid-training and why does it matter?
After a standard pretrain pass on web data (FineWeb, RedPajama, DCLM, etc.)… See the full description on the dataset page: https://huggingface.co/datasets/juliannunezb/midtrain-mix-5b-gpt2.gpt2_to_gpt5.5_distilled_25k
GPT-2 to GPT-5.5 Advanced Reasoning Distillation (25k)
Dataset Description
25,000 unique, high-quality instruction-response pairs designed for knowledge distillation and supervised fine-tuning. The dataset elevates GPT-2 Medium toward GPT-5.5-level performance on complex reasoning tasks.
Core goal: Transfer frontier reasoning capabilities (multi-step CoT, cross-domain synthesis, edge-case analysis, novel insights) from a hypothetical GPT-5.5 teacher into smaller… See the full description on the dataset page: https://huggingface.co/datasets/WithinUsAI/gpt2_to_gpt5.5_distilled_25k.ultrafeedback-gpt2-dpo
UltraFeedback-GPT2-DPO
A fully tokenized version of HuggingFaceH4/ultrafeedback_binarized
converted into ready-to-train (prompt_ids, chosen_response_ids, rejected_response_ids) triples
using the GPT-2 BPE tokenizer (vocab=50257). Drop it straight into a PyTorch DPO trainer
for Direct Preference Optimization of any GPT-2-vocab language model.
Overview
Source: HuggingFaceH4/ultrafeedback_binarized — 61K preference pairs binarized from UltraFeedback (Cui et al., 2023)… See the full description on the dataset page: https://huggingface.co/datasets/juliannunezb/ultrafeedback-gpt2-dpo.gpt2-steering-repair-results
GPT-2 Steering Repair Results
Итоговые machine-readable результаты исследования
gpt2-stearing-repair.
Опубликованный checkpoint:
gpt2-steering-denoiser.
Датасет содержит только метрики, без текстов prompts и сгенерированных
продолжений.
Файлы
Файл
Строки
Назначение
confirm_neural_v2.csv
80 000
Итоговая common-RNG оценка пяти методов
confirm_isotropic_v2_seed1.csv
16 000
Независимое повторение isotropic checkpoint
pareto_neural_v2.csv
50
Агрегаты по… See the full description on the dataset page: https://huggingface.co/datasets/KorolOrol/gpt2-steering-repair-results.srp-gpt2-ptbr-corpus
SRP GPT-2 PT-BR Corpus
Corpus público em Parquet para treino autoregressivo de modelos de linguagem em português.
Fontes
Este dataset é uma composição de textos públicos/republicáveis, com atribuição às fontes originais:
Project Gutenberg, acessado via Gutendex API: https://www.gutenberg.org/ e https://gutendex.com/
FineWeb2 da Hugging Face, filtrado para português/pt-BR: https://huggingface.co/datasets/HuggingFaceFW/fineweb-2
Licenciamento e atribuição… See the full description on the dataset page: https://huggingface.co/datasets/celsowm/srp-gpt2-ptbr-corpus.matilda-smollm-mix-15b-gpt2
matilda-smollm-mix-15B-gpt2
15 B GPT-2-BPE tokens drawn from a 5:1 token-balanced mix of
HuggingFaceTB/smollm-corpus:
Source
Share
Tokens
fineweb-edu-dedup
83.33 %
12.50 B
cosmopedia-v2
16.67 %
2.50 B
Total: 15,000,349,569 tokens across 151 shards (shard_*.bin, uint16,
100 M tokens per shard).
The full SmolLM recipe is 75 / 15 / 10 fineweb-edu / cosmopedia-v2 / python-edu.
python-edu was dropped because the HuggingFaceTB/smollm-corpus subset
ships only blob_id… See the full description on the dataset page: https://huggingface.co/datasets/prometheus04/matilda-smollm-mix-15b-gpt2.fineweb-edu-100BT-gpt2-bin
fineweb-edu 100BT — GPT-2 pre-tokenized (.bin)
Pre-tokenized HuggingFaceFW/fineweb-edu :: sample/100BT for LLM pretraining without on-the-fly
tokenization or HF streaming (flat uint16 token ids, nanoGPT layout).
Tokenizer: gpt2 (tiktoken == HF AutoTokenizer('gpt2'), ids identical)
Format: uint16 little-endian, headerless
Layout: documents concatenated, eos=50256 appended after each doc
eos token id: 50256
train tokens: 100,146,465,071
val tokens: 20,000,000
Usage… See the full description on the dataset page: https://huggingface.co/datasets/AverageMetaheuristicsEnjoyer/fineweb-edu-100BT-gpt2-bin.gpt2gpt2-m-storybook
Your Storybook Dataset
Welcome to the repository for the "Your Storybook Dataset." This dataset contains a collection of stories and narratives suitable for various natural language processing tasks.
Under Collection
sozkz-corpus-balanced-kk-gpt2-v1
Kazakh Balanced GPT-2 Style
A balanced multi-domain Kazakh corpus, pre-tokenized in GPT-2 style (packed 1024-token blocks).
Overview
Property
Value
Format
Pre-tokenized, 1024-token blocks
Style
GPT-2
Status
Legacy — superseded by kazakh-clean-pretrain
License
Apache 2.0
Note
This is a legacy dataset. For new experiments, use kazakh-clean-pretrain instead.
Usage
from datasets import load_dataset
ds =… See the full description on the dataset page: https://huggingface.co/datasets/stukenov/sozkz-corpus-balanced-kk-gpt2-v1.gpt2-general-qa-4kThis is a general Dataset for basic GPT2 fine tuning, with instructions and answers.
anlp-tinystories-dpo-gpt2
TinyStoriesInstruct DPO preference pairs
Preference pairs for the Advanced NLP block course at TH Koeln, block
07 (DPO demo). Built by sampling two completions per held-out prompt
from the SFT'd model (Richard-Sieg-TH-Koln/anlp-sft-sanity-checkpoint) at temperature 1.0,
scoring each by how many of the prompt's required words (the
Words: field in TinyStoriesInstruct) it actually contains, and
keeping the higher-scoring completion as chosen and the other as
rejected. Ties are… See the full description on the dataset page: https://huggingface.co/datasets/Richard-Sieg-TH-Koln/anlp-tinystories-dpo-gpt2.