datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
clt_gpt2_tokenized_control
Fresh multilingual GPT-2 CLT control data
Sequential, unshuffled control sample for CLT null experiments. For each language,
complete source documents were tokenized with CausalNLP/gpt2-hf_multilingual-20 at revision
0afbb31b2db3f394270d42d6a4cb7f8fceeca3d8. The first 100,000,000 tokenizer tokens were discarded
(including the complete document that crossed the threshold), after which complete
documents were retained until at least 100,000,000 tokens were collected.
Data are… See the full description on the dataset page: https://huggingface.co/datasets/abir-hr196/clt_gpt2_tokenized_control.clt_gpt2_tokenized_control
Fresh multilingual GPT-2 CLT control data
Sequential, unshuffled control sample for CLT null experiments. For each language,
complete source documents were tokenized with CausalNLP/gpt2-hf_multilingual-20 at revision
0afbb31b2db3f394270d42d6a4cb7f8fceeca3d8. The first 100,000,000 tokenizer tokens were discarded
(including the complete document that crossed the threshold), after which complete
documents were retained until at least 100,000,000 tokens were collected.
Data are… See the full description on the dataset page: https://huggingface.co/datasets/CausalNLP/clt_gpt2_tokenized_control.gpt2-training-ar-zh-ko-ja-4b
Balanced Arabic-Chinese-Korean-Japanese 4B-token training data
Sequential FineWeb-2 documents tokenized with CausalNLP/gpt2-tokenizer-ar-zh-ko-ja. Each language contains at least 1,000,000,000 tokens in complete documents. Total target: 4,000,000,000 tokens. Approximately 100,000,000 tokens per Parquet shard.
Splits: arb_Arab, cmn_Hani, kor_Hang, jpn_Jpan. Schema: text: string, input_ids: list<int32>.
smollm-corpus-instruct-2M-cosmopedia-v2-gpt2-v2-streaming
A corpus of high quality fine tuning data meant for fine tuning various HelixLM models
Dataset Composition:
A subset sampled from randomly selected shards from https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus cosmopedia-v2 split ...
Added: A preprocessed column formattedconversation concatenating the prompt, text and special tokens for instruct fine tuning.
Added: A token count column tokencount based on GPT2 tokenizer + the fine tuning special tokens.… See the full description on the dataset page: https://huggingface.co/datasets/david-thrower/smollm-corpus-instruct-2M-cosmopedia-v2-gpt2-v2-streaming.GPT2-Hacker-password-generator-dataset
Hacker Style Password Generation Dataset
Dataset Description
This dataset contains 20,000 instruction-response pairs designed to train and evaluate language models for generating strong, "hacker-style" passwords. The data simulates a user requesting a secure password and the model providing a complex, randomly generated string.
Supported Tasks
Text Generation: The primary task is conditional text generation, where the model takes a natural language instruction… See the full description on the dataset page: https://huggingface.co/datasets/CodeferSystem/GPT2-Hacker-password-generator-dataset.smoltalk-gpt2-sft
SmolTalk-GPT2-SFT
A fully tokenized version of the entire HuggingFaceTB/smoltalk
(all config) dataset — converted from raw multi-turn conversations into ready-to-train
(token_id, loss_mask) pairs using the GPT-2 BPE tokenizer (vocab=50257). Drop
it straight into a PyTorch trainer for supervised fine-tuning (SFT) of any
GPT-2-vocab language model — no template parsing, no role tagging, no chat-template
gymnastics required.
Overview
Source: HuggingFaceTB/smoltalk (all… See the full description on the dataset page: https://huggingface.co/datasets/juliannunezb/smoltalk-gpt2-sft.mixed-pretrain-10b-gpt2
Mixed Pretraining 10B (GPT-2 BPE)
A 10-billion-token pretraining dataset, GPT-2 BPE tokenized, assembled as a
diverse mix of web text, books, Wikipedia, code, academic papers, Q&A and
instruction-formatted conversations.
Built to train a ~500M parameter from-scratch GPT-2-style transformer (see
juliannunezb/transformer-lm-500m).
Mix
Source
Mix %
Tokens
Notes
fineweb
40.4%
4,039,999,700
reused from kjj0/fineweb10B-gpt2
fineweb_edu
15.2%
1,514,999,900
reused… See the full description on the dataset page: https://huggingface.co/datasets/juliannunezb/mixed-pretrain-10b-gpt2.ultrafeedback-gpt2-dpo
UltraFeedback-GPT2-DPO
A fully tokenized version of HuggingFaceH4/ultrafeedback_binarized
converted into ready-to-train (prompt_ids, chosen_response_ids, rejected_response_ids) triples
using the GPT-2 BPE tokenizer (vocab=50257). Drop it straight into a PyTorch DPO trainer
for Direct Preference Optimization of any GPT-2-vocab language model.
Overview
Source: HuggingFaceH4/ultrafeedback_binarized — 61K preference pairs binarized from UltraFeedback (Cui et al., 2023)… See the full description on the dataset page: https://huggingface.co/datasets/juliannunezb/ultrafeedback-gpt2-dpo.gpt2_to_gpt5.5_distilled_25k
GPT-2 to GPT-5.5 Advanced Reasoning Distillation (25k)
Dataset Description
25,000 unique, high-quality instruction-response pairs designed for knowledge distillation and supervised fine-tuning. The dataset elevates GPT-2 Medium toward GPT-5.5-level performance on complex reasoning tasks.
Core goal: Transfer frontier reasoning capabilities (multi-step CoT, cross-domain synthesis, edge-case analysis, novel insights) from a hypothetical GPT-5.5 teacher into smaller… See the full description on the dataset page: https://huggingface.co/datasets/WithinUsAI/gpt2_to_gpt5.5_distilled_25k.matilda-smollm-mix-15b-gpt2
matilda-smollm-mix-15B-gpt2
15 B GPT-2-BPE tokens drawn from a 5:1 token-balanced mix of
HuggingFaceTB/smollm-corpus:
Source
Share
Tokens
fineweb-edu-dedup
83.33 %
12.50 B
cosmopedia-v2
16.67 %
2.50 B
Total: 15,000,349,569 tokens across 151 shards (shard_*.bin, uint16,
100 M tokens per shard).
The full SmolLM recipe is 75 / 15 / 10 fineweb-edu / cosmopedia-v2 / python-edu.
python-edu was dropped because the HuggingFaceTB/smollm-corpus subset
ships only blob_id… See the full description on the dataset page: https://huggingface.co/datasets/prometheus04/matilda-smollm-mix-15b-gpt2.srp-gpt2-ptbr-corpus
SRP GPT-2 PT-BR Corpus
Corpus público em Parquet para treino autoregressivo de modelos de linguagem em português.
Fontes
Este dataset é uma composição de textos públicos/republicáveis, com atribuição às fontes originais:
Project Gutenberg, acessado via Gutendex API: https://www.gutenberg.org/ e https://gutendex.com/
FineWeb2 da Hugging Face, filtrado para português/pt-BR: https://huggingface.co/datasets/HuggingFaceFW/fineweb-2
Licenciamento e atribuição… See the full description on the dataset page: https://huggingface.co/datasets/celsowm/srp-gpt2-ptbr-corpus.gpt2-m-storybook
Your Storybook Dataset
Welcome to the repository for the "Your Storybook Dataset." This dataset contains a collection of stories and narratives suitable for various natural language processing tasks.
Under Collection
sozkz-corpus-balanced-kk-gpt2-v1
Kazakh Balanced GPT-2 Style
A balanced multi-domain Kazakh corpus, pre-tokenized in GPT-2 style (packed 1024-token blocks).
Overview
Property
Value
Format
Pre-tokenized, 1024-token blocks
Style
GPT-2
Status
Legacy — superseded by kazakh-clean-pretrain
License
Apache 2.0
Note
This is a legacy dataset. For new experiments, use kazakh-clean-pretrain instead.
Usage
from datasets import load_dataset
ds =… See the full description on the dataset page: https://huggingface.co/datasets/stukenov/sozkz-corpus-balanced-kk-gpt2-v1.gpt2-general-qa-4kThis is a general Dataset for basic GPT2 fine tuning, with instructions and answers.