datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
gpt2-toksuite-detokenizedTraining data of the model detokenized in the exact order seen by the model.
The training data is partitioned into 8 chunks (chunk-0 through chunk-7), based on the GPU rank that generated the data. Each chunk contains detokenized text files in JSON Lines format (.jsonl).
mixed-pretrain-10b-gpt2
Mixed Pretraining 10B (GPT-2 BPE)
A 10-billion-token pretraining dataset, GPT-2 BPE tokenized, assembled as a
diverse mix of web text, books, Wikipedia, code, academic papers, Q&A and
instruction-formatted conversations.
Built to train a ~500M parameter from-scratch GPT-2-style transformer (see
juliannunezb/transformer-lm-500m).
Mix
Source
Mix %
Tokens
Notes
fineweb
40.4%
4,039,999,700
reused from kjj0/fineweb10B-gpt2
fineweb_edu
15.2%
1,514,999,900
reused… See the full description on the dataset page: https://huggingface.co/datasets/juliannunezb/mixed-pretrain-10b-gpt2.gpt2-details
Dataset Card for Evaluation run of gpt2
Dataset automatically created during the evaluation run of model gpt2
The dataset is composed of 85 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 52 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An additional configuration "results" store all the… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/gpt2-details.shaer-eval-raw-gpt2-small-arabic-poetry
Shaer Evaluation Results
Models: gpt2_small_arabic_poetry
Source dataset: Shaer-AI/shaer-sft-test-generations-k5
Rows: 3481
Validation passed: True
Scored rows included: True
Dataset repo: Shaer-AI/shaer-eval-raw-gpt2-small-arabic-poetry
Files
generations.jsonl: raw generation rows
generations.csv: raw generation rows in CSV
generations_scored.jsonl: raw rows plus meter/count evaluation
validation.json: validation summary
generations_scored.csv: scored rows in… See the full description on the dataset page: https://huggingface.co/datasets/Shaer-AI-2/shaer-eval-raw-gpt2-small-arabic-poetry.openai-community__gpt2-details
Dataset Card for Evaluation run of openai-community/gpt2
Dataset automatically created during the evaluation run of model openai-community/gpt2
The dataset is composed of 39 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 2 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An additional… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/openai-community__gpt2-details.openai-community__gpt2-medium-details
Dataset Card for Evaluation run of openai-community/gpt2-medium
Dataset automatically created during the evaluation run of model openai-community/gpt2-medium
The dataset is composed of 44 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/openai-community__gpt2-medium-details.openai-community__gpt2-large-details
Dataset Card for Evaluation run of openai-community/gpt2-large
Dataset automatically created during the evaluation run of model openai-community/gpt2-large
The dataset is composed of 44 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/openai-community__gpt2-large-details.openai-community__gpt2-xl-details
Dataset Card for Evaluation run of openai-community/gpt2-xl
Dataset automatically created during the evaluation run of model openai-community/gpt2-xl
The dataset is composed of 44 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/openai-community__gpt2-xl-details.shaer-eval-raw-gpt2-medium-arabic-poetry
Raw Shaer Continuation Generations - gpt2_medium_arabic_poetry
This dataset contains cumulative raw continuation generations for gpt2_medium_arabic_poetry.
The main table is data/test.jsonl; it includes source/prompt/reference fields plus the model output in generated_text and raw_generated_text.
Current uploaded progress target: 3481 rows out of 3481.
Scored upload: 1. When scored upload is true, reward/evaluation columns are included in the main table.
Artifacts such as… See the full description on the dataset page: https://huggingface.co/datasets/Shaer-AI-2/shaer-eval-raw-gpt2-medium-arabic-poetry.nuzzle-scan-openai-community-gpt2fineweb-gpt2bpe-20B
fineweb-gpt2bpe-20B
FineWeb-Edu (from karpathy/fineweb-edu-100b-shuffle), pre-tokenized with the gpt2bpe tokenizer (vocab 50,257)
and packaged as flat uint16 token-id .bin files for fast memmap training.
file
split
tokens
train.bin
train
20,000,000,000
val.bin
val
54,798,984
train and val are disjoint held-out partitions. Each .bin is a raw
little-endian uint16 stream (no header); token count = filesize / 2, and
train.meta.json / val.meta.jsoncarry the full… See the full description on the dataset page: https://huggingface.co/datasets/alexkstern/fineweb-gpt2bpe-20B.gpt-2-outputmatilda-smollm-mix-15b-gpt2
matilda-smollm-mix-15B-gpt2
15 B GPT-2-BPE tokens drawn from a 5:1 token-balanced mix of
HuggingFaceTB/smollm-corpus:
Source
Share
Tokens
fineweb-edu-dedup
83.33 %
12.50 B
cosmopedia-v2
16.67 %
2.50 B
Total: 15,000,349,569 tokens across 151 shards (shard_*.bin, uint16,
100 M tokens per shard).
The full SmolLM recipe is 75 / 15 / 10 fineweb-edu / cosmopedia-v2 / python-edu.
python-edu was dropped because the HuggingFaceTB/smollm-corpus subset
ships only blob_id… See the full description on the dataset page: https://huggingface.co/datasets/prometheus04/matilda-smollm-mix-15b-gpt2.owt-gpt2bpe-9B
owt-gpt2bpe-9B
OpenWebText (from apollo-research/Skylion007-openwebtext-tokenizer-gpt2), pre-tokenized with the gpt2bpe tokenizer (vocab 50,257)
and packaged as flat uint16 token-id .bin files for fast memmap training.
file
split
tokens
train.bin
train
9,015,870,208
val.bin
val
20,000,000
train and val are disjoint held-out partitions. Each .bin is a raw
little-endian uint16 stream (no header); token count = filesize / 2, and
train.meta.json/ val.meta.json carry… See the full description on the dataset page: https://huggingface.co/datasets/alexkstern/owt-gpt2bpe-9B.gpt2-large-generated
Synthetic Text Corpus - GPT-2 Large
Dataset Description
This dataset contains synthetically generated text sequences sampled from GPT-2 Large. It was created to provide a large-scale text corpus for research in natural language processing, particularly for studies on model behavior, text generation, and language modeling.
Dataset Summary
Size: ~100M tokens
Number of sequences: ~500,000
Source model: gpt2-large (774M parameters)
Sequence length: Maximum 256… See the full description on the dataset page: https://huggingface.co/datasets/vesteinn/gpt2-large-generated.DeepAutoAI__d2nwg_causal_gpt2_v1-details
Dataset Card for Evaluation run of DeepAutoAI/d2nwg_causal_gpt2_v1
Dataset automatically created during the evaluation run of model DeepAutoAI/d2nwg_causal_gpt2_v1
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 2 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/DeepAutoAI__d2nwg_causal_gpt2_v1-details.postbot__gpt2-medium-emailgen-details
Dataset Card for Evaluation run of postbot/gpt2-medium-emailgen
Dataset automatically created during the evaluation run of model postbot/gpt2-medium-emailgen
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/postbot__gpt2-medium-emailgen-details.Sharathhebbar24__chat_gpt2_dpo-details
Dataset Card for Evaluation run of Sharathhebbar24/chat_gpt2_dpo
Dataset automatically created during the evaluation run of model Sharathhebbar24/chat_gpt2_dpo
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/Sharathhebbar24__chat_gpt2_dpo-details.gpt2-vocabDeepAutoAI__causal_gpt2-details
Dataset Card for Evaluation run of DeepAutoAI/causal_gpt2
Dataset automatically created during the evaluation run of model DeepAutoAI/causal_gpt2
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An additional… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/DeepAutoAI__causal_gpt2-details.gpt20b_labeledDeepAutoAI__d2nwg_causal_gpt2-details
Dataset Card for Evaluation run of DeepAutoAI/d2nwg_causal_gpt2
Dataset automatically created during the evaluation run of model DeepAutoAI/d2nwg_causal_gpt2
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/DeepAutoAI__d2nwg_causal_gpt2-details.c4-gpt2bpe-10B
c4-gpt2bpe-10B
C4 (en) (from allenai/c4), pre-tokenized with the gpt2bpe tokenizer (vocab 50,257)
and packaged as flat uint16 token-id .bin files for fast memmap training.
file
split
tokens
train.bin
train
10,000,000,000
val.bin
val
174,033,902
train and val are disjoint held-out partitions. Each .bin is a raw
little-endian uint16 stream (no header); token count = filesize / 2, and
train.meta.json / val.meta.json carry the full metadata. The tokenizer/files (when… See the full description on the dataset page: https://huggingface.co/datasets/alexkstern/c4-gpt2bpe-10B.gpt2
Dataset Card for Evaluation run of gpt2
Dataset automatically created during the evaluation run of model gpt2
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 5 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An additional configuration "results" store all the… See the full description on the dataset page: https://huggingface.co/datasets/SaylorTwift/gpt2.