datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
pretrain_v1_20b
Chess Pre-to-Post — Pretraining Corpus v1 (20B)
Raw tokenized pretraining data for the Chess Pre-to-Post project, stored as
sharded NumPy arrays (shard_XXXX/raw.NNNN.npy).
[!IMPORTANT]
This is an earlier, smaller (20B) snapshot and is no longer maintained.
The maintained version of this dataset is
pavelslab-nyu/pretrain_v1_54B.
Please use that version for any new work — it supersedes this one.
Maintained version
➡️ pavelslab-nyu/pretrain_v1_54B… See the full description on the dataset page: https://huggingface.co/datasets/chess-pre-to-post/pretrain_v1_20b.gpt-oss-20b-moe-expert-power-traces-320k
GPT-OSS-20B MoE Expert Power Traces (320k, ChipWhisperer)
This dataset contains analog power traces captured with a ChipWhisperer Husky while running forced single-expert MoE computations derived from openai/gpt-oss-20b on an NVIDIA H100.
What is recorded
Each trace corresponds to one capture trial where:
A fixed expert id is selected (expert_00 ... expert_31).
A random hidden-state tensor is generated once per trial.
The selected expert computation is executed… See the full description on the dataset page: https://huggingface.co/datasets/masterpieceexternal/gpt-oss-20b-moe-expert-power-traces-320k.finepdfs_50BT-dclm_30BT-fineweb_edu_20BT
FinePDFs 50BT + DCLM 30BT + FineWeb-Edu 20BT
A ~100 billion token pretraining mixture combining three high-quality English data sources in a 50-30-20 ratio.
Part of the Smol-Data collection — tried and tested mixes for strong pretraining. Inspired by optimal dataset mixing.
Dataset Description
Component
Source
Tokens
FinePDFs 100BT
FinePDFs
~50B
DCLM 100BT
DCLM-Baseline 1.0
~30B
FineWeb-Edu 100BT
FineWeb-Edu
~20B
The schema is reduced to the… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT.finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT
FinePDFs-Edu 50BT + DCLM 30BT + FineWeb-Edu 20BT
A ~100 billion token pretraining mixture combining three high-quality English data sources in a 50-30-20 ratio, using the educational subset of FinePDFs.
Part of the Smol-Data collection — tried and tested mixes for strong pretraining. Inspired by optimal dataset mixing.
Dataset Description
Component
Source
Tokens
FinePDFs-Edu 100BT
FinePDFs-Edu
~50B
DCLM 100BT
DCLM-Baseline 1.0
~30B
FineWeb-Edu 100BT… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT.Nemotron-CC-HQ-20B
Nemotron-CC-HQ-20B
This Dataset consists of approximately 20B tokens of Nemotron-CC-HQ, consisting of randomly sampled slices from crawls in the range CC-MAIN-2013-20-part-00012 to CC-MAIN-2019-04-part-00007.
For more information about Nemotron-CC check the Paper by Nvidia
Disclaimer:
Derived from Nemotron-CC (Common Crawl). No ownership of underlying content is claimed.
Data may be subject to third-party rights. Use at your own risk and in compliance with… See the full description on the dataset page: https://huggingface.co/datasets/Fredithefish/Nemotron-CC-HQ-20B.code-20b
Dataset Card for "code_20b2"
More Information needed
code_20b
Dataset Card for "code_20b"
More Information needed
finewebedu-20B
FineWebEDU 20B
A copy of FineWebEDU-20B used for out tokenizer experiments. The subsets are as follows:
bytelevel: the full dataset tokenized using our bytelevel tokenizer
bytelevel-subset_1: a 100k-row subset of the bytelevel subset, used to train bytelevel models.
bytelevel-subset_2: a 100k-row subset of the bytelevel subset, used to extract llm predictions.
bytelevel-llm-data: a copy of bytelevel-subset_2 with lm predictions, used to train bytespan tokenizers… See the full description on the dataset page: https://huggingface.co/datasets/ByteSpanTokenisers/finewebedu-20B.monology-pile-uncopyrighted-tokenizer-EleutherAI-gpt-neox-20bFW_tokenized_20B
Reproducing GPT-2 for Indic Languages
This Dataset is part of my project on "Reproducing GPT-2 for Indic Languages". Check out the main repository: https://github.com/Shaligram-Dewangan/GPT-2-for-Indic-Languages
Dataset Description
This is a tokenized text data for pre-training small sized language models and learning purposes.
This dataset contains approximately 20 billion English + Hindi language tokens. It is tokenized using a custom BPE Tokeinzer. The English dataset… See the full description on the dataset page: https://huggingface.co/datasets/Shaligram-Dewangan/FW_tokenized_20B.finepdfs_50BT-dclm_30BT-fineweb_edu_20BT-shuffled
FinePDFs 50BT + DCLM 30BT + FineWeb-Edu 20BT (Shuffled)
A globally shuffled version of HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT.
Part of the Smol-Data collection — tried and tested mixes for strong pretraining.
Dataset Description
This dataset contains the same ~100B token mixture (50B FinePDFs + 30B DCLM + 20B FineWeb-Edu) but with all documents globally shuffled (seed=42). Use this version when you need randomized document ordering for pretraining —… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT-shuffled.finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled
FinePDFs-Edu 50BT + DCLM 30BT + FineWeb-Edu 20BT (Shuffled)
A globally shuffled version of HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT.
Part of the Smol-Data collection — tried and tested mixes for strong pretraining.
Dataset Description
This dataset contains the same ~100B token mixture (50B FinePDFs-Edu + 30B DCLM + 20B FineWeb-Edu) but with all documents globally shuffled (seed=42). Use this version when you need randomized document ordering for… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled.dolma_20bn_wiki_upsamplefineweb-nanochatbpe-20B
fineweb-nanochatbpe-20B
FineWeb-Edu (from karpathy/fineweb-edu-100b-shuffle), pre-tokenized with the nanochatbpe tokenizer (vocab 65,536)
and packaged as flat uint16 token-id .bin files for fast memmap training.
file
split
tokens
train.bin
train
20,000,000,000
val.bin
val
52,336,096
train and val are disjoint held-out partitions. Each .bin is a raw
little-endian uint16 stream (no header); token count = filesize / 2, and
train.meta.json / val.meta.jsoncarry the full… See the full description on the dataset page: https://huggingface.co/datasets/alexkstern/fineweb-nanochatbpe-20B.SimpleDeco-gptoss20bdataset_mix_1.7B_20Btokensgpt-oss-20b-rollouts
GPT-OSS-20B Rollouts
Generated rollouts from GPT-OSS-20B with parsed Harmony channels (assistant thinking/final).
Schema: user_content, system_reasoning_effort, assistant_thinking, assistant_content.
Loading example: load_dataset("andyrdt/gpt-oss-20b-rollouts", "HarmBench", split="standard_train").
Notes
This repository uses manual configuration to expose both subset (config) and split dropdowns in the viewer.
Safety and jailbreak
HarmBench: Safety prompts… See the full description on the dataset page: https://huggingface.co/datasets/andyrdt/gpt-oss-20b-rollouts.dolma_20bn_instruct_upsamplegpt-oss-20b-combined-outputsFuseChat-Mixture-InternLM2-Chat-20B-Representation
Dataset Card for FuseChat-Mixture
Dataset Description
FuseChat-Mixture is the training dataset used in 📑FuseChat: Knowledge Fusion of Chat Models
FuseChat-Mixture is a comprehensive training dataset covers different styles and capabilities, featuring both human-written and model-generated, and spanning general instruction-following and specific skills. These sources include:
Orca-Best: We sampled 20,000 examples from Orca-Best, which is filtered from the original GPT-4… See the full description on the dataset page: https://huggingface.co/datasets/FuseAI/FuseChat-Mixture-InternLM2-Chat-20B-Representation.gpt_oss_20b_doorkey_boundary_activations20BN_jester_V1_videosdetails_EleutherAI__gpt-neox-20b
Dataset Card for Evaluation run of EleutherAI/gpt-neox-20b
Dataset Summary
Dataset automatically created during the evaluation run of model EleutherAI/gpt-neox-20b on the Open LLM Leaderboard.
The dataset is composed of 122 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 4 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_EleutherAI__gpt-neox-20b.Skylion007-openwebtext-tokenizer-EleutherAI-gpt-neox-20bthe-stack-v2-20B-samplefineweb-edu-20B-E1-Scramble-Hier-flatteneddolma_20bn_cc_high_qualitydolma_20bn_prop_stratified_sampleSmolLM2-1.7B-stage-4-20BFineWeb2-HQ-ja-20B元のデータセットFineWeb2-HQ
元のデータセットは多言語で巨大なため、扱いやすい用に日本語データを約200GBだけ抽出したデータセットです
wc 結果
1763269 38541549 5370473709 fineweb_jpn_Jpan_chunk_0.jsonl
1784158 37430170 5370514369 fineweb_jpn_Jpan_chunk_1.jsonl
1639554 40065129 5370372344 fineweb_jpn_Jpan_chunk_10.jsonl
1575127 42167166 5370298354 fineweb_jpn_Jpan_chunk_11.jsonl
1686375 39225898 5370402506 fineweb_jpn_Jpan_chunk_12.jsonl
1786948 36456352 5370498572… See the full description on the dataset page: https://huggingface.co/datasets/dahara1/FineWeb2-HQ-ja-20B.