Datasetpublic

HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT-shuffled

FinePDFs 50BT + DCLM 30BT + FineWeb-Edu 20BT (Shuffled) A globally shuffled version of HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT. Part of the Smol-Data collection — tried and tested mixes for strong pretraining. Dataset Description This dataset contains the same ~100B token mixture (50B FinePDFs + 30B DCLM + 20B FineWeb-Edu) but with all documents globally shuffled (seed=42). Use this version when you need randomized document ordering for pretraining… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT-shuffled.

sourceHugging Faceodc-byupdated 7mo agoView on Hugging Face

6likes1.4kdownloads

Dataset Card

FinePDFs 50BT + DCLM 30BT + FineWeb-Edu 20BT (Shuffled)

A globally shuffled version of HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT.

Part of the Smol-Data collection — tried and tested mixes for strong pretraining.

Dataset Description

This dataset contains the same ~100B token mixture (50B FinePDFs + 30B DCLM + 20B FineWeb-Edu) but with all documents globally shuffled (seed=42). Use this version when you need randomized document ordering for pretraining — the unshuffled version has documents grouped by source.

How It Was Created

The unshuffled dataset was loaded into memory, shuffled with dataset.shuffle(seed=42), and re-uploaded with 100 shards. See the smol_data.py script for details.

Usage

python

from datasets import load_dataset

ds = load_dataset("HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT-shuffled", split="train", streaming=True)
for sample in ds:
    print(sample["text"][:200])
    break

Citation

bibtex

@misc{niklaus2026smoldata,
      title={SmolData},
      author={Joel Niklaus and Hynek Kydl{\'\i}{\v{c}}ek},
      year={2026},
      publisher={Hugging Face},
      journal={Hugging Face repository},
      howpublished={\url{https://huggingface.co/collections/HuggingFaceFW/smol-data}}
}