CoolFace
Datasetpublic

ZhuofengLi/pretraining-pretokenized-smollm3

SmolLM3 Pretokenized Pretraining Sources Datatrove/Nanotron tokenized-byte versions of three public pretraining sources: fineweb-edu-10bt: HuggingFaceFW/fineweb-edu, sample/10BT finemath-4plus: HuggingFaceTB/finemath, finemath-4plus stack-edu-python: HuggingFaceTB/stack-edu, Python, with content retrieved from the public Software Heritage S3 bucket using the upstream dataset-card procedure All subsets use HuggingFaceTB/SmolLM3-3B at revision… See the full description on the dataset page: https://huggingface.co/datasets/ZhuofengLi/pretraining-pretokenized-smollm3.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face
0likes149downloads
Dataset Card

SmolLM3 Pretokenized Pretraining Sources

Datatrove/Nanotron tokenized-byte versions of three public pretraining sources:

  • —fineweb-edu-10bt: HuggingFaceFW/fineweb-edu, sample/10BT
  • —finemath-4plus: HuggingFaceTB/finemath, finemath-4plus
  • —stack-edu-python: HuggingFaceTB/stack-edu, Python, with content retrieved from the public Software Heritage S3 bucket using the upstream dataset-card procedure

All subsets use HuggingFaceTB/SmolLM3-3B at revision a07cc9a04f16550a088caea529712d1d335b0ac1. Tokenization appends <|end_of_text|> (ID 128001) to each document, performs no other special-token insertion, and does not shuffle or truncate documents.

Format

Each subset contains a small metadata.parquet table and tokenized files under <subset>/files/:

  • —*.ds: little-endian uint32 token stream
  • —*.ds.index: little-endian uint64 cumulative document-end offsets
  • —*.ds.metadata: tokenizer, token width, and token count
  • —manifest.json: pinned inputs and aggregate validation metadata

The .ds folders can be used as Nanotron dataset_folder values with token_size_in_bytes: 4.

Licensing

The source datasets retain their own terms. FineWeb-Edu and FineMath are released under ODC-By. Stack-Edu points to code with per-file licensing and Software Heritage compliance requirements; consult its source metadata before redistribution or use.