Datasetpublic

ZhuofengLi/fineweb-edu-pretokenized-llama3-100b

FineWeb-Edu Pretokenized with Llama 3.1 (100B) This repository contains the sample/100BT subset of FineWeb-Edu, pretokenized for Megatron-LM-style training with meta-llama/Meta-Llama-3.1-8B. It is a derived, pretokenized version of the upstream data, not an official Hugging Face FineWeb release. Dataset summary 140 indexed shards 97,270,686 non-empty documents 97,458,793,013 tokens English web text from FineWeb-Edu sample/100BT Source dataset revision:… See the full description on the dataset page: https://huggingface.co/datasets/ZhuofengLi/fineweb-edu-pretokenized-llama3-100b.

sourceHugging Faceodc-byupdated 2mo agoView on Hugging Face

0likes255downloads

Dataset Card

FineWeb-Edu Pretokenized with Llama 3.1 (100B)

This repository contains the sample/100BT subset of FineWeb-Edu, pretokenized for Megatron-LM-style training with `meta-llama/Meta-Llama-3.1-8B`.

It is a derived, pretokenized version of the upstream data, not an official Hugging Face FineWeb release.

Dataset summary

—140 indexed shards
—97,270,686 non-empty documents
—97,458,793,013 tokens
—English web text from FineWeb-Edu sample/100BT
—Source dataset revision: 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9

Tokenization

Text was tokenized using the Llama 3.1 tokenizer with add_special_tokens=False. For each non-empty document, one eos_token_id was appended. No padding or truncation was applied.

The files are in Megatron Core indexed-dataset format. Each shard consists of matching files:

text

fineweb_edu_100bt_000_text_document.bin
fineweb_edu_100bt_000_text_document.idx

Use the path without .bin or .idx as the dataset prefix. prefixes.txt lists all prefixes in shard-index order, and manifest.jsonl records each shard's upstream source, document count, and token count.

Download

Download only the needed shard pairs, or snapshot the entire repository:

bash

hf download ZhuofengLi/fineweb-edu-pretokenized-llama3-100b \
  --repo-type dataset \
  --include 'fineweb_edu_100bt_000_text_document.*'

The data is binary indexed data rather than a datasets/Parquet configuration, so it should be read with Megatron Core's indexed-dataset reader (or another reader compatible with its .bin/.idx format).

License and attribution

The upstream FineWeb-Edu dataset is released under ODC-BY. This derivative retains that license and should be attributed to FineWeb-Edu / Hugging Face as well as this preprocessing repository. Please consult the upstream dataset card for its documentation, provenance, and usage considerations.