Datasetpublic

YangyiH/DepthBench-FineWeb-Edu-100BT-tokenized

DepthBench FineWeb-Edu 100BT Tokenized This repository contains the tokenized FineWeb-Edu 100BT sample used by DepthBench pretraining experiments. Splits train/: 139 shards, 99,585,913,529 tokens, and 97,045,608 documents. eval/: 013_00008, containing 234,993,701 tokens and 225,078 documents. All remaining source shards are assigned to training. Each source document is terminated by an EOS token before documents are concatenated. Format Each shard… See the full description on the dataset page: https://huggingface.co/datasets/YangyiH/DepthBench-FineWeb-Edu-100BT-tokenized.

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes225downloads

Dataset Card

DepthBench FineWeb-Edu 100BT Tokenized

This repository contains the tokenized FineWeb-Edu 100BT sample used by DepthBench pretraining experiments.

Splits

—train/: 139 shards, 99,585,913,529 tokens, and 97,045,608 documents.
—eval/: 013_00008, containing 234,993,701 tokens and 225,078 documents.

All remaining source shards are assigned to training. Each source document is terminated by an EOS token before documents are concatenated.

Format

Each shard contains:

—*.npy: a flat uint16 array of concatenated token IDs.
—*.csv.gz: document-boundary metadata.
—*.meta.json: token counts, document counts, split, and source-shard metadata.

The tokenizer is included as tokenizer/allenai_gpt-neox-olmo-dolma-v1_5.json. It has a vocabulary size of 50,280 and uses token ID 50,279 as EOS.

python

import numpy as np

tokens = np.load("train/000_00000.npy", mmap_mode="r")
print(tokens.dtype, tokens.shape)

Source

The source text is the 100BT sample of HuggingFaceFW/fineweb-edu. Users are responsible for complying with the source dataset's terms and licenses.