Datasetpublic

minhnguyent546/Fineweb-Edu-10BT

Dataset Summary This is the tokenized Fineweb-Edu (10BT subset) using SmolLM2-135M tokenzier. Data is divided into shards (.npy files) for easier to load with PyTorch IterableDataset. Each .npy file can be loaded with numpy.load('file_name.npy'). Split # Documents # Shards # Tokens train 9,575,380 101 10,004,991,326 (10.0B) val 96,721 1 101,807,253 (0.1B) Total 9,672,101 102 10,106,798,579 (10.1B) Example of usage uvx hf download… See the full description on the dataset page: https://huggingface.co/datasets/minhnguyent546/Fineweb-Edu-10BT.

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face

0likes731downloads

Dataset Card

Dataset Summary

This is the tokenized Fineweb-Edu (10BT subset) using SmolLM2-135M tokenzier. Data is divided into shards (.npy files) for easier to load with PyTorch IterableDataset.

Each .npy file can be loaded with numpy.load('file_name.npy').

Split	# Documents	# Shards	# Tokens
`train`	9,575,380	101	10,004,991,326 (10.0B)
`val`	96,721	1	101,807,253 (0.1B)
Total	9,672,101	102	10,106,798,579 (10.1B)

Example of usage

bash

uvx hf download minhnguyent546/Fineweb-Edu-10BT \
  --repo-type dataset \
  --local-dir Fineweb-Edu-10BT

python

import numpy as np

tokens = np.load("Fineweb-Edu-10BT/train/fineweb_edu_10bt_train_000000-of-000101.npy")

print(tokens.shape)
# (100000000,)

print(tokens.dtype)
# uint16

print(tokens[:20])
# [   0   29 9230   28  451  314  253 1123  655  288 1783 4534 5372 5047
#   281  260 2118   30 1206  654]