Datasetpublic

ShallowU/FineWeb-Edu-10B-Tokens-NPY

FineWeb-Edu 10B Tokens (NPY Format) 数据集概述这是一个预处理好的教育文本数据集，包含约100亿个tokens，专门为训练小型语言模型（如GPT-2 124M）而设计。数据来源于高质量的FineWeb-Edu数据集，已经使用GPT-2的tiktoken分词器进行预处理，并保存为numpy格式以提高训练效率。 Followed by Let's reproduce GPT-2 (124M). Thanks to Andrej Karpathy!!! 🎯 适用场景小型语言模型训练：特别适合GPT-2 124M/350M等参数规模的模型教育研究：高质量教育内容，适合教学和学术研究快速原型开发：预处理完成，可直接用于训练间 📊 数据统计总token数量：~10,000,000,000 tokens 分片大小：100M tokens/分片数据格式：numpy (.npy) uint16数组分词器：GPT-2… See the full description on the dataset page: https://huggingface.co/datasets/ShallowU/FineWeb-Edu-10B-Tokens-NPY.

sourceHugging Facemitupdated 1y agoView on Hugging Face

3likes2.3kdownloads

root

file.gitattributes2 KBdownload

filedataset_infos.json1 KBdownload

fileedufineweb_train_000001.npy190.7 MBdownload

fileedufineweb_train_000002.npy190.7 MBdownload

fileedufineweb_train_000003.npy190.7 MBdownload

fileedufineweb_train_000004.npy190.7 MBdownload

fileedufineweb_train_000005.npy190.7 MBdownload

fileedufineweb_train_000006.npy190.7 MBdownload

fileedufineweb_train_000007.npy190.7 MBdownload

fileedufineweb_train_000008.npy190.7 MBdownload

fileedufineweb_train_000009.npy190.7 MBdownload

fileedufineweb_train_000010.npy190.7 MBdownload

fileedufineweb_train_000011.npy190.7 MBdownload

fileedufineweb_train_000012.npy190.7 MBdownload

fileedufineweb_train_000013.npy190.7 MBdownload

fileedufineweb_train_000014.npy190.7 MBdownload

fileedufineweb_train_000015.npy190.7 MBdownload

fileedufineweb_train_000016.npy190.7 MBdownload

fileedufineweb_train_000017.npy190.7 MBdownload

fileedufineweb_train_000018.npy190.7 MBdownload

fileedufineweb_train_000019.npy190.7 MBdownload

fileedufineweb_train_000020.npy190.7 MBdownload

fileedufineweb_train_000021.npy190.7 MBdownload

fileedufineweb_train_000022.npy190.7 MBdownload

fileedufineweb_train_000023.npy190.7 MBdownload

fileedufineweb_train_000024.npy190.7 MBdownload

fileedufineweb_train_000025.npy190.7 MBdownload

fileedufineweb_train_000026.npy190.7 MBdownload

fileedufineweb_train_000027.npy190.7 MBdownload

fileedufineweb_train_000028.npy190.7 MBdownload

fileedufineweb_train_000029.npy190.7 MBdownload

fileedufineweb_train_000030.npy190.7 MBdownload

fileedufineweb_train_000031.npy190.7 MBdownload

fileedufineweb_train_000032.npy190.7 MBdownload

fileedufineweb_train_000033.npy190.7 MBdownload

fileedufineweb_train_000034.npy190.7 MBdownload

fileedufineweb_train_000035.npy190.7 MBdownload

fileedufineweb_train_000036.npy190.7 MBdownload

fileedufineweb_train_000037.npy190.7 MBdownload

fileedufineweb_train_000038.npy190.7 MBdownload

fileedufineweb_train_000039.npy190.7 MBdownload

fileedufineweb_train_000040.npy190.7 MBdownload

fileedufineweb_train_000041.npy190.7 MBdownload

fileedufineweb_train_000042.npy190.7 MBdownload

fileedufineweb_train_000043.npy190.7 MBdownload

fileedufineweb_train_000044.npy190.7 MBdownload

fileedufineweb_train_000045.npy190.7 MBdownload

fileedufineweb_train_000046.npy190.7 MBdownload

fileedufineweb_train_000047.npy190.7 MBdownload

fileedufineweb_train_000048.npy190.7 MBdownload

fileedufineweb_train_000049.npy190.7 MBdownload

fileedufineweb_train_000050.npy190.7 MBdownload

fileedufineweb_train_000051.npy190.7 MBdownload

fileedufineweb_train_000052.npy190.7 MBdownload

fileedufineweb_train_000053.npy190.7 MBdownload

fileedufineweb_train_000054.npy190.7 MBdownload

fileedufineweb_train_000055.npy190.7 MBdownload

fileedufineweb_train_000056.npy190.7 MBdownload

fileedufineweb_train_000057.npy190.7 MBdownload

fileedufineweb_train_000058.npy190.7 MBdownload

fileedufineweb_train_000059.npy190.7 MBdownload

fileedufineweb_train_000060.npy190.7 MBdownload

fileedufineweb_train_000061.npy190.7 MBdownload

fileedufineweb_train_000062.npy190.7 MBdownload

fileedufineweb_train_000063.npy190.7 MBdownload

fileedufineweb_train_000064.npy190.7 MBdownload

fileedufineweb_train_000065.npy190.7 MBdownload

fileedufineweb_train_000066.npy190.7 MBdownload

fileedufineweb_train_000067.npy190.7 MBdownload

fileedufineweb_train_000068.npy190.7 MBdownload

fileedufineweb_train_000069.npy190.7 MBdownload

fileedufineweb_train_000070.npy190.7 MBdownload

fileedufineweb_train_000071.npy190.7 MBdownload

fileedufineweb_train_000072.npy190.7 MBdownload

fileedufineweb_train_000073.npy190.7 MBdownload

fileedufineweb_train_000074.npy190.7 MBdownload

fileedufineweb_train_000075.npy190.7 MBdownload

fileedufineweb_train_000076.npy190.7 MBdownload

fileedufineweb_train_000077.npy190.7 MBdownload

fileedufineweb_train_000078.npy190.7 MBdownload

fileedufineweb_train_000079.npy190.7 MBdownload

fileedufineweb_train_000080.npy190.7 MBdownload

fileedufineweb_train_000081.npy190.7 MBdownload

fileedufineweb_train_000082.npy190.7 MBdownload

fileedufineweb_train_000083.npy190.7 MBdownload

fileedufineweb_train_000084.npy190.7 MBdownload

fileedufineweb_train_000085.npy190.7 MBdownload

fileedufineweb_train_000086.npy190.7 MBdownload

fileedufineweb_train_000087.npy190.7 MBdownload

fileedufineweb_train_000088.npy190.7 MBdownload

fileedufineweb_train_000089.npy190.7 MBdownload

fileedufineweb_train_000090.npy190.7 MBdownload

fileedufineweb_train_000091.npy190.7 MBdownload

fileedufineweb_train_000092.npy190.7 MBdownload

fileedufineweb_train_000093.npy190.7 MBdownload

fileedufineweb_train_000094.npy190.7 MBdownload

fileedufineweb_train_000095.npy190.7 MBdownload

fileedufineweb_train_000096.npy190.7 MBdownload

fileedufineweb_train_000097.npy190.7 MBdownload

fileedufineweb_train_000098.npy190.7 MBdownload

fileedufineweb_train_000099.npy103.0 MBdownload

fileedufineweb_val_000000.npy190.7 MBdownload

fileREADME.md5 KBdownload

ShallowU/FineWeb-Edu-10B-Tokens-NPY · main · files are served by the source, never re-hosted here