ShallowU/FineWeb-Edu-10B-Tokens-NPY
FineWeb-Edu 10B Tokens (NPY Format) 数据集概述 这是一个预处理好的教育文本数据集，包含约100亿个tokens，专门为训练小型语言模型（如GPT-2 124M）而设计。数据来源于高质量的FineWeb-Edu数据集，已经使用GPT-2的tiktoken分词器进行预处理，并保存为numpy格式以提高训练效率。 Followed by Let's reproduce GPT-2 (124M). Thanks to Andrej Karpathy!!! 🎯 适用场景 小型语言模型训练：特别适合GPT-2 124M/350M等参数规模的模型 教育研究：高质量教育内容，适合教学和学术研究 快速原型开发：预处理完成，可直接用于训练间 📊 数据统计 总token数量：~10,000,000,000 tokens 分片大小：100M tokens/分片 数据格式：numpy (.npy) uint16数组 分词器：GPT-2… See the full description on the dataset page: https://huggingface.co/datasets/ShallowU/FineWeb-Edu-10B-Tokens-NPY.
Create dataset_infos.json
Update README.md
add description
Upload edufineweb_train_000099.npy with huggingface_hub
Upload edufineweb_train_000098.npy with huggingface_hub
Upload edufineweb_train_000097.npy with huggingface_hub
Upload edufineweb_train_000096.npy with huggingface_hub
Upload edufineweb_train_000095.npy with huggingface_hub
Upload edufineweb_train_000094.npy with huggingface_hub
Upload edufineweb_train_000093.npy with huggingface_hub
Upload edufineweb_train_000092.npy with huggingface_hub
Upload edufineweb_train_000091.npy with huggingface_hub
Upload edufineweb_train_000090.npy with huggingface_hub
Upload edufineweb_train_000089.npy with huggingface_hub
Upload edufineweb_train_000088.npy with huggingface_hub
Upload edufineweb_train_000087.npy with huggingface_hub
Upload edufineweb_train_000086.npy with huggingface_hub
Upload edufineweb_train_000085.npy with huggingface_hub
Upload edufineweb_train_000084.npy with huggingface_hub
Upload edufineweb_train_000083.npy with huggingface_hub
Upload edufineweb_train_000082.npy with huggingface_hub
Upload edufineweb_train_000081.npy with huggingface_hub
Upload edufineweb_train_000080.npy with huggingface_hub
Upload edufineweb_train_000079.npy with huggingface_hub
Upload edufineweb_train_000078.npy with huggingface_hub
Upload edufineweb_train_000077.npy with huggingface_hub
Upload edufineweb_train_000076.npy with huggingface_hub
Upload edufineweb_train_000075.npy with huggingface_hub
Upload edufineweb_train_000074.npy with huggingface_hub
Upload edufineweb_train_000073.npy with huggingface_hub
Upload edufineweb_train_000072.npy with huggingface_hub
Upload edufineweb_train_000071.npy with huggingface_hub
Upload edufineweb_train_000070.npy with huggingface_hub
Upload edufineweb_train_000069.npy with huggingface_hub
Upload edufineweb_train_000068.npy with huggingface_hub
Upload edufineweb_train_000067.npy with huggingface_hub
Upload edufineweb_train_000066.npy with huggingface_hub
Upload edufineweb_train_000065.npy with huggingface_hub
Upload edufineweb_train_000064.npy with huggingface_hub
Upload edufineweb_train_000063.npy with huggingface_hub
Upload edufineweb_train_000062.npy with huggingface_hub
Upload edufineweb_train_000061.npy with huggingface_hub
Upload edufineweb_train_000060.npy with huggingface_hub
Upload edufineweb_train_000059.npy with huggingface_hub
Upload edufineweb_train_000058.npy with huggingface_hub
Upload edufineweb_train_000057.npy with huggingface_hub
Upload edufineweb_train_000056.npy with huggingface_hub
Upload edufineweb_train_000055.npy with huggingface_hub
Upload edufineweb_train_000054.npy with huggingface_hub
Upload edufineweb_train_000053.npy with huggingface_hub