Datasetpublic

kjj0/fineweb10B-gpt2

fineweb10B-gpt2 This repo contains the GPT-2 tokens for fineweb10B, just as would be generated by https://github.com/KellerJordan/modded-nanogpt/tree/master (or llm.c). You can download from this repo instead of re-tokenizing to save a couple hours of setup on a new machine.

sourceHugging Facemitupdated 2y agoView on Hugging Face

11likes12kdownloads

Dataset Card

fineweb10B-gpt2

This repo contains the GPT-2 tokens for fineweb10B, just as would be generated by https://github.com/KellerJordan/modded-nanogpt/tree/master (or llm.c).

You can download from this repo instead of re-tokenizing to save a couple hours of setup on a new machine.