Datasetpublic

sinatras/fineweb10B-gpt2-fluxentropy

FineWeb Dataset - GPT-2 Tokenized This dataset contains preprocessed and tokenized FineWeb data using the GPT-2 tokenizer. It consists of multiple training folders containing the processed data. Dataset structure: fineweb_train_000001 to fineweb_train_000005: Training folders

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes23downloads

Dataset Card

FineWeb Dataset - GPT-2 Tokenized

This dataset contains preprocessed and tokenized FineWeb data using the GPT-2 tokenizer. It consists of multiple training folders containing the processed data.

Dataset structure:

—finewebtrain000001 to finewebtrain000005: Training folders