sinatras/fineweb10B-gpt2-fluxentropy
FineWeb Dataset - GPT-2 Tokenized This dataset contains preprocessed and tokenized FineWeb data using the GPT-2 tokenizer. It consists of multiple training folders containing the processed data. Dataset structure: fineweb_train_000001 to fineweb_train_000005: Training folders
Upload fineweb_train_000005/bucket_6.bin with huggingface_hub
Upload fineweb_train_000005/bucket_4.bin with huggingface_hub
Upload fineweb_train_000005/bucket_2.bin with huggingface_hub
Upload fineweb_train_000005/bucket_boundaries.json with huggingface_hub
Upload fineweb_train_000005/bucket_5.bin with huggingface_hub
Upload fineweb_train_000005/bucket_3.bin with huggingface_hub
Upload fineweb_train_000005/bucket_1.bin with huggingface_hub
Upload fineweb_train_000004/bucket_6.bin with huggingface_hub
Upload fineweb_train_000004/bucket_4.bin with huggingface_hub
Upload fineweb_train_000004/bucket_2.bin with huggingface_hub
Upload fineweb_train_000004/bucket_boundaries.json with huggingface_hub
Upload fineweb_train_000004/bucket_5.bin with huggingface_hub
Upload fineweb_train_000004/bucket_3.bin with huggingface_hub
Upload fineweb_train_000004/bucket_1.bin with huggingface_hub
Upload fineweb_train_000003/bucket_6.bin with huggingface_hub
Upload fineweb_train_000003/bucket_4.bin with huggingface_hub
Upload fineweb_train_000003/bucket_2.bin with huggingface_hub
Upload fineweb_train_000003/bucket_boundaries.json with huggingface_hub
Upload fineweb_train_000003/bucket_5.bin with huggingface_hub
Upload fineweb_train_000003/bucket_3.bin with huggingface_hub
Upload fineweb_train_000003/bucket_1.bin with huggingface_hub
Upload fineweb_train_000002/bucket_6.bin with huggingface_hub
Upload fineweb_train_000002/bucket_4.bin with huggingface_hub
Upload fineweb_train_000002/bucket_2.bin with huggingface_hub
Upload fineweb_train_000002/bucket_boundaries.json with huggingface_hub
Upload fineweb_train_000002/bucket_5.bin with huggingface_hub
Upload fineweb_train_000002/bucket_3.bin with huggingface_hub
Upload fineweb_train_000002/bucket_1.bin with huggingface_hub
Upload fineweb_train_000001/bucket_6.bin with huggingface_hub
Upload fineweb_train_000001/bucket_4.bin with huggingface_hub
Upload fineweb_train_000001/bucket_2.bin with huggingface_hub
Upload fineweb_train_000001/bucket_boundaries.json with huggingface_hub
Upload fineweb_train_000001/bucket_5.bin with huggingface_hub
Upload fineweb_train_000001/bucket_3.bin with huggingface_hub
Upload fineweb_train_000001/bucket_1.bin with huggingface_hub
Upload README.md with huggingface_hub
initial commit