anisoleai/fineweb-tokenized
FineWeb Tokenized > 4 trillion tokens of the pre-tokenized data the 🌐 web has to offer What is it? This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards. By distributing the pre-tokenized corpus, we eliminate… See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.
Update README.md
Upload data_9/metadata.json with huggingface_hub
Upload data_9/shard-00035.parquet with huggingface_hub
Upload data_9/metadata.json with huggingface_hub
Upload data_9/shard-00034.parquet with huggingface_hub
Upload data_9/metadata.json with huggingface_hub
Upload data_9/shard-00033.parquet with huggingface_hub
Upload data_33/metadata.json with huggingface_hub
Upload data_37/metadata.json with huggingface_hub
Upload data_45/metadata.json with huggingface_hub
Upload data_37/shard-00049.parquet with huggingface_hub
Upload data_45/shard-00031.parquet with huggingface_hub
Upload data_26/metadata.json with huggingface_hub
Upload data_26/shard-00051.parquet with huggingface_hub
Upload data_36/metadata.json with huggingface_hub
Upload data_36/shard-00067.parquet with huggingface_hub
Upload data_20/metadata.json with huggingface_hub
Upload data_20/shard-00056.parquet with huggingface_hub
Upload data_40/metadata.json with huggingface_hub
Upload data_40/shard-00078.parquet with huggingface_hub
Upload data_24/metadata.json with huggingface_hub
Upload data_24/shard-00041.parquet with huggingface_hub
Upload data_38/metadata.json with huggingface_hub
Upload data_38/shard-00073.parquet with huggingface_hub
Upload data_28/metadata.json with huggingface_hub
Upload data_28/shard-00031.parquet with huggingface_hub
Upload data_7/metadata.json with huggingface_hub
Upload data_7/shard-00077.parquet with huggingface_hub
Upload data_15/metadata.json with huggingface_hub
Upload data_15/shard-00065.parquet with huggingface_hub
Upload data_22/metadata.json with huggingface_hub
Upload data_22/shard-00044.parquet with huggingface_hub
Upload data_17/metadata.json with huggingface_hub
Upload data_17/shard-00051.parquet with huggingface_hub
Upload data_2/metadata.json with huggingface_hub
Upload data_2/shard-00069.parquet with huggingface_hub
Upload data_39/metadata.json with huggingface_hub
Upload data_39/shard-00041.parquet with huggingface_hub
Upload data_33/metadata.json with huggingface_hub
Upload data_33/shard-00071.parquet with huggingface_hub
Upload data_45/metadata.json with huggingface_hub
Upload data_45/shard-00030.parquet with huggingface_hub
Upload data_26/metadata.json with huggingface_hub
Upload data_26/shard-00050.parquet with huggingface_hub
Upload data_21/metadata.json with huggingface_hub
Upload data_37/metadata.json with huggingface_hub
Upload data_21/shard-00033.parquet with huggingface_hub
Upload data_37/shard-00048.parquet with huggingface_hub
Upload data_18/metadata.json with huggingface_hub
Upload data_18/shard-00049.parquet with huggingface_hub