marin-community/fineweb-edu-pretokenized-10B
Marin/Levanter Subsampled Pretokenized Dataset Dataset Train Urls: gs://marin-us-central2/raw/fineweb-edu-c2beb4/3c452cb/huggingface.co/datasets/HuggingFaceFW/fineweb-edu/resolve/3c452cb Factsheet Original cache: gs://marin-us-central2/tokenized/fineweb-edu-24698d Tokenizer: stanford-crfm/marin-tokenizer Seed 42 Number of tokens: 10,000,000,738 (This readme is automatically generated by Marin.)
Uploading small files from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
Commiting these files to the repo from gs://marin-us-central2/tokenized/subcache/fineweb-edu-10B-bfd467:
initial commit