Datasetpublic

yvonne90190/allenai_olmo2_superbpe_10G_1M_tokenizer_tulu-3-sft-olmo-2-mixture-0225-bpe-to-superbpe

samples processed: 866138 Total user BPE tokens: 252198220 Total assistant BPE tokens: 402038966 Total assistant SuperBPE tokens: 198274197 Assistant reduction vs BPE = 50.68284077717979%

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes17downloads

2 commits on main

62b24af1y ago

Upload folder using huggingface_hub

yvonne90190

8bc4b401y ago

initial commit

yvonne90190