datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
allenai_olmo2_superbpe_10G_1M_tokenizer_tulu-3-sft-olmo-2-mixture-0225-bpe-to-superbpesamples processed: 866138
Total user BPE tokens: 252198220
Total assistant BPE tokens: 402038966
Total assistant SuperBPE tokens: 198274197
Assistant reduction vs BPE = 50.68284077717979%
bigcode-bpe-tokenizerbpe-tokenizeruniversal-bpe