yvonne90190/allenai_olmo2_superbpe_10G_1M_tokenizer_tulu-3-sft-olmo-2-mixture-0225-bpe-to-superbpe
samples processed: 866138 Total user BPE tokens: 252198220 Total assistant BPE tokens: 402038966 Total assistant SuperBPE tokens: 198274197 Assistant reduction vs BPE = 50.68284077717979%
017
samples processed: 866138<br> Total user BPE tokens: 252198220<br> Total assistant BPE tokens: 402038966<br> Total assistant SuperBPE tokens: 198274197<br> Assistant reduction vs BPE = 50.68284077717979%<br>