CoolFace
Datasetpublic

yvonne90190/allenai_olmo2_superbpe_10G_1M_tokenizer_tulu-3-sft-olmo-2-mixture-0225-bpe-to-superbpe

samples processed: 866138 Total user BPE tokens: 252198220 Total assistant BPE tokens: 402038966 Total assistant SuperBPE tokens: 198274197 Assistant reduction vs BPE = 50.68284077717979%

sourceHugging Faceupdated 1y agoView on Hugging Face
0likes17downloads
discussions and pull requests

Conversations for this repository live on Hugging Face.

CoolFace shows imported repositories read-only. Posting into someone else’s repository from here would need an authorised integration and the account holder’s consent, so the link goes to the source instead.

Open discussions on Hugging Face
yvonne90190/allenai_olmo2_superbpe_10G_1M_tokenizer_tulu-3-sft-olmo-2-mixture-0225-bpe-to-superbpe · CoolFace