CoolFace
Datasetpublic

catherinearnett/montok

MonTok: A Suite of Monolingual Tokenizers This is a set of monolingual tokenizers for 98 languages. For each language, there are Unigram, BPE, and SuperBPE tokenizers, ranging in vocabulary size from around 6k to over 200k. Training Details Training Data All tokenizers are trained on samples of the data used to the train the Goldfish language models. The tokenizers were either trained on scaled or unscaled data. This refers to whether the models… See the full description on the dataset page: https://huggingface.co/datasets/catherinearnett/montok.

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face
4likes18kdownloads
50 commits on main
33d1f6d1y ago

Update README.md

catherinearnett
8c80b851y ago

Update README.md

catherinearnett
6d885cd1y ago

Upload folder using huggingface_hub

catherinearnett
89b4d701y ago

Upload folder using huggingface_hub

catherinearnett
9efa8a81y ago

Upload folder using huggingface_hub

catherinearnett
07c13971y ago

Upload folder using huggingface_hub

catherinearnett
c693f451y ago

Upload folder using huggingface_hub

catherinearnett
14a924f1y ago

Upload folder using huggingface_hub

catherinearnett
e54e2071y ago

Upload folder using huggingface_hub

catherinearnett
78b1cd91y ago

Upload folder using huggingface_hub

catherinearnett
43be9591y ago

Upload folder using huggingface_hub

catherinearnett
7b20e4a1y ago

Upload folder using huggingface_hub

catherinearnett
c85cb011y ago

Upload folder using huggingface_hub

catherinearnett
05dea581y ago

Upload folder using huggingface_hub

catherinearnett
0228d911y ago

Upload folder using huggingface_hub

catherinearnett
bef27611y ago

Upload folder using huggingface_hub

catherinearnett
a2af5e61y ago

Upload folder using huggingface_hub

catherinearnett
9603ac61y ago

Upload folder using huggingface_hub

catherinearnett
9b42fb41y ago

Upload folder using huggingface_hub

catherinearnett
7c371011y ago

Upload folder using huggingface_hub

catherinearnett
506d9a51y ago

Upload folder using huggingface_hub

catherinearnett
30a41971y ago

Upload folder using huggingface_hub

catherinearnett
757fc171y ago

Upload folder using huggingface_hub

catherinearnett
519eb341y ago

Upload folder using huggingface_hub

catherinearnett
e7a16a91y ago

Upload folder using huggingface_hub

catherinearnett
1d31e021y ago

Upload folder using huggingface_hub

catherinearnett
4bb50591y ago

Upload folder using huggingface_hub

catherinearnett
b9c4c851y ago

Upload folder using huggingface_hub

catherinearnett
6171b751y ago

Upload folder using huggingface_hub

catherinearnett
a8cee171y ago

Upload folder using huggingface_hub

catherinearnett
785135b1y ago

Upload folder using huggingface_hub

catherinearnett
a37628a1y ago

Upload folder using huggingface_hub

catherinearnett
c472c991y ago

Upload folder using huggingface_hub

catherinearnett
f8c11531y ago

Upload folder using huggingface_hub

catherinearnett
a272d161y ago

Upload folder using huggingface_hub

catherinearnett
54cce981y ago

Upload folder using huggingface_hub

catherinearnett
ce00b7f1y ago

Upload folder using huggingface_hub

catherinearnett
859934e1y ago

Upload folder using huggingface_hub

catherinearnett
ad2f6ed1y ago

Upload folder using huggingface_hub

catherinearnett
ea3212e1y ago

Upload folder using huggingface_hub

catherinearnett
ec80bad1y ago

Upload folder using huggingface_hub

catherinearnett
a7681a71y ago

Upload folder using huggingface_hub

catherinearnett
726e6e51y ago

Upload folder using huggingface_hub

catherinearnett
805c7eb1y ago

Upload folder using huggingface_hub

catherinearnett
329be501y ago

Upload folder using huggingface_hub

catherinearnett
f7651ed1y ago

Upload folder using huggingface_hub

catherinearnett
7e50adb1y ago

Upload folder using huggingface_hub

catherinearnett
255bccb1y ago

Upload folder using huggingface_hub

catherinearnett
b4485fa1y ago

Upload folder using huggingface_hub

catherinearnett
36a5f1a1y ago

Upload folder using huggingface_hub

catherinearnett