Apppublic

qikp/tokenizers

sourceHugging Facemitupdated 27d agoView on Hugging Face

0likes

README.md13 linesDownload Raw Back to root

1---2title: Some Tokenizers3emoji: ✂️4colorFrom: red5colorTo: yellow6sdk: static7pinned: false8license: mit9short_description: Tokenize text using some obscure-ish tokenizers10---11 12Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference13