yhavinga/dutch-tokenizer-arena

Update requirements.txt

Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.

Pin protobuf dep

Hack some Dutch tokenizers into it

update

update

add compression leaderboard

update compress rate

update compress rate

update

add grok mixtral

add compress rate

add zephyr

requi

requi

requi

config python version

add xlm-roberta

add amber and crystal_coder

add character glm

update

fix PyO3PanicException

fix unicode error: 'unicodeescape' codec can't decode bytes in position 602-608: unknown Unicode character name

fix fastchat_t5_3b

fix tiktoken special tokens

add aya

fix olmo

add olmo tokenizer

update

update

fix tiktoken

fix gemma_7b

add gemma_7b

add more tokenizer

fix tokenize

add more tokenizer

update

update

fix chatglm; new feature about add_special_tokens;

update

Merge branch 'main' of hf.co:spaces/eson/tokenizer-arena

add more tokenizer

Update README.md

update

add more tokenizers

add more tokenizer

add more tokenizer

fix moss

update

update