Modelpublic

SupraLabs/Supra-Router-51M

sourceHugging Faceupdated 3mo agoView on Hugging Face

148likes2.4kdownloads

tokenizer_config.json13 linesDownload Raw Back to root

1{2  "backend": "tokenizers",3  "bos_token": "<s>",4  "clean_up_tokenization_spaces": false,5  "eos_token": "</s>",6  "is_local": true,7  "local_files_only": false,8  "model_max_length": 5120,9  "pad_token": "<pad>",10  "tokenizer_class": "TokenizersBackend",11  "unk_token": "<unk>"12}13