Modelpublic

CodeSoft/sorbet-mini-experimental

sourceHugging Faceapache-2.0updated 22d agoView on Hugging Face

1likes1.4kdownloads

tokenizer_config.json12 linesDownload Raw Back to root

1{2  "backend": "tokenizers",3  "bos_token": "<|endoftext|>",4  "eos_token": "<|endoftext|>",5  "is_local": true,6  "local_files_only": false,7  "model_max_length": 1000000000000000019884624838656,8  "pad_token": "<|endoftext|>",9  "tokenizer_class": "TokenizersBackend",10  "unk_token": "<|endoftext|>"11}12