Modelpublic

CodeSoft/sorbet-25m

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

4likes500downloads

tokenizer_config.json9 linesDownload Raw Back to tokenizer

1{2  "backend": "tokenizers",3  "bos_token": "<|endoftext|>",4  "eos_token": "<|endoftext|>",5  "model_max_length": 1000000000000000019884624838656,6  "pad_token": "<|endoftext|>",7  "tokenizer_class": "PreTrainedTokenizerFast",8  "unk_token": "<|endoftext|>"9}