Modelpublic

pythonstudentiam/tinyllm

sourceHugging Facecdla-sharing-1.0updated 2mo agoView on Hugging Face

0likes70downloads

tokenizer_config.json13 linesDownload Raw Back to root

1{2  "add_prefix_space": null,3  "backend": "tokenizers",4  "bos_token": "<s>",5  "clean_up_tokenization_spaces": false,6  "eos_token": "<|im_end|>",7  "model_max_length": 512,8  "pad_token": "<pad>",9  "tokenizer_class": "LlamaTokenizer",10  "unk_token": "<unk>",11  "use_default_system_prompt": false12}13