Modelpublic

internlm/internlm2-7b

sourceHugging Faceotherupdated 2y agoView on Hugging Face

43likes25kdownloads

tokenizer_config.json15 linesDownload Raw Back to root

1{2  "auto_map": {3    "AutoTokenizer": [4      "tokenization_internlm2.InternLM2Tokenizer",5      "tokenization_internlm2_fast.InternLM2TokenizerFast"6    ]7  },8  "bos_token": "<s>",9  "clean_up_tokenization_spaces": false,10  "eos_token": "</s>",11  "model_max_length": 1000000000000000019884624838656,12  "pad_token": "</s>",13  "tokenizer_class": "InternLM2Tokenizer",14  "unk_token": "<unk>"15}