Modelpublic

MartinNav/compliantLLM

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes25downloads

tokenizer_config.json15 linesDownload Raw Back to root

1{2  "auto_map": {3    "AutoTokenizer": [4      "tokenization_compliant_llm.CompliantLLMTokenizer",5      null6    ]7  },8  "clean_up_tokenization_spaces": false,9  "model_max_length": 1024,10  "pad_token": "<0x00>",11  "padding_side": "right",12  "tokenizer_class": "CompliantLLMTokenizer",13  "truncation_side": "left"14}15