Modelpublic

refactai/codify_3b_multi

sourceHugging Facebigscience-openrail-mupdated 4y agoView on Hugging Face

8likes25downloads

tokenizer_config.json14 linesDownload Raw Back to root

1{2  "add_prefix_space": false,3  "auto_map": {4    "AutoTokenizer": [5      null,6      "tokenization_codify_fast.CodifyTokenizerFast"7    ]8  },9  "bos_token": "<|endoftext|>",10  "eos_token": "<|endoftext|>",11  "tokenizer_class": "CodifyTokenizer",12  "unk_token": "<|endoftext|>"13}14