Modelpublic

hf-tiny-v2/tiny-random-GPTBigCodeForSequenceClassification

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes16downloads

tokenizer_config.json18 linesDownload Raw Back to root

1{2  "backend": "tokenizers",3  "bos_token": "<|endoftext|>",4  "eos_token": "<|endoftext|>",5  "errors": "replace",6  "extra_special_tokens": [7    "<|endoftext|>",8    "<fim-prefix>",9    "<fim-middle>",10    "<fim-suffix>",11    "<fim-pad>"12  ],13  "is_local": true,14  "local_files_only": false,15  "model_max_length": 512,16  "tokenizer_class": "TokenizersBackend"17}18