Modelpublic

peteryushunli/Fill_Mask_Tutorial_Model

sourceHugging Faceupdated 3y agoView on Hugging Face

0likes9downloads

tokenizer_config.json24 linesDownload Raw Back to root

1{2  "additional_special_tokens": [3    "<s>NOTUSED",4    "</s>NOTUSED"5  ],6  "bos_token": "<s>",7  "clean_up_tokenization_spaces": true,8  "cls_token": "<s>",9  "eos_token": "</s>",10  "mask_token": {11    "__type": "AddedToken",12    "content": "<mask>",13    "lstrip": true,14    "normalized": true,15    "rstrip": false,16    "single_word": false17  },18  "model_max_length": 512,19  "pad_token": "<pad>",20  "sep_token": "</s>",21  "tokenizer_class": "CamembertTokenizer",22  "unk_token": "<unk>"23}24