Modelpublic

stringbot/ioai

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes6downloads

special_tokens_map.json25 linesDownload Raw Back to tokenizer

1{2  "bos_token": {3    "content": "<|startoftext|>",4    "lstrip": false,5    "normalized": true,6    "rstrip": false,7    "single_word": false8  },9  "eos_token": {10    "content": "<|endoftext|>",11    "lstrip": false,12    "normalized": true,13    "rstrip": false,14    "single_word": false15  },16  "pad_token": "<|endoftext|>",17  "unk_token": {18    "content": "<|endoftext|>",19    "lstrip": false,20    "normalized": true,21    "rstrip": false,22    "single_word": false23  }24}25