Modelpublic

0zuttag/agentic-shield

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes13downloads

tokenizer_config.json17 linesDownload Raw Back to root

1{
2  "backend": "tokenizers",
3  "clean_up_tokenization_spaces": true,
4  "cls_token": "[CLS]",
5  "is_local": false,
6  "mask_token": "[MASK]",
7  "model_input_names": [
8    "input_ids",
9    "attention_mask"
10  ],
11  "model_max_length": 8192,
12  "pad_token": "[PAD]",
13  "sep_token": "[SEP]",
14  "tokenizer_class": "PreTrainedTokenizerFast",
15  "unk_token": "[UNK]"
16}
17