OpenFormosa/PangolinTokenizer

1---2license: other3library_name: transformers4tags:5  - tokenizer6  - byte-level-bpe7  - traditional-chinese8  - taiwan9  - multilingual10---11 12# PangolinTokenizer13 14Byte-level BPE tokenizer for Traditional Chinese, Taiwan text, multilingual text,15rich transcription, OCR-style text, and generic control formats.16 17This revision adds the Open Formosa required control tokens as special tokens.18The base BPE vocabulary size remains 114,688. The effective tokenizer length,19including added special tokens, is 114,822.20 21## Usage22 23```python24from transformers import AutoTokenizer25 26tokenizer = AutoTokenizer.from_pretrained(27    "voidful/PangolinTokenizer",28    trust_remote_code=False,29)30 31text = "<|system|>台灣健保與注音ㄅㄆㄇ，Tailo: Tâi-uân"32ids = tokenizer.encode(text)33decoded = tokenizer.decode(ids)34```35 36## Open Formosa Compatibility37 38- Required special tokens present: 15739- Required special tokens encode as single IDs: yes40- Standard special tokens: `<unk>`, `<s>`, `</s>`, `<pad>`41- Model max length metadata: 131,07242- `trust_remote_code`: not required43- No discrete audio codec token ranges are included.44- No dense timestamp token ranges are included.45