Modelpublic

OpenMachineAI/UnitronX-Tokenizer-32k-v1

sourceHugging Facemitupdated 11mo agoView on Hugging Face

0likes

Model Card

{DISPLAY_NAME}

UnitronX is a 32k byte-level BPE tokenizer optimized for English, multilingual (ru/ar/de/es/fr/it/cs/hr/sr), and code. It enforces safe merge boundaries (script changes, ZWJ, letter↔digit), preserves code identifiers, and uses placeholder tokens for URLs/emails/paths/hashes/UUIDs/handles/hashtags.

Files

—tokenizer.json, merges.txt, vocab.json
—tokenizer_config.json, special_tokens_map.json
—meta.json
—(optional) unitronx.tiktoken.json (tiktoken-compatible)

Load with Transformers

python

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("UnitronX-Tokenizer-32k-v1")
print(tok.encode("don't split hyphen-words or fooBar123_id in code!"))