flexitok/mod-tokenizers-ltr_5digit

Byte-Level BPE Tokenizer: numeric (100K)

A Byte-Level BPE tokenizer trained on numeric data from Fineweb-2-HQ.

Training Details

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("None")
tokens = tokenizer.encode("Hello, world!")

Files

• —tokenizer.json — Full HuggingFace tokenizer
• —vocab.json — Vocabulary mapping
• —merges.txt — BPE merge rules

Sample Encoding