CoolFace
Modelpublic

flexitok/mod-tokenizers-individual

sourceHugging Facemitupdated 7mo agoView on Hugging Face
0likes32downloads
Model Card

Byte-Level BPE Tokenizer: numeric (0K)

A Byte-Level BPE tokenizer trained on numeric data from Fineweb-2-HQ.

Training Details

ParameterValue
AlgorithmByte-Level BPE
Languagenumeric
Target Vocab Size17
Final Vocab Size17
Pre-tokenizerbyte_level
Number handlingindividual
Contraction handlingFalse
NormalizerNONE
Special Tokens<s>, </s>, <pad>, <unk>
Training Shards1

Usage

python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("None")
tokens = tokenizer.encode("Hello, world!")

Files

  • —tokenizer.json — Full HuggingFace tokenizer
  • —vocab.json — Vocabulary mapping
  • —merges.txt — BPE merge rules

Sample Encoding

TextTokensToken IDs
123500119 mod 671, 2, 3, 5, 0, 0, 1, 1, 9, , mod, , 6, 78, 9, 10, 12, 7, 7, 8, 8, 16, 6, 4, 6, 13, 14