tachiwin/tokenizer_test2

Tachiwin multilingual tokenizer

A multilingual ByteLevel-BPE tokenizer trained with Hugging Face Tokenizers.

Corpus weighting

The complete available exotic-language corpus is used as the 70% anchor. Its existing modern/old composition is preserved.

Tokenizer

• —Model: BPE
• —Vocabulary target: 256,000
• —Initial alphabet: complete ByteLevel alphabet
• —ByteLevel GPT-2 regex: disabled
• —Unicode normalizer: none
• —Special tokens: 126
• —Human-language tags: 92

Corpus size

Total materialized corpus:

113,921,401 bytes (0.106 GiB)

Important training note

The Hugging Face BPE trainer does not expose an internal resumable merge-state checkpoint. The recipe therefore treats the completed tokenizer.json as the training checkpoint:

• —corpus preparation is resumable;
• —recipe/statistics/checksums are stored in recipe/;
• —if tokenizer.json already exists, subsequent runs skip BPE training;
• —an interrupted BPE computation itself must be restarted.

This avoids changing the training procedure merely to obtain artificial checkpointing.