AETHORIA-AI/TR-HASH-Tokenizer-32K-Agentic

TR-HASH Tokenizer 32K Agentic

Fresh 32,000-ID ByteLevel BPE tokenizer for the new TR-HASH agentic pretraining line. It was trained on a balanced, filtered 2B-token sample:

The general slice combines DCLM, FineWeb-Edu and Cosmopedia. The agentic slice combines Stack-Edu, filtered FineWeb-Edu, FineMath and InfiWebMath. Exact source quotas, record counts and input SHA-256 digests are recorded in selection_manifest.json.

Properties

• —Vocabulary size: exactly 32,000 IDs, including special tokens
• —Model: ByteLevel BPE
• —Declared model context: 2,048 tokens
• —Training format: tr_hash_agentic_reasoning
• —Native chat template with reasoning and tool-use markers

The 20 reserved IDs are:

0  <|begin|>              10 <|tool_result_start|>
1  <|end|>                11 <|tool_result_end|>
2  <|pad|>                12 <|plan_start|>
3  <|unk|>                13 <|plan_end|>
4  <|system|>             14 <|memory_start|>
5  <|user|>               15 <|memory_end|>
6  <|assistant|>          16 <|think_start|>
7  <|end_of_turn|>        17 <|think_end|>
8  <|tool_call_start|>    18 <|final_start|>
9  <|tool_call_end|>      19 <|final_end|>

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "AETHORIA-AI/TR-HASH-Tokenizer-32K-Agentic"
)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Plan, execute, and verify the task."}],
    tokenize=False,
    add_generation_prompt=True,
)

With add_generation_prompt=True, the template ends with <|assistant|><|think_start|>.

Fixed-suite tokenizer benchmark

The reproducible suite in benchmark_tiktoken.json covers English, French, code, JSON/tool payloads, mathematics and complex emoji sequences. Counts do not include automatic BOS/EOS insertion.

Natural text, code, JSON, math and emoji

TR-HASH uses 2.3% fewer tokens than r50k_base on this suite. The modern 100K and 200K tiktoken vocabularies compress natural text better, at the cost of much larger embedding tables.

Including native TR-HASH agentic markers

Native one-token control markers make TR-HASH 14.0% shorter than r50k_base and within 0.9% of cl100k_base on the combined suite. o200k_base remains 8.5% shorter. These are fixed-suite measurements, not universal compression claims.

Exact encode/decode round-trips pass for every category, including accented French, emoji, flags, skin tones, zero-width-joiner families and all 20 native markers.

pip install tokenizers tiktoken
python benchmark_tiktoken.py . --output benchmark_tiktoken.json

Compatibility

This is a new tokenizer, not an extension of AETHORIA-AI/TR-HASH-Tokenizer-32K. Token IDs were relearned from scratch. It is therefore incompatible with all existing TR-HASH checkpoints and with the previously tokenized 200B corpus.

Use it only for models pretrained from scratch on text retokenized with this exact tokenizer revision. Do not resize, migrate or pair an existing TR-HASH checkpoint with this vocabulary.

Integrity

agentic_tokenizer_manifest.json records the format, vocabulary size, special-token IDs and tokenizer-training contract. selection_manifest.json records the selected source material and its SHA-256 lineage.