NaolBM/Geez-BBPE

🚀 Geez-BBPE Tokenizer (NaolBM/Geez-BBPE)

A highly efficient Byte-Level BPE tokenizer specifically trained for Geez-script languages (Amharic, Tigrinya). With 25,000 vocabulary size, it achieves 3.67 chars/token on Amharic text - significantly outperforming generic tokenizers.

📊 Benchmark Comparison

Sample: "የኢትዮጵያ ታሪክ በጣም ረጅም እና ባለብዙ ደረጃዎች ነው።" (8 characters)

🧠 Why Geez-BBPE?

Byte-Pair Encoding (BPE) tokenizers trained on English or Latin-script languages often fail to tokenize Geez-script languages efficiently, breaking words into meaningless byte sequences.

Geez-BBPE solves this by:

• —✅ Preserving semantic meaning - common words are single tokens
• —✅ Reducing sequence length - fit more text in same context
• —✅ Faster inference - fewer tokens = faster generation
• —✅ Better learning - model sees whole words, not fragments
• —✅ Native support - trained on 10M+ authentic Amharic sentences

📚 Training Details

• —Tokenizer Type: Byte-Level BPE
• —Vocabulary Size: 25,000 (optimized for Geez script)
• —Training Data: 10M+ sentences from b1n1yam/amharic-combined-corpus
• —Pre-tokenizer: Byte-Level with space preservation
• —Special Tokens: <|startoftext|>, <|im_end|>, <|pad|>

📁 Files

• —tokenizer.json: Full tokenizer config
• —tokenizer_config.json: Hugging Face-compatible configuration
• —special_tokens_map.json: Maps for special tokens

🚀 Usage

Standalone Usage

from transformers import AutoTokenizer

# Load Geez-BBPE tokenizer
tokenizer = AutoTokenizer.from_pretrained("NaolBM/Geez-BBPE")

# Tokenize Amharic text efficiently!
text = "ኢትዮጵያ ዋና ከተማዋ አዲስ አበባ ናት።"
tokens = tokenizer.tokenize(text)
ids = tokenizer.encode(text)

print(f"Tokens ({len(tokens)}): {tokens}")
print(f"Token IDs: {ids}")

Extending Other Models with Geez-BBPE

If you want to extend another model's tokenizer (like Qwen, llama, LMF etc...) using Geez-BBPE:

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load tokenizers
geez = AutoTokenizer.from_pretrained("NaolBM/Geez-BBPE")
target = AutoTokenizer.from_pretrained("your-target-model")

new_tokens_ids = [
    id for id in range(geez.vocab_size)
    if geez.decode([id]) not in target.all_special_tokens
]

# 🔑 CRITICAL: Decode Geez tokens to readable Amharic before adding!
# decode() converts 'ĠáĬł...' back to " አበባ" or "አዲስ"
tokens_to_add = [geez.decode([id]) for id in new_tokens_ids]

# Remove duplicates
decoded_tokens = list(set(tokens_to_add))
print(f"Adding {len(decoded_tokens)} Amharic tokens")

# Add to target tokenizer
target.add_tokens(decoded_tokens)

# Resize model
model = AutoModelForCausalLM.from_pretrained("your-target-model")
model.resize_token_embeddings(len(target))

❌ Common Mistake to Avoid

# WRONG: Adding raw byte tokens (they won't match input text!)
target.add_tokens(list(geez.get_vocab().keys()))  
# This adds things like 'áĬłáĭ²áĪµ' - the tokenizer will NEVER find these in text!

# RIGHT: Decode first, then add
decoded = geez.decode([token_id])  # Converts 'áĬłáĭ²áĪµ' → 'አዲስ'
target.add_tokens([decoded])  # Now matches actual input text!

Why this works: Geez-BBPE stores tokens as byte-level strings internally, but the tokenizer looks for actual Amharic text in the input. decode() bridges this gap by converting byte representations back to readable characters.

📊 Intended Use

This tokenizer is best suited for:

• —Low-resource NLP pipelines
• —Machine Translation
• —Question Answering
• —Named Entity Recognition
• —Morphological analysis
• —Continued Pre-Training (CPT) for Amharic/Tigrinya

⚠️ Limitations

• —Optimized for Geez-script languages (Amharic, Tigrinya) - may not generalize to others
• —Some compound words may still benefit from linguistic preprocessing
• —Currently focused on Amharic/Tigrinya; doesn't support multilingual code-switching

✅ Evaluation

The tokenizer was evaluated on:

• —Token coverage of Amharic/Tigrinya corpora
• —Morphological preservation
• —Character-to-token ratio (target: >3.0 chars/token)
• —Real-world downstream task performance

📜 License

This tokenizer is licensed under the MIT License.

📌 Citation

@misc{naol2025geezbbpe,
  title={Geez-BBPE: An Efficient Byte-Level BPE Tokenizer for Amharic and Tigrinya},
  author={Naol},
  year={2026},
  howpublished={\url{https://huggingface.co/NaolBM/Geez-BBPE}},
}

🙏 Acknowledgments

• —Built with 🤗 Hugging Face Transformers
• —Trained on b1n1yam/amharic-combined-corpus
• —Optimized for efficient Amharic NLP
• —Inspired by the Ethiopian AI community's need for better language tools

<div align="center"> <h3>⭐ If you find this useful, please star the repo! ⭐</h3> <p>Made with ❤️ for Ethiopian AI</p> </div>