NaolBM/Africa-BBPE

Africa-BBPE Tokenizer

📋 Model Card

Model Overview

• —Model Name: NaolBM/Africa-BBPE
• —Model Type: Byte-Level BPE Tokenizer
• —Base Model: Trained on NaolBM/african-corpus
• —Vocabulary Size: 50000
• —Languages Supported: Amharic, Swahili, Hausa, Oromo, Yoruba, Tigrinya, English

🎯 Performance Benchmarks

Tokenizer Battle Results

Comparison against Gemma-3 and Qwen3 tokenizers:

Summary Statistics

Language Family Performance

🌍 Language Support

🚀 Usage

from transformers import AutoTokenizer

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("NaolBM/Africa-BBPE")

# Example usage
text = "አማርኛ ቋንቋ በኢትዮጵያ"
tokens = tokenizer.tokenize(text)
ids = tokenizer.encode(text)

print(f"Tokens: {tokens}")
print(f"Token IDs: {ids}")
print(f"Number of tokens: {len(ids)}")

📊 Training Data

• —Dataset: NaolBM/african-corpus
• —Total Rows: 35,344,339
• —Languages: Amharic, Swahili, Hausa, Oromo, Yoruba, Tigrinya, English

Dataset Composition

🔍 Key Strengths

• —✅ 6x more efficient than Gemma-3 on Amharic (4 vs 15 tokens)
• —✅ 4x more efficient than Qwen-3 on Tigrinya (5 vs 21 tokens)
• —✅ 2x more efficient on code-switched text (10 vs 17-21 tokens)
• —✅ Best-in-class for 6/8 language categories
• —✅ Optimized for Ge'ez script (Amharic, Tigrinya)
• —✅ Excellent for Cushitic languages (Oromo)

📈 Efficiency Gains

Compared to Gemma-3:

• —73% fewer tokens for Amharic
• —62% fewer tokens for Tigrinya
• —55% fewer tokens for Oromo
• —41% fewer tokens overall

Compared to Qwen-3:

• —78% fewer tokens for Tigrinya
• —72% fewer tokens for code-switching
• —52% fewer tokens for Amharic
• —44% fewer tokens overall

📜 License

MIT

🙏 Acknowledgments

• —Trained on NaolBM/african-corpus
• —Built with Hugging Face Tokenizers library
• —All original dataset contributors

🔗 Links

• —Tokenizer: NaolBM/Africa-BBPE
• —Dataset: NaolBM/african-corpus