Datasetpublic

Abzalbek89/kk-tokenizer-fertility-baseline

Kazakh Tokenizer Fertility Baseline Reproducible fertility benchmark of subword tokenizers on the Kazakh language. Companion artifact for the paper "Tokenizer Optimization for Kazakh Small Language Models" (in preparation, target: ACM TALLIP). Headline numbers Tokenizer Fertility 🥇 Best overall kk-bpe-32k 1.679 🚨 Worst GPT-4 (cl100k) 5.895 GPT-4 penalty GPT-4 (cl100k) is 3.51× worse than the best Kazakh-trained tokenizer → The custom Kazakh… See the full description on the dataset page: https://huggingface.co/datasets/Abzalbek89/kk-tokenizer-fertility-baseline.

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes601downloads

Dataset Card

Kazakh Tokenizer Fertility Baseline

Reproducible fertility benchmark of subword tokenizers on the Kazakh language. Companion artifact for the paper "Tokenizer Optimization for Kazakh Small Language Models" (in preparation, target: ACM TALLIP).

Headline numbers

	Tokenizer	Fertility
🥇 Best overall	`kk-bpe-32k`	1.679
🚨 Worst	`GPT-4 (cl100k)`	5.895
GPT-4 penalty	`GPT-4 (cl100k)` is 3.51× worse than the best Kazakh-trained tokenizer

→ The custom Kazakh BPE tokenizer is 3.51× more efficient than GPT-4 for Kazakh text, which translates to ~72% lower API cost and ~3.5× longer effective context window.

Full ranking (13 tokenizers)

Ours — Kazakh-trained, no morphology pre-segmentation

Tokenizer	Vocab	Fertility ↓	Chars/Token ↑
`kk-bpe-32k`	32,000	1.679	4.672
`kk-sp-bpe-32k`	32,000	1.758	4.463
`kk-sp-unigram-32k`	32,000	1.762	4.454
`kk-hf-unigram-32k`	32,000	2.026	3.872

Ours — Kazakh-trained, morphology-aware (Morfessor → tokenizer)

Tokenizer	Vocab	Fertility ↓	Chars/Token ↑
`kk-morph-hf-bpe-32k`	32,000	1.761	4.456
`kk-morph-sp-unigram-32k`	32,000	1.945	4.034
`kk-morph-hf-unigram-32k`	32,000	2.346	3.345

Reference — multilingual / industry tokenizers

Tokenizer	Vocab	Fertility ↓	Chars/Token ↑
`XLM-R`	250,002	2.181	3.597
`GPT-4o (o200k)`	200,019	2.599	3.020
`mBERT`	119,547	2.929	2.679
`Qwen2.5`	151,643	4.786	1.639
`Llama-3`	128,000	4.803	1.634
`GPT-4 (cl100k)`	100,277	5.895	1.331

Methodology

Held-out evaluation set: 2,966,207 whitespace-words (≈15K documents) from the validation split of `Abzalbek89/corpus_clean`.

Metrics:

—Fertility = total tokens / total whitespace-words (lower is better).
—Compression (chars/token) = total characters / total tokens (higher is better).
—Compression (bytes/token) = total UTF-8 bytes / total tokens (higher is better).

Trained tokenizers (vocab = 32,000, all share the same training corpus):

—kk-bpe-32k — HF tokenizers ByteLevel BPE
—kk-sp-bpe-32k — SentencePiece BPE (character_coverage=1.0, NFKC)
—kk-sp-unigram-32k — SentencePiece Unigram
—kk-hf-unigram-32k — HF tokenizers Unigram with ByteLevel pre-tok
—kk-morph-hf-bpe-32k — HF BPE on Morfessor-segmented corpus
—kk-morph-hf-unigram-32k — HF Unigram on Morfessor-segmented corpus
—kk-morph-sp-unigram-32k — SentencePiece Unigram on Morfessor-segmented corpus

Reference tokenizers: mBERT, XLM-R, Llama-3, Qwen 2.5, GPT-4 (cl100k_base), GPT-4o (o200k_base).

Files

Path	Description
`experiment.py`	v1 — single Unigram baseline
`experiment_v2.py`	v2 — extended (BPE/Unigram across libraries)
`experiment_v3_fix.py`	v2 fix — re-measure SentencePiece tokenizers via raw `SentencePieceProcessor`
`experiment_morph.py`	Experiment 2 — morphology-aware variants
`morfessor.bin`	trained Morfessor segmentation model (top-1M Kazakh words)
`v2/fertility_v2.csv`, `.json`, `.png`	fixed v2 numbers
`v3/fertility_v3.csv`, `.json`, `.png`	extended v3 numbers (with morph-aware)
`v3/RESULTS_V3.md`	human-readable v3 report

Reproducing

On a vast.ai / RunPod / Colab instance with ≥30GB disk and ≥16GB RAM:

bash

pip install 'datasets>=2.14' 'transformers>=4.45' 'tokenizers>=0.20' \
            'sentencepiece>=0.2.0' 'huggingface_hub>=0.25' \
            tiktoken matplotlib morfessor

export HF_TOKEN=hf_...   # write access
export HF_USER=Abzalbek89

curl -fsSL -o experiment_morph.py \
  https://huggingface.co/datasets/Abzalbek89/kk-tokenizer-fertility-baseline/resolve/main/experiment_morph.py
python experiment_morph.py

Total runtime: ~30 min on cached Morfessor model, ~70 min cold start.

Companion tokenizer repos

¹ Morphology-aware variants — apply Morfessor (morfessor.bin in this repo) before encoding.

Citation

bibtex

@misc{kk_tokenizer_fertility_2026,
  title        = {Kazakh Tokenizer Fertility Baseline},
  author       = {Abzalbek Ulasbek},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/Abzalbek89/kk-tokenizer-fertility-baseline}},
}

License

Apache 2.0

Last updated: fertilityv3.csv (13 tokenizers)_