Datasetpublic

CipherSenseAI/afri-fertility-results

afri-fertility: African Language Tokenization Results Measurement dataset for The African Language Tax — the first systematic audit of the subword tokenization penalty imposed on African languages by frontier large language models. Every row is one (language, tokenizer, corpus) triple, with fertility, English-relative premium, and confidence intervals computed from a parallel corpus using sum-then-divide aggregation. Dataset summary Property Value Rows… See the full description on the dataset page: https://huggingface.co/datasets/CipherSenseAI/afri-fertility-results.

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes20downloads

Dataset Card

afri-fertility: African Language Tokenization Results

Measurement dataset for [The African Language Tax](https://arxiv.org/abs/2606.24460) — the first systematic audit of the subword tokenization penalty imposed on African languages by frontier large language models.

Every row is one (language, tokenizer, corpus) triple, with fertility, English-relative premium, and confidence intervals computed from a parallel corpus using sum-then-divide aggregation.

Dataset summary

Property	Value
Rows	616
Languages	22 (20 African + English + French baselines)
Tokenizers	11
Corpora	3 (FLORES-200+, SIB-200, MAFAND-MT)
File size	~58 KB
Format	Parquet
Tool	`afri-fertility`
Paper	arXiv 2606.24460

Key findings

—Every African language carries a tokenization premium above English on every tokenizer — 209 FLORES-200+ pairs (19 languages × 11 tokenizers), zero exceptions
—Premiums range from 1.29× (Swahili on BLOOM) to 8.92× (N'Ko on o200k_base)
—Ethiopic-script languages (Amharic, Tigrinya) reach 6.8–9.3×; N'Ko reaches 8.9×
—Best available tokenizer for African languages: Gemma 4 (mean 2.38× vs 3.31× for cl100k_base)
—Premium is near-invariant across corpora (FLORES vs SIB-200: Pearson r = 0.9998)

Load the dataset

python

import pandas as pd
from datasets import load_dataset

# Via HuggingFace datasets
ds = load_dataset("CipherSenseAI/afri-fertility-results")
df = ds["train"].to_pandas()

# Or directly as parquet
df = pd.read_parquet("hf://datasets/CipherSenseAI/afri-fertility-results/data/results.parquet")

Column reference

Column	Type	Description
`corpus`	str	Source corpus: `flores`, `sib200`, or `mafand`
`domain`	str	Text domain (where applicable)
`language`	str	Language display name
`iso639_3`	str	ISO 639-3 code
`script`	str	Writing script: `Latin`, `Ethiopic`, `N'Ko`
`family`	str	Language family
`tokenizer`	str	Tokenizer identifier (e.g. `openai/o200k_base`)
`vocab_size`	int	Tokenizer vocabulary size
`inspectable`	bool	Whether subword segmentation is accessible
`n_sentences`	int	Number of parallel sentences in the corpus slice
`n_words`	int	Total word count (UAX-29 word boundaries)
`n_tokens`	int	Total subword token count
`n_chars`	int	Total character count (NFC)
`n_bytes`	int	Total UTF-8 byte count
`fertility`	float	`n_tokens / n_words` — subword tokens per word
`premium`	float	`fertility(L,T) / fertility(English,T)` — English-relative multiplier
`cpt`	float	Characters per token
`bpt`	float	Bytes per token
`fertility_ci_low`	float	Bootstrap 95% CI lower bound for fertility
`fertility_ci_high`	float	Bootstrap 95% CI upper bound for fertility
`premium_ci_low`	float	Bootstrap 95% CI lower bound for premium
`premium_ci_high`	float	Bootstrap 95% CI upper bound for premium

Languages covered

Language	ISO	Script	Tier
Yoruba	yor	Latin	Core
Hausa	hau	Latin	Core
Igbo	ibo	Latin	Core
Wolof	wol	Latin	Core
Swahili	swh	Latin	Core
Amharic	amh	Ethiopic	Core
Zulu	zul	Latin	Breadth
Xhosa	xho	Latin	Breadth
Shona	sna	Latin	Breadth
Kinyarwanda	kin	Latin	Breadth
Luganda	lug	Latin	Breadth
Akan/Twi	aka	Latin	Breadth
Lingala	lin	Latin	Breadth
Oromo	gaz	Latin	Breadth
Nigerian Pidgin	pcm	Latin	Breadth
Sesotho	sot	Latin	Breadth
Bambara	bam	Latin	Breadth
N'Ko	nqo	N'Ko	Non-Latin
Tigrinya	tir	Ethiopic	Non-Latin
Afrikaans	afr	Latin	Control
English	eng	Latin	Baseline
French	fra	Latin	Baseline

Tokenizers measured

Tokenizer ID	Model family	Vocab size
`openai/o200k_base`	GPT-5 / o-series	200k
`openai/o200k_harmony`	gpt-oss open weights	201k
`openai/cl100k_base`	GPT-4 (legacy)	100k
`meta/llama-3.1`	Llama 3.1	128k
`meta/llama-4`	Llama 4 Scout/Maverick	~200k
`google/gemma-4`	Gemma 4	262k
`mistral/tekken`	Mistral / Tekken	131k
`deepseek/v3`	DeepSeek V3	128k
`bigscience/bloom`	BLOOM	251k
`cohere/aya-expanse`	Aya Expanse	256k
`qwen/qwen3`	Qwen 3	152k

Reproduce

Results are fully reproducible using the open-source afri-fertility tool:

bash

pip install afri-fertility
afri-fertility reproduce   # runs offline reference suite, prints headline numbers

Full study replication (requires HuggingFace token for gated models):

bash

git clone https://github.com/CipherSenseAI/afri-fertility
cd afri-fertility
pip install -e ".[dev]"
afri-fertility run configs/study_main.yaml

Citation

bibtex

@article{somide2026africanlanguagetax,
  title   = {The African Language Tax: Quantifying the Cost, Latency, and Context
             Penalty of Tokenizing African Languages in Frontier LLMs},
  author  = {Somide, Olaoye Anthony},
  year    = {2026},
  journal = {arXiv preprint arXiv:2606.24460},
  url     = {https://arxiv.org/abs/2606.24460}
}

License

Data: Apache 2.0. Source corpora: FLORES-200+ (CC-BY-SA-4.0), SIB-200 (CC-BY-SA-4.0), MAFAND-MT (per-dataset Masakhane licences).