CipherSenseAI/afri-fertility-results

afri-fertility: African Language Tokenization Results Measurement dataset for The African Language Tax — the first systematic audit of the subword tokenization penalty imposed on African languages by frontier large language models. Every row is one (language, tokenizer, corpus) triple, with fertility, English-relative premium, and confidence intervals computed from a parallel corpus using sum-then-divide aggregation. Dataset summary Property Value Rows… See the full description on the dataset page: https://huggingface.co/datasets/CipherSenseAI/afri-fertility-results.