AfriSpeech/shola-machine-translations

SHOLA machine translations

Machine translations of everyday English words into African languages, from four systems, published so their output can be compared against each other and against what native speakers actually say.

These are candidate translations, not verified ones. They exist so a speaker evaluating them at SHOLA has something to agree with or correct instead of a blank box. Some are wrong. That is the point: which ones speakers pick is the measurement.

Licence

CC-BY-NC-4.0 — non-commercial. The restriction comes from NLLB-200, whose weights are CC-BY-NC-4.0 under Meta's original release. Anything its output touches inherits that, and the whole dataset is published under it so there is no ambiguity about which directory carries which terms.

Contents

One JSON object per line:

{"phrase": "water", "language": "yor", "text": "omi"}

language is ISO 639-3 as SHOLA stores it.

How the extra languages were chosen

gemma-4-31b-it-jw/ began as 282 African languages drawn from a JW.org language list. A large language model produces confident, fluent output for a language it has never been trained on, so the list was filtered before any of it was translated:

1. 1.Echo test. Translate 40 everyday words. A model with nothing to say returns the English unchanged. This removed 25 languages.
2. 2.Self-consistency. Sample the same 20 words three times at temperature 0.8 and compare on case-folded, accent-stripped forms. A word the model knows comes back the same each time; one it is inventing does not. Six languages Gemma demonstrably handles — Swahili, Zulu, Hausa, Shona, Lingala, Yoruba — scored 15–20 out of 20 and set the threshold. This removed a further 182.

75 of the original 282 survived. The rest were not translated, because options a speaker cannot recognise waste the time of the one person whose time this project cannot buy more of.

Comparing raw strings scored Yoruba 2/20 while it was returning the correct word every time — its tone marks moved between samples. That is why the comparison is on normalised forms.

Even so

A language can pass both tests and still be partly invented. Treat everything in gemma-4-31b-it-jw/ as a prompt for a speaker to correct, not as a translation to rely on.

Word list

The words come from GhanaNouns: English noun phrases drawn from Ghanaian news, research and speech. Of the 478,822 there, the 71,014 seen five or more times in the corpus are translated here. The rest are a long tail of extraction noise — over 122,000 appear exactly once.

Coverage is not the same across systems

Google and NLLB do not cover the same languages. NLLB reaches Chokwe, Kabyle, Kamba, Kabiye, Kabuverdianu, Kikuyu, Kimbundu, Mossi and Umbundu, which Google does not. Google reaches Ga, Krio, Afar, Acholi, Fula, Malagasy, Tiv, Venda and Baoule, which NLLB does not. Neither is a superset of the other.

Why gemini-3.6-flash/ is superseded

That run asked Gemini for three wordings per word while every other system gives one. Three wordings get three chances to match what a speaker picks, so its pick rate measured the extra chances as much as the translation. gemini-3.6-flash-one/ is the same model asked for a single wording. Use that.

The 75 added languages

Abbey, Abua, Ahanta, Bembe, Bissau Guinean Creole, Budza, Chitonga, Chitonga (Malawi), Chiyao, Chuabo, Comorian (Ngazidja), Fante, Fulfulde (Cameroon), Giryama, Gitonga, Haya, Hehe, Ibinda, Itsekiri, Kabuverdianu, Kalanga (Botswana), Kamba, Kamwe, Kanyok, Kihemba, Kiluba, Kinande, Kipende, Kisi, Kisonge, Kunda, Kuria, Lamba, Lari, Lenje, Lhukonzo, Lomongo, Lomwe, Luvale, Mahorian (Roman), Makonde, Mambwe-Lungu, Mashi, Matengo, Meru, Nambya, Ndali, Ndau, Ndebele (Zimbabwe), Ngangela, Nsenga (Mozambique), Nyamwezi, Nyaneka, Nyungwe, Olunyole, Otetela, Phimbi, Punu, Rumanyo, Runyankore, Sanga, Sena, Silozi, Songo, Songomeno, Sukuma, Swahili (Congo), Sãotomense, Taita, Tamazight, Teke (Congo Brazzaville), Teke (Congo Kinshasa), Tshwa, Yaka, Yakoma