PrinceAlhassanNasamu/tekyerema-whisper-tiny-twi

Twi ASR (whisper-tiny, on-device) — CONTAMINATED, do not use

Author: Prince Nasamu Alhassan

Overview

Do not use this model, and do not quote any score it has ever produced. Its training pool contained all 144 of the evaluation recordings this project scores against, repeated 20 times. SHA-256 confirms 72 of 72 Twi GOLD clips are byte-identical to files in that pool. On the phone it replays 71 of 72 reference transcripts verbatim. It is published, and this card is written, so that the contamination has a citable object instead of a rumour.

Use it

import torch, soundfile as sf
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq

repo  = "PrinceAlhassanNasamu/tekyerema-whisper-tiny-twi"
proc  = AutoProcessor.from_pretrained(repo)
model = AutoModelForSpeechSeq2Seq.from_pretrained(repo).eval()

wav, sr = sf.read("clip.wav", dtype="float32")   # 16 kHz mono
inp = proc(wav, sampling_rate=16_000, return_tensors="pt")
with torch.no_grad():
    ids = model.generate(**inp, language="sw", task="transcribe")
print(proc.batch_decode(ids, skip_special_tokens=True)[0])

language="sw" is deliberate and is not a typo. Whisper has no Akan or Kusaal token, so training borrowed the Swahili slot as a proxy. Decode with any other language token and the model answers in the wrong language with full confidence.

What its numbers actually mean

The project's own report recorded 46/72 predictions verbatim gold — leakage caught before any of this was published, and the number 1.000 continued to be cited afterwards. TWI_ASR_RESULTS.md marks the model INVALID — TEST-SET LEAKAGE and disqualifies it from model selection.

The only natural-speech evidence for it is negative: free-form Twi phone commands produce garbage, and English audio fed through it returns confident pseudo-Twi.

The recipe, for the record

openai/whisper-tiny (39 M), 3 epochs, LR 1e-4, batch 16, 20 s clip budget, gold capped at 1,500 per config and pool sources at 1,500, with the owner's demo-exact recordings repeated 20× so ~144 clips could weigh against ~10k pool rows. That repetition is the contamination; it was a deliberate over-sampling choice whose consequence for evaluation was not followed through.

Decoding uses the Swahili language token sw as an Akan proxy, because Whisper has no Akan token.

Two further problems on this repo

• —The licence was wrong. It declared apache-2.0, inherited from the whisper-tiny base. The training pool includes CC BY-NC material, so the weights cannot be Apache. Corrected here to CC BY-NC 4.0.
• —It carries the author's voice. 144 personal recordings, over-sampled 20×, memorised to the point of verbatim replay — published under a permissive licence that was never intended to cover them.

What to use instead

• —Offline Twi: the project's clean line is whisper-base fine-tuned on WAXAL aka_asr_v2 (CC BY-4.0, spontaneous, speaker-disjoint, leakage gate dropped_leak=0) — 58.1% entity, above the clean reference. Held privately pending release review.
• —A clean public reference: `Qlerqly/griot-nano-1` scores 56.0% entity on the same 72 rows and was not trained on them. CC BY-NC-SA, prototype use only.
• —Server-side Twi: `tekyerema-asr-ctc-twi-v2`.

Limitations, stated plainly

• —A validation loss is not a benchmark. Several cards on this account used to print the training run's own eval loss under a heading that read like a result. Where this card gives a WER or CER it names the slice and the scorer; where no such number exists it says the model is unscored.
• —Scripture-derived audio does not generalise to speech. The Kusaal and much of the Ewe material is read scripture with a small number of narrators. Held out by CLIP rather than by SPEAKER it produces scores that look excellent and mean nothing -- the public bible_Kusaal figure of 11.69 is invalid for exactly this reason and must never be quoted.
• —Evaluation is on read and machine-translated text. No corpus of people speaking natural phone commands in these languages exists. Numbers measured this way are optimistic about phrasing and pessimistic about code-switching, and should not be read as field performance.
• —Research work from a hackathon entry, not a supported product.

The rest of the family

Twi — `tekyerema-asr-ctc` (v1, w2v-BERT CTC) · `tekyerema-asr-ctc-twi-v2` (continuation)

Kusaal — `tekyerema-asr-mms-kus` (MMS adapter, the one to use) · `tekyerema-asr-ctc-kus-v2` (new w2v-BERT line) · `kasa42-asr` (third-party baseline)

Ewe — `tekyerema-asr-mms-ewe` (the one to use) · `tekyerema-asr-ctc-ewe` · `tekyerema-asr-ctc-ewe-v3`

Other recognisers — `tekyerema-asr-mms-dag` · `tekyerema-asr-mms-hau` · `whisper-large-v3-turbo-tekyerema-eng`

Voices — `tekyerema-tts-twi` · `tekyerema-tts-kus` · `tekyerema-tts-ewe` · `tekyerema-tts-hau` · `tekyerema-tts-eng`

Agent models — `tekyerema-1-v2-tool-pilot` · `tekyerema-1-native-v2-tool-pilot` · `tekyerema-audio-native-4k`

Translation & routing — `tekyerema-nllb600m-v1` · `kusaal-nllb-600M` · `tekyerema-intent-afroxlmr`

Acknowledgements

Compute resources provided by AI Skills and Compute Africa (AISCA). Trained on the Ghana NLP H200 GPU and on Kaggle T4s. Please keep derivatives non-commercial and share improvements back with the Ghana NLP community (ghananlpcommunity).

Farmerline

Twi ASR training data is used with the permission of Farmerline, granted for non-commercial research use on 2026-08-28 through Ghana NLP. That permission is a condition of this model existing and is acknowledged here as required.