addisai/amharic-tts-benchmark

Amharic TTS Benchmark

Seven text-to-speech systems and the original human recordings, evaluated on 100 Amharic prompts from three open datasets. Run date 2026-08-12.

Published results: addisassistant.com/benchmarks

Reproduce the CER/WER results

python score.py

No arguments. It reads data/judge_rows.jsonl, recomputes every character and word edit count from the transcripts and writes data/summary.json.

This covers the CER/WER results only. Listening scores, latency and cost are published on the results page and are not derivable from ASR transcripts.

Primary table, 87 of 100 prompts, digits excluded. Lower is better. The original human recordings are included in data/ and audio/ as a reference point but are not a synthetic system and are not ranked here.

Contents

Prompt set

Public test-split rows pinned to exact dataset revisions, drawn once with seed addis-voice-public-benchmark-v1, balanced 33 short / 34 medium / 33 long.

Every shipped file is hashed in checksums.sha256, including all 800 audio clips. Each row of data/judge_rows.jsonl also carries audio_path and audio_sha256, so a transcript can be tied to the exact file that produced it without guessing.

Scoring definition

WER and CER were measured with Chirp 3 as the ASR judge, so the scores are a recognition-based proxy for intelligibility rather than a direct measure of it.

All audio is converted to 16 kHz mono PCM WAV before scoring. Then:

1. 1.Unicode NFC normalisation
2. 2.punctuation stripped, Ethiopic and ASCII
3. 3.Ethiopic homophone families folded to one canonical form, see normalize.py
4. 4.whitespace collapsed
5. 5.CER is Levenshtein distance over the normalised text with spaces removed
6. 6.WER is Levenshtein distance over whitespace-split tokens
7. 7.rates are micro-averages of raw edit counts, not means of per-clip rates

Step 5 is the one that catches people out. Scoring CER with spaces included gives roughly double the error rate.

Prompts containing digits are excluded from the primary table, because a written digit and its spoken form are not the same string and no equivalence rule has been validated. The strict full-set figures over all 100 prompts are in data/summary.json.

Running it with a different recogniser

The pipeline is recogniser-agnostic and an independent judge is the most useful thing an outside party can contribute. Two open Amharic ASR models:

• —`b1n1yam/shook-medium-amharic-2k`
• —`b1n1yam/qwen3-asr-0.6b-amharic-mixed`

Transcribe audio/<system>/, replace hypothesis_raw in data/judge_rows.jsonl, rerun score.py and compare the ordering.

Addis Voice 2 is reachable on the public API as model addis-voice-2.

Limitations

• —Recognition error is a proxy for intelligibility. It says nothing about prosody, emphasis or how a long passage sounds.
• —Systems without documented Amharic support were run anyway and their rows describe an unsupported-language condition, not their quality elsewhere.
• —Word error above 100% is possible when a system inserts more words than the reference contains.
• —Human reference recordings are field audio with variable level and microphone quality, and two thirds of them are spontaneous rather than read speech. They are a reference point, not a quality ceiling.
• —One run on one date. Hosted models change under their version strings.

Licensing

This repository contains material under several different licences. There is no single licence covering all of it, and Addis AI does not relicense any content it does not own.

The per-clip source dataset and its licence are recorded for every prompt in data/prompts.jsonl, so the terms applying to any individual human recording can be resolved from that file.

Third-party generated audio is included as evaluation evidence so the published error rates can be independently verified. It is not offered for reuse, and anyone redistributing it is responsible for their own compliance with the relevant provider's terms.

Contact

contact@addisassistant.com