hadamard-2/leyu-amharic-shewa-dialect

Leyu Amharic - Shewa Dialect Speech Corpus

Dataset Description

A parallel speech corpus of audio recordings paired with their transcripts, focused on the Shewa dialect of Amharic, for ASR and TTS research. Leyu reports that recordings were collected from contributors on mobile devices in real-world environments, and that each audio–text pair was manually reviewed for transcript accuracy and audio clarity.

This repository is a copy of `leyu-amharic/leyu-amharic-shewa-dialect`, collected and published by Leyu (iCog Labs). The audio and parquet files are byte-identical to that source; only this card has been revised, to correct figures that did not match the data.

Dataset Summary

Recordings by gender: 6,028 female, 2,152 male.

Dataset Structure

Recordings are of scripted passages rather than free speech, and the same passage is recorded by several different speakers. This corpus contains 1,221 distinct passages over 8,180 recordings — a mean of 6.7 recordings per passage (median 4, maximum 19). Some passages are registered under more than one id: the 1,221 passage ids resolve to 1,219 distinct transcripts, so grouping by passage id is not the same as grouping by text. Transcripts have a median length of 347 characters.

Splits

This corpus ships as a single train split. A leakage-free train/validation/test assignment is published alongside it as `split.json` — the parquet is untouched.

Guarantees. No speaker and no transcript in dev or test appears in the train split of any of the five Leyu dialect corpora — the assignment was computed jointly over all five, because speaker ids and some transcripts recur across them. dev and test share no speaker with each other.

`dropped` is why the first three do not sum to 8,180. These are recordings by a held-out speaker of a passage that a training speaker also read: they cannot go in train without exposing a held-out voice, and cannot go in dev or test without putting a memorised transcript in the evaluation. They are assigned to neither. There is nothing wrong with the audio.

`dev` and `test` do share transcripts with each other — 57% of test recordings here have their text somewhere in dev. Separating them on the text axis too would have discarded roughly another 50 hours across the five corpora, so it was declined. Both remain fully disjoint from train, which is the guarantee a reported error rate rests on; but a heavily validation-driven hyperparameter search will read slightly optimistically on test.

Join on `record_id`, the MTA (microtask assignment) id parsed from audio.path, which is unique across all 49,478 rows of all five corpora. Row positions are included as a convenience only: they are valid against the current parquet row order and would resolve to the wrong recordings, without raising anything, if the files were ever re-sharded.

Building your own split instead

Reasons to: different proportions, k-fold rotation so the dropped recordings get used across folds, pooling these corpora with other Amharic speech, or wanting dev and test transcript-disjoint as well.

If you do, group by transcript text as well as by `speaker_id`. Splitting on speaker alone leaves the identical transcript on both sides of the split; splitting on transcript alone leaves the same voices on both sides. Group by the text itself rather than the passage id — as noted above, distinct passage ids can carry identical text.

Nine speaker ids appear in more than one of the five Leyu dialect corpora, and 30 transcripts appear in more than one, so the same grouping applies when these corpora are pooled.

Data Fields

• —text (string): transcript of the passage
• —audio (Audio): 8 kHz, 8-bit mono PCM WAV
• —dialect (string): shewa
• —speaker_id (string): anonymized speaker identifier
• —gender (string): male / female / unknown

Provenance of the figures on this card

Recording counts, speaker counts, gender breakdown, passage counts and transcript statistics were computed over all 8,180 rows of the published parquet files. Total duration was derived from the audio column's uncompressed byte size at 8 kHz, 8-bit, mono (8,000 bytes/second), less the 44-byte WAV header per recording. Per-split hours in the table above are approximate: per-recording durations are not stored in the parquet, so every recording is assigned its shard's mean duration. Corpus totals and all recording counts are exact.

iCog Blogs

• —Leyu: Crowdsourcing Datasets for Ethiopian Languages
• —Dialects & Socioeconomics: Shaping Inclusive Language Models
• —Progress of Natural Language Processing (NLP) for Ethiopian Languages – Part One
• —Progress of Natural Language Processing (NLP) for Ethiopian Languages – Part Two
• —Data Collection with Purpose: The Leyu Approach