ghananlpcommunity/ghana-speech-ipa-asr-ready

ghana-speech-ipa-asr-ready

The training-ready corpus behind ghananlpcommunity/ghana-speech-phoneme-asr: 16 kHz audio paired with IPA phoneme targets, already in the partitioned parquet layout that omnilingual-asr's MixtureParquetStorage expects.

Use this if you want to resume or repeat the training without redoing ingestion, which takes a couple of hours of streaming and re-encoding.

Note on the dataset preview

In the Hub viewer, audio_bytes shows as a list of numbers and text shows as blank or truncated. That is only the storage format the trainer needs — the data is correct. See Schema below for how to decode both columns.

Layout

version=0/
└── corpus=ghana_speech/
    ├── split=train/
    │   ├── language=twi-asante_Latn/part-0.parquet
    │   ├── language=ewe_Latn/part-0.parquet
    │   └── ...                                    (42 languages)
    └── split=dev/
        └── ...                                    (42 languages)

corpus, split and language are Hive partition keys only — they are deliberately not columns inside the files. pyarrow materialises them as dictionary<string> from the directory names; duplicating them in-file causes incompatible types: string vs dictionary on dataset discovery.

Schema

Files are written with row_group_size=100, per the omniASR data-prep guide.

The text column is proxy-encoded

text is not readable IPA. Each phoneme unit is stored as one codepoint in the Private Use Area (U+E000…), because omniASR's tokenizer family segments by character and would otherwise split multi-character units like kʰ and k͡p into pieces.

To read it as IPA, use the mapping published with the model (tokenizer/phonemes.json):

import json, pyarrow.parquet as pq
from huggingface_hub import hf_hub_download

spec = json.load(open(hf_hub_download(
    "ghananlpcommunity/ghana-speech-phoneme-asr", "tokenizer/phonemes.json")))
back = {v: k for k, v in spec["proxy"].items()}

t = pq.read_table("version=0/corpus=ghana_speech/split=dev/language=any_Latn/part-0.parquet")
text = t.column("text")[0].as_py()
print(" ".join(back[c] for c in text))
# ɛ h ɪ , ɐ n i ɐ n m ɐ n - m ɔ , m ʔ ɔ t ʊ d e ...

Decoding the audio:

import io, numpy as np, soundfile as sf
raw = np.array(t.column("audio_bytes")[0].as_py(), dtype=np.int8).tobytes()
wav, sr = sf.read(io.BytesIO(raw), dtype="float32")   # 16000 Hz mono

How it was built

From ghana-speech audio and ghana-speech-phonemes targets (ipa_phonemes_spaced_punct), joined on id.

Clips were dropped when they were:

• —shorter than 0.4 s or longer than 39 s (omniASR CTC accepts under 40 s)
• —CTC-infeasible — the target needs more frames than the audio provides. CTC requires one frame per label plus a blank between identical neighbours; targets that do not fit yield infinite loss rather than an error, so they are removed here rather than silently poisoning training. About 0.35% of clips.
• —missing a phoneme target (digit-only transcriptions produce none)

34,646 of 1,411,467 source clips were dropped, 2.5%.

Akuapem and Asante Twi share ISO 639-3 `twi` and are kept apart as twi-akuapem_Latn and twi-asante_Latn. Using the bare code would put two different dialects in one partition and silently lose one of them.

Language distribution

language_distribution.tsv (corpus / language / hours) accompanies the model and drives temperature sampling. It matters: the corpus runs from Dagaare at 15 h to Asante Twi at 200 h, and with beta_language=0.5 the sampler weights by the square root of hours so the small languages are not swamped.

Training with it

dataset:
  name: "ghana_speech_ipa"
  train_split: "train"
  valid_split: "dev"
  storage_mode: "MIXTURE_PARQUET"
  task_mode: "ASR"
  mixture_parquet_storage_config:
    dataset_summary_path: "/path/to/language_distribution.tsv"
    beta_corpus: 0.5
    beta_language: 0.5

The full recipe config, asset cards and scripts are in GhanaNLP/ghana-phoneme-asr.

Licence

CC BY-NC 4.0, following the source audio corpus.