Datasetpublic

galsenai/WaxalNLP

Google WaxalNLP Wolof Re-alignment Google introduced WAXAL, a new open dataset for 21 African languages, to tackle data scarcity and build inclusive speech technology. However, the Wolof language has experienced alignment issues between the audio files and their transcriptions, making the dataset unusable. We therefore propose to correct this using a simple and effective approach: For each audio clip, we generated a transcription using Google Gemini ASR. For each generated… See the full description on the dataset page: https://huggingface.co/datasets/galsenai/WaxalNLP.

sourceHugging Faceupdated 6mo agoView on Hugging Face

10likes151downloads

Dataset Card

Google WaxalNLP `Wolof` Re-alignment

Google introduced `WAXAL`, a new open dataset for 21 African languages, to tackle data scarcity and build inclusive speech technology. However, the Wolof language has experienced alignment issues between the audio files and their transcriptions, making the dataset unusable.

We therefore propose to correct this using a simple and effective approach:

1.For each audio clip, we generated a transcription using Google Gemini ASR.
2.For each generated transcription, we calculated the Levenshtein distance with all the initial transcriptions.
3.The lowest distance obtained indicates the most similar initial transcription to the one generated by the ASR.
4.The index corresponding to this initial transcription is the correct index that will be used to correct the misalignment.

We also identified a couple of corrupted files during the process that could not be read. As part of this filtering process:

—171 samples were removed from the train split
—20 samples were removed from the test split
—22 samples were removed from the validation split

Leaving the final dataset with the following stats:

DatasetDict({
  train: Dataset({
  features: ['id', 'speaker_id', 'locale', 'gender', 'audio', 'text'],
  num_rows: 834
    })
  test: Dataset({
  features: ['id', 'speaker_id', 'locale', 'gender', 'audio', 'text'],
  num_rows: 111
    })
  validation: Dataset({
  features: ['id', 'speaker_id', 'locale', 'gender', 'audio', 'text'],
  num_rows: 97
    })
})

NOTE: Some audio files show a duration of 00:00/00:00 in the HuggingFace player but play properly once loaded into your script.

Dataset duration

Grouping by split:

Split	Duration	Total (seconds)	Nb of samples
Train	411 min 10 s	24 670 s	834
Test	52 min 18 s	3 138 s	111
Validation	39 min 46 s	2 386 s	97
---	---	---
Total	503 min 15 s	30 195 s	1042

Grouping by speaker id:

Split	Speaker ID	Duration (H, M, S)	Nb of samples	Gender
Train	1	1 h 28 min 46 s	150	male
	8	1 h 09 min 33 s	129	female
	5	1 h 07 min 27 s	128	female
	3	1 h 05 min 41 s	171	female
	2	1 h 00 min 49 s	128	female
	4	0 h 58 min 55 s	128	male
---	---	---	---	---
Test	2	0 h 15 min 12 s	20	female
	3	0 h 11 min 20 s	20	female
	5	0 h 10 min 02 s	23	female
	1	0 h 06 min 00 s	19	male
	4	0 h 05 min 52 s	14	male
	8	0 h 03 min 52 s	15	female
---	---	---	---	---
Validation	8	0 h 07 min 57 s	18	female
	2	0 h 07 min 10 s	18	female
	4	0 h 07 min 08 s	15	male
	1	0 h 06 min 40 s	17	male
	3	0 h 05 min 52 s	16	female
	5	0 h 05 min 00 s	13	female

The speakers' genders were missing from the initial dataset and were marked as unknown. To correct this, we started by grouping the audio files by speaker_id, then listened to samples from each speaker to manually determine their gender. We ended up identifying 06 genders: 02 males and 04 females.

Load the dataset

You can download the dataset with the following script:

python

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id        = "galsenai/WaxalNLP",
    repo_type      = "dataset",
    allow_patterns = "data/*.parquet",
    local_dir      = "./waxal_wol"
)

And then load the dataset with the following:

python

from datasets import load_dataset

dataset = load_dataset("parquet", data_files={
    "train": "waxal_wol/data/train-*.parquet",
    "test":  "waxal_wol/data/test-*.parquet",
    "validation": "waxal_wol/data/validation-*.parquet",
})

print(dataset)

The notebook used to make these corrections is available on Google Colab to help you fix similar issues in your language, pending the upcoming fixes planned by the Waxal project team.

This work has been carried out by Derguene, with Abdou Aziz who helped to identify the misalignment issue.