AAdonis/multilingual_audio_alignments

Multilingual MFA-Aligned Speech Dataset

A large-scale multilingual speech dataset with word-level and phoneme-level alignments produced using the Montreal Forced Aligner (MFA).

Dataset Description

This dataset consolidates multiple speech corpora across various languages, all processed through MFA to provide precise phoneme and word alignments. Each sample includes the original audio, transcript, and detailed timing information for both words and phonemes.

Features

Languages & Statistics

Total: ~20,000+ hours (estimated)

Usage

Load a specific language

from datasets import load_dataset

# Load English data
dataset = load_dataset("AAdonis/multilingual_audio_alignments", "english", split="train")

# Load German data
dataset = load_dataset("AAdonis/multilingual_audio_alignments", "german", split="train")

Access alignments

sample = dataset[0]

# Get audio
audio = sample["audio"]["array"]
sample_rate = sample["audio"]["sampling_rate"]

# Get transcript, phonemes, and source
transcript = sample["transcript"]
phonemes = sample["phoneme_sequence"]  # "h ɛ l oʊ w ɜː l d"
source = sample["source"]  # e.g., "voxpopuli"

# Get word-level alignments
for word_info in sample["words"]:
    print(f"{word_info['word']}: {word_info['start']:.2f}s - {word_info['end']:.2f}s")

# Get phoneme-level alignments
for phon_info in sample["phonemes"]:
    print(f"{phon_info['phoneme']}: {phon_info['start']:.3f}s - {phon_info['end']:.3f}s")

Filter by source

# Get only VoxPopuli samples
voxpopuli_samples = dataset.filter(lambda x: x["source"] == "voxpopuli")

# Get only Common Voice samples
cv_samples = dataset.filter(lambda x: x["source"] == "common_voice")

Processing Details

MFA Alignment

All samples were aligned using the Montreal Forced Aligner (MFA) with language-specific acoustic models and pronunciation dictionaries.

Quality Filtering

During processing, samples were filtered and split based on:

• —`<unk>` words: Samples containing unknown words are split at those boundaries
• —`spn` phonemes: Spoken noise markers cause sample splits
• —Duration: Samples are filtered by minimum/maximum duration thresholds
• —Word count: Minimum word requirements per segment

Phoneme Sequence Format

The phoneme_sequence column contains IPA phonemes with:

• —Phonemes within a word are concatenated directly
• —Words are separated by spaces
• —Example: "h ɛ l oʊ" for "hello" (4 phonemes, 1 word)

Citation

If you use this dataset in your research, please cite the following manuscript:

@misc{aasonitis2026multilingualspeechediting,
      title={Multilingual Speech Editing},
      author={Antonis Asonitis and Luca A. Lanzendörfer and Frédéric Berdoz and Roger Wattenhofer},
      year={2026},
      note={Manuscript in preparation},
}

License

This dataset is released under CC-BY-4.0. Please also respect the licenses of the original source datasets and MFA.