Scicom-intl/YouTube-Cantonese-Emilia

YouTube Cantonese — Emilia

2,064,679 speaker-homogeneous Cantonese speech segments — 5,312.6 hours — produced by running `alvanlii/cantonese-youtube` through the Emilia speech-data pipeline (source separation → diarization → VAD segmentation → ASR → MOS filtering).

Each row is one clean, single-speaker segment of 3–30 s with a transcript, a speaker turn label and a DNSMOS quality score. Audio is shipped separately as MP3s inside zip parts, in both an original and a silence-trimmed edition. A derived `permutation` config supplies 1,635,566 same-speaker (reference, target) pairs for voice cloning.

Configs

The viewer shows these tables — transcripts and metadata only. Audio is not embedded in the parquet; it lives in the zip parts and is joined by audio_filename, so there is no inline playback. See Loading the audio.

Files

a and b are the two machines that ran the pipeline. They processed disjoint sets of clips — the two parquets share no id, so concatenating them introduces no duplicates.

Both audio sets use identical arcnames, so one audio_filename resolves in either: pick the untrimmed zips or the trimmed ones, and the metadata rows need no change.

Schema

Example row:

{
  "id": "0000038032",
  "shard": "train-00027-of-01090",
  "segment_index": 0,
  "audio_filename": "train-00027-of-01090/0000038032/0000038032_0.mp3",
  "text": " 我地呢,就嚟咗第七区食一个Cribs嘅甜品。因为呢,我地等间諗住去一个百货公司,都系喺呢一区嚟嘅。…",
  "start": 0.2344375,
  "end": 23.3624375,
  "speaker": "SPEAKER_01",
  "language": "yue",
  "dnsmos": 2.8213813060420665
}

Statistics

Audio format: 24 kHz mono MP3, loudness-normalized, and taken from the separated vocal stem (not the original mix).

Loading

Metadata

from datasets import load_dataset

ds = load_dataset("Scicom-intl/YouTube-Cantonese-Emilia", split="train")
print(ds[0])

Or straight from the parquet, which is faster if you only want to filter:

import pandas as pd

df = pd.read_parquet("hf://datasets/Scicom-intl/YouTube-Cantonese-Emilia/data/part-a.parquet")
df = df[df.dnsmos > 3.2]

Loading the audio

Download the zip parts (152.6 GB — use allow_patterns to take a subset):

from huggingface_hub import snapshot_download

snapshot_download(
    "Scicom-intl/YouTube-Cantonese-Emilia",
    repo_type="dataset",
    local_dir="ycd",
    allow_patterns=["output-audio-a-*.zip"],       # untrimmed, box a only
    # allow_patterns=["output-audio-trim-*.zip"],  # silence-trimmed, both boxes
)

Take one of the two sets — output-audio-*.zip and output-audio-trim-*.zip hold the same arcnames, so downloading both and indexing them together makes the later one win.

Joining audio to metadata

audio_filename is the arcname inside whichever zip part happens to hold it, so build an index once and reuse it:

import glob, zipfile, io
import soundfile as sf

index = {}
handles = {}
for path in glob.glob("ycd/output-audio-*.zip"):
    handles[path] = zipfile.ZipFile(path)
    for name in handles[path].namelist():
        index[name] = path

def read_segment(audio_filename):
    zf = handles[index[audio_filename]]
    return sf.read(io.BytesIO(zf.read(audio_filename)))

wav, sr = read_segment(df.audio_filename.iloc[0])

Silence-trimmed audio

output-audio-trim-*.zip holds a second copy of every segment with its internal silences shortened. Same arcnames, same 24 kHz mono MP3 format — only the samples differ.

The segments are already VAD-cut, so this is a light touch: over a 4,000-file sample the trimmed copy keeps 98.7 % of the original duration on average (median 99.8 %, p10 96.8 %), and 25 % of files come through untouched. The tail is where it earns its keep — the heaviest trim found was 6.83 s → 3.35 s. What it removes is the occasional long pause inside a segment, which is the part that hurts TTS alignment.

Per file: 30 ms frames are labelled by WebRTC VAD (aggressiveness 3) on a 16 kHz peak-normalised copy; runs of same-labelled frames are grouped; then each silence run is shortened — leading silence keeps only its last 0.3 s, trailing silence only its first 0.3 s, and an interior silence of ≥ 0.4 s is cut to 0.2 s from each end. Speech is never touched.

Use the trimmed set for TTS/voice-cloning training where dead air is wasted context; use the untrimmed set when you need timings that line up with start/end, or are doing ASR where the pauses are harmless.

Produced by `trim_silence.py` in the pipeline repo, after malaya-speech.

Voice-cloning pairs (permutation config)

Beyond the segments themselves, the dataset ships (reference, target) pairs — two utterances by the same speaker, for training or evaluating voice cloning.

from datasets import load_dataset

pairs = load_dataset("Scicom-intl/YouTube-Cantonese-Emilia", "permutation", split="train")
pairs[0]
# {'reference_audio': 'train-00027-of-01090/0000038032/0000038032_0.mp3',
#  'reference_text':  '...',
#  'target_audio':    'train-00027-of-01090/0000038032/0000038032_1.mp3',
#  'target_text':     '...'}

reference_audio / target_audio use the same paths as `audio_filename` in the default config, so they resolve against either zip set with no rewriting.

How a pair is made: within one clip, segments are kept only if their transcript passes the quality filters (drops ASR boilerplate, mostly-single-character filler, Whisper repetition loops, and any text with a 3-gram repeated more than three times). Surviving segments are then paired within each diarized speaker, and a pair is emitted only if the two segments' TitaNet-L speaker embeddings have cosine similarity ≥ 0.8 — a guard against diarization having merged two voices under one label.

1,635,566 pairs drawn from 476,368 clips. Most clips contribute none: a (clip, speaker) turn holds at most 7 segments and usually 1–2, and a lone segment cannot form a pair.

permutation_sample applies the same construction but caps each reference at 3 targets. Because that cap almost never binds here, it retains 1,626,542 pairs — 99.4 % of `permutation`. It exists for parity with sibling datasets; for this corpus the two are effectively the same table, so just use permutation.

Pairs are within a single clip, never across clips. Speaker labels are clip-local (see Known limitations), so there is no way to pair the same person across two different videos — and no claim that different clips with the same label are the same speaker.

How it was built

Per source clip, in order:

1. 1.Standardization — 24 kHz, mono, 16-bit, loudness-normalized.
2. 2.Source separation — UVR-MDX-NET (UVR-MDX-NET-Inst_HQ_3) vocal extraction.
3. 3.Speaker diarization — pyannote/speaker-diarization-3.1.
4. 4.Segmentation — Silero VAD, merged and trimmed per speaker to 3–30 s.
5. 5.ASR — WhisperX / faster-whisper large-v3, forced to `yue`. The source is known to be single-language, and per-segment language detection reliably mislabels Cantonese as zh; forcing the label keeps every segment rather than dropping it.
6. 6.Quality filter — DNSMOS OVRL ≥ 2.8, duration 3–30 s, ≥ 2 characters of text, plus a per-clip IQR outlier rejection on seconds-per-character (catches badly aligned segments).
7. 7.Export — one MP3 per surviving segment.

Of the 1,477,757 source clips processed, 1,106,929 (74.9 %) kept at least one segment; the rest were emptied by the quality filter or were unreadable.

Pipeline code: Scicom-AI-Enterprise-Organization/Emilia — a fork of Amphion's Emilia adapted for streaming HF parquet input and multi-GPU sharding.

Known limitations

• —Transcripts are machine-generated and unverified. Base large-v3 normalizes Cantonese toward written/simplified Chinese even when the decode language is forced to yue. Many segments do keep authentic Cantonese morphology (嘅, 咁, 㗎, 喺, 唔), but others read closer to Mandarin. If you need reliable Cantonese orthography, re-transcribe with a Cantonese-finetuned model — or use the source dataset's own transcript_whisper field.
• —Whisper repetition loops survive the filter. A small fraction of segments degenerate into a repeated token (e.g. 再,再,再,…). Filter on character-repetition ratio if this matters to you.
• —`speaker` is clip-local. SPEAKER_00 in two different clips is not the same person. There is no global speaker identity resolution.
• —`dnsmos` has a floor of 2.8 by construction — it is a filter threshold, not a full quality ranking, so the column's dynamic range is narrow.
• —`start`/`end` index the standardized, vocals-separated clip, which shares a timeline with the source clip but not its audio content.
• —Coverage is 99.96 %, not 100 %. 1,477,757 of the source dataset's 1,478,373 clips were processed; 616 clips (0.04 %) were never completed because their worker was killed mid-clip. They are simply absent — no partial rows.

Provenance and licensing

Derived from `alvanlii/cantonese-youtube` (gated), which is itself sourced from YouTube. The underlying recordings remain subject to their original terms; no additional license is granted here, and no license is asserted over the source audio. Review the upstream dataset's terms before redistributing or training on this data. The processing code is Apache-2.0.

Citation

The pipeline:

@inproceedings{emilia,
    author={He, Haorui and Shang, Zengqiang and Wang, Chaoren and Li, Xuyuan and Gu, Yicheng and Hua, Hua and Liu, Liwei and Yang, Chen and Li, Jiaqi and Shi, Peiyang and Wang, Yuancheng and Chen, Kai and Zhang, Pengyuan and Wu, Zhizheng},
    title={Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation},
    booktitle={Proc.~of SLT},
    year={2024}
}