quinnlue/common-voice-17-regmix-webdataset

Common Voice 17 RegMix WebDataset

Public, training-oriented WebDataset conversion of `fsicoli/common_voice_17_0`, pinned to source revision 8262c16bf297c87a9cd88c51997c4758ed7a8ba2.

Layout

Each language is a RegMix cluster at data/<language>/*.tar. Every sample is a pair with the same key:

• —<key>.opus: mono Ogg Opus audio at 24 kbps, preserving the source sample rate (normally 48 kHz)
• —<key>.json: UTF-8 training metadata and the complete original TSV row

The source split is retained in metadata. train, dev, and test are treated as validated; other is used only to backfill a language below 500 hours. invalidated is excluded.

Sampling

The cap is 500 hours per language. Validated speech is selected before other. Within each tier, selection maximizes unique speakers before speaker reuse, then balances the combined age, gender, accent, and variant stratum. Sampling is deterministic (cv17-regmix-v1).

Streaming and training

Download only the language clusters needed for an experiment:

from glob import glob
from huggingface_hub import snapshot_download
import webdataset as wds

root = snapshot_download(
    "quinnlue/common-voice-17-regmix-webdataset",
    repo_type="dataset",
    allow_patterns=["data/en/*.tar"],
)
dataset = wds.WebDataset(sorted(glob(root + "/data/en/*.tar"))).decode().to_tuple("opus", "json")

Per-language summaries and RegMix patterns are under manifest/. The repository is uploaded incrementally and is usable while curation is still running.

Storage model

Only one source file is handled at a time. Each source tar is downloaded with the parallel Xet client into a disposable cache, processed with bounded audio-worker concurrency, and removed after its output shards are uploaded. The full source corpus is never mirrored locally.

License and citation

The source dataset declares CC0-1.0. See the source dataset card for the Common Voice citation and provenance.