humanify/Env-TTS-SD-Corpus

env-tts-sd-corpus

Environment-aware text-to-speech training corpus. Each row pairs three short 16 kHz mono FLAC clips with a transcript:

• —an environment sample (different speaker, same acoustic scene),
• —a speaker reference (same speaker, optionally with augmented acoustics),
• —the target speech,

so a TTS model can learn to synthesise a target utterance with both a specified voice and a specified environment.

Schema

Source corpora

Processing pipeline

The pipeline is three streaming stages running in parallel as separate processes, with a small filesystem-based handoff for backpressure:

1. 1.download — one thread per source. Hugging Face mirrors and direct tar URLs are streamed with httpx.stream; the tar bytes are never written to disk in full. Each upstream "conversation" emits a JSON sentinel under state/ready/ as soon as its audio is locally addressable.
2. 2.process — an asyncio loop drains sentinels with a bounded semaphore (default 64 concurrent conversations). For each conversation it
3. 3.resamples the audio to 16 kHz mono,
4. 4.walks the diarisation turns, chunks each turn into 3–15 s pieces,
5. 5.picks a same-speaker reference (≥3 s, concatenating short turns when needed) and a different-speaker environment slice (≥3 s, extended into surrounding audio if necessary),
6. 6.submits any speech pieces whose transcript is missing or whose turn was split mid-utterance to a Qwen3-ASR-1.7B Flask worker for fresh ASR,
7. 7.FLAC-encodes the three clips and appends a row to the sharded parquet writer.

The ASR worker coalesces concurrent requests into length-bucketed batches ([0–4 s], [4–9 s], [9–16 s], 16 s+) so that the HuggingFace padding=True step inside qwen-asr does not waste GPU on long zero-pad tails. Single-clip OOMs are dropped silently (the row is dropped, not the sibling 255 clips).

1. 1.upload — watches final/upload_queue/group_*/ for sealed groups and uploads them to this repo via HfApi.upload_folder. Each group bundles ≈3 200 rows (4 parquet shards × 800 rows). Commits are rate-limited.

The reader, ASR worker, augmenter, and writer are all designed to recover cleanly from SIGKILL: all state is captured in a few small JSON files under state/ and an atomic-rename .tmp → final write protocol for each parquet shard.

Provenance: row breakdown by dataset

Counts are approximate (depend on streaming end + final partial groups).

Drop reasons that account for "records ingested" > "records emitted with rows" in MSDWILD/M3SD: conversations with <2 speakers (no candidate for the env source), conversations whose total speech time per speaker can't yield a ≥3 s focal clip + a ≥3 s reference, or clips where Qwen3-ASR returned empty text after segmentation.

Licensing

The derived corpus is released under CC-BY-SA-4.0, which inherits the most-restrictive licence among the four sources. Note in particular:

• —M3SD is for academic and non-commercial research only (Wu et al., 2025).
• —MSDWILD uses the X-LANCE research-only agreement (Liu et al., 2022).
• —AISHELL-4 (Apache 2.0) and CHiME-6 (CC-BY-SA-4.0) are open.

If you redistribute audio extracted from this dataset, you must comply with M3SD's and MSDWILD's non-commercial restriction.

Citation

If you use this corpus, please cite the four source papers:

@article{wu2025m3sd,
  title={M3SD: Multi-modal, Multi-scenario and Multi-language Speaker
         Diarization Dataset},
  author={Wu, Shilong and others},
  journal={arXiv preprint arXiv:2506.14427},
  year={2025}
}

@inproceedings{fu2021aishell4,
  title={AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation,
         Recognition and Speaker Diarization in Conference Scenario},
  author={Fu, Yihui and others},
  booktitle={Interspeech},
  year={2021}
}

@inproceedings{liu2022msdwild,
  title={MSDWILD: Multi-modal Speaker Diarization Dataset in the Wild},
  author={Liu, Tao and others},
  booktitle={Interspeech},
  year={2022}
}

@inproceedings{watanabe2020chime6,
  title={CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for
         Unsegmented Recordings},
  author={Watanabe, Shinji and others},
  booktitle={CHiME Workshop},
  year={2020}
}

ASR re-labelling was performed with Qwen3-ASR-1.7B.

Loading

from datasets import load_dataset

ds = load_dataset("ChristianYang/env-tts-sd-corpus", split="train", streaming=True)
row = next(iter(ds))
print(row["text"])
print(row["speech"]["sampling_rate"], len(row["speech"]["array"]))

The audio columns are typed as the HF Audio feature (16 kHz, mono), so they decode automatically on access.

Files on disk

data/
  group_00000/
    manifest.json
    data_000000.parquet
    data_000001.parquet
    data_000002.parquet
    data_000003.parquet
  group_00001/
    ...

Each group is one atomic HF commit. Each parquet shard is ≈800 rows; group size is 4 × 800 = 3 200 rows ≈ 250 MB (snappy-compressed, audio columns already FLAC).

Source pipeline: <https://github.com/...> (see the linked repository for the streaming download/process/upload code that produced this dataset).