Datasetpublic

KRAFTON/Raon-OpenTTS-Eval

Raon-OpenTTS-Eval Technical Report A robustness-oriented evaluation benchmark for zero-shot text-to-speech, covering 4 acoustic regimes (Clean, Noisy, Wild, Expressive) across 12 datasets with 6,000 prompt–text pairs. Existing zero-shot TTS benchmarks typically evaluate models using prompts drawn from a single read-speech dataset, providing an incomplete view of robustness under realistic and challenging recording scenarios. Raon-OpenTTS-Eval… See the full description on the dataset page: https://huggingface.co/datasets/KRAFTON/Raon-OpenTTS-Eval.

sourceHugging Facecc-by-nc-nd-4.0updated 4mo agoView on Hugging Face

9likes1.9kdownloads

Dataset Card

Raon-OpenTTS-Eval

<p align="center"> <a href="https://arxiv.org/abs/2605.20830">Technical Report</a> </p>

A robustness-oriented evaluation benchmark for zero-shot text-to-speech, covering 4 acoustic regimes (Clean, Noisy, Wild, Expressive) across 12 datasets with 6,000 prompt–text pairs.

Existing zero-shot TTS benchmarks typically evaluate models using prompts drawn from a single read-speech dataset, providing an incomplete view of robustness under realistic and challenging recording scenarios. Raon-OpenTTS-Eval addresses this by sampling prompts from diverse real-world conditions, enabling systematic analysis of TTS robustness across controlled, noisy, conversational, and expressive speech.

Dataset Structure

Raon-OpenTTS-Eval/
├── clean/
│   ├── metadata.csv      # 2,500 pairs
│   └── audio/            # reference (prompt) WAV files
├── noisy/
│   ├── metadata.csv      # 1,000 pairs
│   └── audio/
├── wild/
│   ├── metadata.csv      # 1,000 pairs
│   └── audio/
└── expressive/
    ├── metadata.csv      # 1,500 pairs
    └── audio/

Each metadata.csv has the following columns:

Column	Description
`category`	Acoustic regime (CLEAN / NOISY / WILD / EXPRESSIVE)
`source`	Source dataset name
`ref_id`	Prompt utterance ID
`ref_dur`	Prompt duration (seconds)
`ref_text`	Prompt transcription (condition for zero-shot TTS)
`gen_id`	Target utterance ID (used to name generated wav)
`gen_dur`	Target duration (seconds)
`gen_text`	Target text to synthesize
`ref_audio`	Relative path to prompt WAV (`audio/{filename}.wav`)

Construction

For each source dataset, 500 utterances are selected as speech prompts via stratified sampling by speaker metadata (emotion, dialect, speaking style) to ensure representative coverage. Each prompt is paired with a target text drawn from a disjoint utterance in the same dataset, resulting in cross-sentence pairs.

For AMI-SDM, a substantial number of segments contain noisy or misaligned transcriptions due to distant microphone recording conditions. To ensure reliable evaluation, only segments with zero WER (as estimated by Whisper) are retained before sampling, filtering out samples with severe transcription mismatches.

Quick Start: Evaluation

1. Install dependencies

bash

pip install faster-whisper whisper-normalizer jiwer torchaudio soundfile torch

2. Download the WavLM speaker verification checkpoint

The SIM metric uses an ECAPA-TDNN model with WavLM-large features, finetuned for speaker verification. Download the checkpoint from UniSpeech:

bash

# Direct download
wget https://github.com/microsoft/UniSpeech/releases/download/v1.0.0/wavlm_large_finetune.pth

ecapa_tdnn.py (included in this repository) must be in the same directory as eval_raon_tts.py when running evaluation.

3. Generate audio

For each row in metadata.csv, synthesize gen_text conditioned on the prompt audio at ref_audio. Save the output as {gen_id}.wav in a flat directory.

python

for row in metadata:
    wav = your_tts_model.synthesize(
        text=row["gen_text"],
        prompt_audio=f"{split_dir}/{row['ref_audio']}",
        prompt_text=row["ref_text"],
    )
    save_wav(wav, f"{output_dir}/{row['gen_id']}.wav")

4. Run evaluation

bash

python eval_raon_tts.py \
    --gen_dir /path/to/generated_wavs \
    --dataset_dir /path/to/Raon-OpenTTS-Eval \
    --wavlm_ckpt /path/to/wavlm_large_finetune.pth

--gen_dir accepts two layouts:

Layout	Expected structure
Flat	`gen_dir/{gen_id}.wav`
Per-split	`gen_dir/{split}/wavs/{gen_id}.wav`

Split names recognized: clean / raon-clean, noisy / raon-noisy, wild / raon-wild, expressive / raon-emo.

5. Output

RESULTS SUMMARY
==================================================
  clean        WER=0.0199  SIM=0.6793
  noisy        WER=0.0341  SIM=0.6969
  wild         WER=0.0641  SIM=0.6017
  expressive   WER=0.0117  SIM=0.6020
  overall      WER=0.0300  SIM=0.6505
==================================================
Results saved to: /path/to/generated_wavs/raon_eval_results.json

Metrics:

—WER — Word Error Rate computed by transcribing generated audio with Whisper-large-v3 and normalizing via EnglishTextNormalizer (avoids penalizing surface-form variants such as numeric expressions or hyphenated compounds)
—SIM — Cosine speaker similarity between generated and prompt audio using WavLM-large finetuned for speaker verification

Baseline Results

Zero-shot TTS models evaluated under the protocol above. WER (%) via Whisper-large-v3 (normalized); SIM via WavLM-large. Overall is computed over all evaluation samples across the four categories. Bold marks the best result and the Raon-OpenTTS rows. Numbers are from the technical report.

Model	Clean WER ↓	Clean SIM ↑	Noisy WER ↓	Noisy SIM ↑	Wild WER ↓	Wild SIM ↑	Expressive WER ↓	Expressive SIM ↑	Overall WER ↓	Overall SIM ↑
F5-TTS	2.17	0.613	3.82	0.640	136.03	0.324	3.46	0.503	25.08	0.542
MaskGCT	3.39	0.672	5.56	0.727	28.00	0.581	6.44	0.546	8.61	0.635
CosyVoice 2	2.59	0.642	4.39	0.675	49.73	0.535	3.66	0.536	11.02	0.603
CosyVoice 3	2.53	0.678	3.69	0.720	8.31	0.618	5.49	0.567	4.43	0.647
VoxCPM	2.24	0.686	3.42	0.738	43.83	0.553	2.66	0.565	9.48	0.642
Qwen3-TTS	3.38	0.684	4.60	0.726	79.14	0.528	5.81	0.527	17.59	0.626
Raon-OpenTTS-0.3B	1.57	0.645	4.03	0.700	5.83	0.571	2.53	0.570	2.93	0.623
Raon-OpenTTS-1B	1.44	0.718	3.51	0.769	5.61	0.656	2.77	0.633	2.81	0.695

Splits

CLEAN (2,500 pairs)

Controlled read speech from studio and clean recording conditions.

Source	Pairs	License
LibriSpeech-clean	500	CC BY 4.0
ST American English	500	CC BY-NC-ND 4.0
CMU-Arctic	500	BSD
L2-ARCTIC	500	CC BY-NC 4.0
VCTK	500	CC BY 4.0

NOISY (1,000 pairs)

Read and prompted speech in the presence of background noise or reverberation.

Source	Pairs	License
LibriSpeech-other	500	CC BY 4.0
TED-LIUM 3	500	CC BY-NC-ND 4.0

WILD (1,000 pairs)

Unscripted conversational speech from real-world meetings captured under natural conditions. AMI-SDM samples are filtered to WER=0 to ensure transcription reliability.

Source	Pairs	License
AMI-IHM	500	CC BY 4.0
AMI-SDM	500	CC BY 4.0

EXPRESSIVE (1,500 pairs)

Expressive speech covering a wide range of emotions and prosodic styles.

Source	Pairs	License
CREMA-D	500	ODbL 1.0
EmoV-DB	500	Non-commercial research
Expresso	500	CC BY-NC 4.0

Licenses

This dataset is a compilation of audio excerpts from multiple sources, each retaining its original license. The overall dataset is released under CC BY-NC-ND 4.0 (the most restrictive license among the included sources). See the per-source table above for individual licenses.

License	Sources
CC BY 4.0	LibriSpeech-clean, LibriSpeech-other, VCTK, AMI-IHM, AMI-SDM
CC BY-NC 4.0	L2-ARCTIC, Expresso
CC BY-NC-ND 4.0	ST American English, TED-LIUM 3
BSD	CMU-Arctic
ODbL 1.0	CREMA-D
Non-commercial research	EmoV-DB

Citation

bibtex

@article{kim2026raonopentts,
  title     = {Raon-OpenTTS: Open Models and Data for Robust Text-to-Speech},
  author    = {Kim, Semin and Chung, Seungjun and Moon, Taehong and Lee, Sangheon and Ahn, Minyoung and Lee, Keon and Kim, Nam Soo and Cho, Jaewoong and Schmidt, Ludwig and Lee, Kangwook and Park, Dongmin},
  journal   = {arXiv preprint arXiv:2605.20830},
  year      = {2026},
  url       = {https://arxiv.org/abs/2605.20830}
}