holi-lab/SpokenTOD

SpokenTOD

SpokenTOD is an English spoken task-oriented dialogue dataset created with the SpokenTOD augmentation pipeline, and used to train SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue. The pipeline augments task-oriented dialogues with phenomena such as cross-turn slots, barge-in, disfluency, and emotion labels, and synthesizes speech conditioned on reference audio. Speech synthesis uses Qwen3-TTS with its Voice Design then Clone workflow.

Dataset structure

The release uses WebDataset TAR shards so that the complete dataset can be streamed efficiently and displayed by the Dataset Viewer:

data-text-v1/train/train-00000.tar
data-text-v1/validation/validation-00000.tar
data-text-v1/test/test-00000.tar
metadata/train/metadata.jsonl
metadata/validation/metadata.jsonl
metadata/test/metadata.jsonl

Each row contains a playable wav audio value, a json metadata object (original_path, size, and split), and its text transcript. Sample keys inside the raw WebDataset shards are deterministic SHA-256 hashes of the original paths, but those internal keys are not part of the loaded dataset schema.

The original split-level metadata.jsonl files are preserved byte-for-byte under metadata/. They are not part of the Viewer data-file patterns above.

The Dataset Viewer exposes one playable audio utterance per row. Dialogue directory names in original_path identify the source dialogue; they are not classification labels.

Licensing and attribution

SpokenTOD contains data derived from multiple source datasets. Each source dataset remains subject to the license below; the source-specific terms control the portions derived from that dataset. Please retain the required attribution and cite the corresponding source publications when using the data.

Public-release exclusion. SGD-derived samples are not included in this public release because copyright and source-licensing constraints prevent public redistribution of the associated synthesized speech.

SpokenWOZ

SpokenWOZ is distributed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. Use of SpokenTOD records derived from SpokenWOZ is limited to non-commercial use and requires attribution to the SpokenWOZ authors.

EmoWOZ

EmoWOZ is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. Use of SpokenTOD records derived from EmoWOZ is limited to non-commercial use and requires attribution to the EmoWOZ authors.

Schema-Guided Dialogue (SGD)

The Schema-Guided Dialogue dataset is released under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license. Derived portions must preserve attribution and comply with the ShareAlike requirement.

ABCD

The Action-Based Conversations Dataset (ABCD) is released under the MIT License. Please retain the required copyright and license notices for derived portions.

Taskmaster-2

Taskmaster-2 is made available by Google LLC under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. Derived portions require attribution to the Taskmaster-2 authors.

Speech Accent Archive

Reference audio from the Speech Accent Archive (Steven H. Weinberger and Matthew C. Kelley, George Mason University) was used solely as reference audio during speech synthesis. The original Speech Accent Archive recordings, transcripts, and associated metadata are not included in this release. Copyright © The Speech Accent Archive; use is subject to the Archive's official terms and license information.

Please cite the Speech Accent Archive as recommended by its citation guidance, in addition to citing the SpokenTOD project and any relevant original source datasets.

Citation

@misc{lee2026spokenusspokenusersimulator,
  title={SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue},
  author={Jonggeun Lee and Junseong Pyo and Jeongmin Park and Yohan Jo},
  year={2026},
  eprint={2603.16783},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2603.16783}
}

Dataset curators

Jonggeun Lee, Junseong Pyo, Jeongmin Park, and Yohan Jo.