datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
spoken-squad-t2aspoken-multiturn-sft
Spoken Multi-turn SFT Japanese
Japanese spoken multi-turn SFT dataset generated from kanhatakeyama/AutoMultiTurnByCalm3-22B using CosyVoice2 TTS.
Dataset Description
This dataset contains Japanese multi-turn SFT (Supervised Fine-Tuning) data with spoken questions.
q1: First question (text + audio)
a1: First answer (text only)
q2: Follow-up question (text + audio)
a2: Second answer (text only)
Samples
ID
Q1
Q1 Audio
A1
Q2
Q2 Audio
A2
0
鉄は強磁性体ですか？… See the full description on the dataset page: https://huggingface.co/datasets/Atotti/spoken-multiturn-sft.spoken-alpaca-gpt4SpokenNativQA
SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs
The SpokenNativQA dataset consists of question-answer (QA) pairs, where queries are sourced from real users and answers are manually reviewed and edited. The dataset covers a diverse range of 18 topics that reflect culturally and regionally specific knowledge, as well as everyday queries. These topics include animals, business, clothing, education, events, food and drinks, general knowledge, geography, immigration… See the full description on the dataset page: https://huggingface.co/datasets/QCRI/SpokenNativQA.SpokenWOZ-Train-Text
What is SpokenWOZ?
SpokenWOZ is a large-scale multi-domain speech-text dataset for spoken task-oriented dialogue modeling, which consists of 203k turns, 5.7k dialogues and 249 hours audios from realistic human-to-human spoken conversations.
Why SpokenWOZ?
The majority of existing TOD datasets are constructed via writing or paraphrasing from annotators rather than being collected from realistic spoken conversations. The written TDO datasets may not be representative of the… See the full description on the dataset page: https://huggingface.co/datasets/ssz1111/SpokenWOZ-Train-Text.spoken_squad_testThis dataset is licensed under the terms of the CC-BY-SA-4.0 license.
https://github.com/Chia-Hsuan-Lee/Spoken-SQuAD/blob/master/LICENSE.md
Author: @michaellee886
@article{li2018spoken,
title={Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension},
author={Li, Chia-Hsuan and Wu, Szu-Lin and Liu, Chi-Liang and Lee, Hung-yi},
journal={arXiv preprint arXiv:1804.00320},
year={2018}
}
@article{wang2024audiobench,
title={AudioBench: A… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/spoken_squad_test.SpokenSwag
SpokenSwag
We present here SpokenSwag as described in the paper "Slamming: Training a Speech Language Model on One GPU in a Day".
This dataset is based on allenai/swag and synthetised with 4 speakers from hexgrad/Kokoro-82M.
We show that perfoming DPO over the dataset can really improve performance of Speech Language Models.
We encourage you to also see the following resources, for further information:
Project Page: https://pages.cs.huji.ac.il/adiyoss-lab/slamming/ Paper:… See the full description on the dataset page: https://huggingface.co/datasets/slprl/SpokenSwag.spoken_norm_pattern
Vietnamese Inverse Text Normalization
Inverse text normalization (ITN) is the task that transforms spoken to written styles. It is particularly useful in automatic speech recognition (ASR) systems where proper names are often miss-recognized by their pronunciations instead of the written forms. By applying ITN, we can improve the readability of the ASR system’s output significantly. This dataset provides data for doing ITN task in the Vietnamese language.
For example:
Spoken… See the full description on the dataset page: https://huggingface.co/datasets/nguyenvulebinh/spoken_norm_pattern.spoken-multihop-rag
Spoken Multi-hop QA: ASR Transcripts Across Four English Accents
ASR transcriptions of 3,000 multi-hop QA questions, each spoken in four
English accents and transcribed with Whisper-large-v3. Released as the
data companion to Better Retrieval, Worse Robustness: How Multi-hop RAG
Amplifies Upstream ASR Errors
(EMNLP 2026, Main Conference).
The dataset exists to make one thing cheap to study: what happens to a
retrieval pipeline when its query arrives through ASR rather than as… See the full description on the dataset page: https://huggingface.co/datasets/orcarouter/spoken-multihop-rag.Neapolitan-Spoken-Corpus
Neapolitan Spoken Corpus (NSC)
A corpus of read Neapolitan speech for ASR evaluation, with a validated
Neapolitan–Italian lexicon, LOSO fine-tuning splits, trained LoRA adapters,
metric implementations, per-clip results, and error annotations.
This release supersedes the earlier 141-clip single-speaker version of this
repository. The earlier release corresponds to Speaker S1 of the present
corpus; the old audioData/ and transcripts.csv are replaced by
data/audio/ and… See the full description on the dataset page: https://huggingface.co/datasets/michaelcacioli/Neapolitan-Spoken-Corpus.SpokenWOZ-Train-Audiospoken-magpie-ja
Spoken-magpie
LLMの日本語Instruction Tuning用データllm-jp/magpie-sft-v1.0をCosyVoice2 TTSを使用して音声化した商用利用可能な日本語の音声言語モデルのSFT用データセットです。
ある程度の話者多様性を持つように生成されています。
Respone Audioは500文字以下の場合にのみ生成されています。
NVIDIA H200を10枚を使用しvllmで推論しました。
Samples
最初の50サンプルを掲載します。
ID
Instruction
Instruction Audio
Response
Response Audio
0
カボチャを使ったスイーツのレシピをいくつか教えてください。
もちろんです、カボチャを使ったスイーツは秋にぴったりですね。以下にいくつかのレシピをご紹介します。1. カボチャのスフレパウンドケーキ- 材料：カボチャ 200g、生クリーム 50ml、牛乳 50ml、卵 3個、砂糖 100g、薄力粉 70g、バニラエッセンス 少々-… See the full description on the dataset page: https://huggingface.co/datasets/Atotti/spoken-magpie-ja.SpokenTermDetection_Tedlium2Train
Dataset Card for "SpokenTermDetection_Tedlium2Train"
More Information needed
spokenwoz_dst
Dataset: spokenwoz_whisper_dst
Short description
Prepared SpokenWoZ training dataset adapted for Whisper-style DST (dialog state tracking) tasks. Each example contains an utterance, normalized text, and aligned audio (16 kHz).
Key metadata
Number of examples: 73,950
Total size on disk: ~9.08 GB
Splits: train, validation
validation examples: 7,284
Features:
text (string): original utterance text
normalized_text (string): normalized form of the… See the full description on the dataset page: https://huggingface.co/datasets/vendrkat/spokenwoz_dst.spoken_norm_assignment
VietAI assignment: Vietnamese Inverse Text Normalization dataset
Dataset Description
Inverse text normalization (ITN) is the task that transforms spoken to written styles. It is particularly useful in automatic speech recognition (ASR) systems where proper names are often miss-recognized by their pronunciations instead of the written forms. By applying ITN, we can improve the readability of the ASR system’s output significantly. This dataset provides data for doing ITN… See the full description on the dataset page: https://huggingface.co/datasets/VietAI/spoken_norm_assignment.spokenwoz-whisper
SpokenWOZ With Whisper Transcripts
This is a custom version of the SpokenWOZ dataset with all the original transcripts replaced by transcripts generated by Whisper-large-v3. We manually
transcribed 1000 utterances from the test set and like this estimate the WER of the original ASR to be ~29% and the WER of the Whisper transcripts to
be ~5.3%.
Citation Information
If you happen to use these transcripts for your work, we will be glad if you cite the following works:… See the full description on the dataset page: https://huggingface.co/datasets/pirxus/spokenwoz-whisper.The_Spoken_Wikipedia_Corpora_Dutch_ASR_Hiidden
Dataset Card for "The_Spoken_Wikipedia_Corpora_Dutch_ASR_Hiidden"
Homepage: Spoken Wikipedia CorporaRepository: Bitbucket RepositoryPaper: Publication at LREC 2016Leaderboard: Interspeech 2018 PaperPoint of Contact: nats@nats.gitlab.io
Dataset Summary
The Spoken Wikipedia Corpora (SWC) is a collection of aligned spoken Wikipedia articles including articles in Dutch. It includes approximately 210 hours of audio, transcriptions, and metadata. The corpus is licensed… See the full description on the dataset page: https://huggingface.co/datasets/Jaspernl/The_Spoken_Wikipedia_Corpora_Dutch_ASR_Hiidden.chatbot-arena-spoken-voicesspoken-dpo-49k
Spoken DPO Japanese
Japanese spoken DPO dataset generated from ryota39/dpo-49k using CosyVoice2 TTS.
Dataset Description
This dataset contains Japanese DPO (Direct Preference Optimization) data with spoken prompts.
prompt: Multi-turn conversation (human turns have audio)
chosen: Preferred response (text only)
rejected: Non-preferred response (text only)
Samples
ID
Conversation
Chosen
Rejected
0
Human: ドイツに社会問題があるのはなぜだと思いますか？
ドイツは高度に工業化された国なので、公害が多い。… See the full description on the dataset page: https://huggingface.co/datasets/Atotti/spoken-dpo-49k.ghomala-spoken-bible
Ghomálá' Spoken New Testament — aligned audio + trilingual text
Part of the Lingo / NativeAI language-preservation project. This is
~20 hours of spoken Ghomálá' (Ghomala, ISO bbj; a Grassfields Bantu language of
West Cameroon) — recorded readings of the New Testament — aligned chapter-by-chapter
with parallel text in Ghomálá', French, and English.
Spoken-language data is exactly what oral-first Cameroonian languages lack, which makes
this a rare resource for building ASR, TTS… See the full description on the dataset page: https://huggingface.co/datasets/flagship-ai/ghomala-spoken-bible.myanmar_spoken_corpus
Credits and Acknowledgments
This dataset is built upon the foundational work of the Myanmar Spoken Corpus by freococo (Wynn).
Original Dataset Source: freococo/myanmar_spoken_corpus
Modifications: * Curated and filtered for specific training needs of the ShweYon model.
Integrated with 3% English subset for bilingual proficiency maintenance.
Re-formatted into 36 shards for optimized Continued Pre-training (CPT).
We are deeply grateful to freococo for their contribution to the… See the full description on the dataset page: https://huggingface.co/datasets/URajinda/myanmar_spoken_corpus.spoken_squad
Dataset Card for Spoken-SQuAD
Citation
@article{lee2018spoken,
title={Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension},
author={Lee, Chia-Hsuan and Wu, Szu-Lin and Liu, Chi-Liang and Lee, Hung-yi},
journal={Proc. Interspeech 2018},
pages={3459--3463},
year={2018}
}
SpokenCOCO-A2IT
SpokenCOCO Audio-to-(Image+Text) Retrieval
MTEB/MOEB task where queries are spoken audio captions and corpus items
contain both a MSCOCO image and its written text caption.
Task
Given a spoken audio description of an image, retrieve the correct
(image, text) pair from the corpus. Only models that can process all three
modalities — audio, image, and text — can exploit the full corpus signal.
Contents
Queries: 25031 spoken audio captions (WAV… See the full description on the dataset page: https://huggingface.co/datasets/rakshi719/SpokenCOCO-A2IT.SpokenWikipedia-retrieval
Spoken Wikipedia speech-text retrieval (MTEB)
Volunteer readings of Wikipedia articles paired with the article lead, in Dutch,
English, German, Spanish and French.
Recordings come from Wikimedia Commons, which is free by site policy, and the lead
text from each Wikipedia, which is CC-BY-SA. The set is published as cc-by-sa-4.0.
Only the first 60 seconds of each reading is kept, since readers start
at the lead. One recording per article.
Built by… See the full description on the dataset page: https://huggingface.co/datasets/vnahata/SpokenWikipedia-retrieval.spoken-mqa@article{wei2025towards,
title={Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems},
author={Wei, Chengwei and Wang, Bin and Kim, Jung-jae and Chen, Nancy F},
journal={arXiv preprint arXiv:2505.15000},
year={2025}
}
MADIS5-spoken-arabic-dialects
Dataset Overview
MADIS-5 (Multi-domain Arabic Dialect Identification in Speech) is a manually curated dataset
designed to facilitate evaluation of cross-domain robustness of Arabic Dialect Identification (ADI) systems.
This dataset provides a comprehensive benchmark for testing out-of-domain generalization across different speech domains with diverse recording conditions and speaking styles.
Dataset Statistics
Total Duration: ~12 hours of speech
Total… See the full description on the dataset page: https://huggingface.co/datasets/badrex/MADIS5-spoken-arabic-dialects.SpokenTermDetection_LibrispeechTrainClean100
Dataset Card for "speechTermDetection_LibrispeechTrainClean100"
More Information needed
SpokenCOCOA2IRetrieval
SpokenCOCOA2IRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
SpokenCOCO pairs MS COCO images with recordings of human speakers reading the corresponding English captions. This task uses the 5,000-image Karpathy test split with 25,031 spoken captions. Queries are spoken captions and the corpus contains images; the goal is to retrieve the image described by each recording.
Task category
Any2AnyRetrieval (audio-to-image)
Domains
Scene, Spoken
Reference… See the full description on the dataset page: https://huggingface.co/datasets/whybe-choi/SpokenCOCOA2IRetrieval.spoken-squadspoken-alpaca-gpt4