datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
instruction-speech-encodec-v1
Dataset Card for "Instruction Speech"
The largest open-source English speech instruction to text answer dataset
Dataset Overview
This dataset contains nearly 450,000 English speech instruction to text answer samples, using:
A subset of OpenHermes 2.5 with user's prompt length less than 64.
Audio generation using WhisperSpeech.
Tokenized using Encodec.
Usage
from datasets import load_dataset, Audio
# Load Instruction Speech dataset
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/Menlo/instruction-speech-encodec-v1.Speech-InstructionsZeroshot-Audio-Classification-Instructions
Zeroshot-Audio-Classification-Instructions
Convert audio classification dataset into zero-shot format speech instructions, support both single label and multi-label,
VGGSound
FSD50k
Nonspeech7k
urbansound8K
VocalSound
Emotion
Gender
ESD Emotion
Age
Language
TAU Urban Acoustic Scenes 2022
CochlScene
BirdCLEF_2021
EmoBox
AudioSet
We also converted huge WAV files into MP3 16k sample rate to reduce storage size.To prevent leakage, please do not include test set in training session.… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Zeroshot-Audio-Classification-Instructions.InstructTTSEval
InstructTTSEval
InstructTTSEval is a comprehensive benchmark designed to evaluate Text-to-Speech (TTS) systems' ability to follow complex natural-language style instructions. The dataset provides a hierarchical evaluation framework with three progressively challenging tasks that test both low-level acoustic control and high-level style generalization capabilities.
Github Repository: https://github.com/KexinHUANG19/InstructTTSEval
Paper: InstructTTSEval: Benchmarking Complex… See the full description on the dataset page: https://huggingface.co/datasets/CaasiHUANG/InstructTTSEval.Malaysian-Speech-Instructions
Malaysian Speech Instructions
We filter Malaysian short prompts that suitable to convert to voice prompt and generate synthetic voice using mesolitica/Malaysian-F5-TTS-v2 also verified with Force Alignment to make sure the pronunciations almost correct.
We follow chat template from Qwen/Qwen2-Audio-7B-Instruct.
It contains dataset from the following sets:
Malaysian-Instructions
Critics Malaysian
Malaysian General QA
Malaysian Ultrachat
Malaysian Alpaca
Multi-turn Stupid questions… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Speech-Instructions.infinity-instruct-90k-speech
Audio Instruct Dataset
audio instruct dataset, synthesized with (Silero tts)[https://github.com/snakers4/silero-models]
InstructS2S-200Kinstruction-speech-encodec-v1.5
Dataset Card for "Instruction Speech"
The largest open-source English speech instruction to text answer dataset
Dataset Overview
This dataset contains over 332,000 English speech instruction to text answer samples, using:
A subset of jan-hq/prompt-voice-v1.5.
Audio generation using WhisperSpeech.
Tokenized using Encodec.
Usage
from datasets import load_dataset, Audio
# Load Instruction Speech dataset
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/Menlo/instruction-speech-encodec-v1.5.AudioSet-Audio-Instructions
AudioSet-Audio-Instructions
Convert AudioSet up to 527 audio labels to Speech Instruction dataset.
For Speech, we transcribe first using Whisper Large V3 after that use the transcription with the label to generate the synthetic instructions.
Classification-Speech-Instructions
Classification Speech Instructions
Speech instructions for emotion, gender, age and language audio classification.
Source code
Source code at https://github.com/mesolitica/malaysian-dataset/tree/master/llm-instruction/speech-classification-instructions
Animal-Sound-Instructions
Animal Sound Instructions
We gathered from,
Birds, birdclef-2021
Insecta, christopher/birdclef-2025
Amphibia, christopher/birdclef-2025
Mammalia, christopher/birdclef-2025
We use Qwen/Qwen2.5-72B-Instruct to generate the answers based on the metadata.
how to prepare the dataset
huggingface-cli download \
mesolitica/Animal-Sound-Instructions \
--include "*.zip" \
--repo-type "dataset" \
--local-dir './'
wget… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Animal-Sound-Instructions.instructtts-three-model-gemini-zh
InstructTTSEval 三模型 Gemini 评测数据
本目录整理了 InstructTTSEval 中文集上三个 TTS 模型的生成音频和 Gemini 一致性评测结果：Qwen3-TTS-12Hz-1.7B-VoiceDesign、Seed-Audio-1.0、VoxCPM2。
字段
records.jsonl 每行对应一个模型和一种控制格式（APS、DSD 或 RP）：
id：InstructTTSEval 样本 ID
mode：控制格式
model、model_name：模型标识
text：合成文本
instruction：历史评测记录中的输入控制指令，按本行 APS/DSD/RP 格式保留；不等同于各模型 API 的完整请求封装
generated_audio：该模型生成音频的相对路径
reference_audio：原始参考音频的相对路径
gemini_consistent：Gemini judge 的一致性判断
inconsistency_reason：判断为不一致时的原因… See the full description on the dataset page: https://huggingface.co/datasets/zsy814/instructtts-three-model-gemini-zh.libri960h_instructCoVoST2-Instructions
CoVoST2 Instruction
Originally from https://huggingface.co/datasets/facebook/covost2, we converted to speech instruction format. We also provide test split.
We highly recommend to not include test set in training set to prevent contamination. Test set supposely to become a speech translation benchmark.
how to prepare the dataset
huggingface-cli download \
mesolitica/CoVoST2-Instructions \
--include "*.zip" \
--repo-type "dataset" \
--local-dir './'
wget… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/CoVoST2-Instructions.Speech-Translation-Instructions
Speech-Translation-Instructions
The instructions translated from 120 languages Common Voice to english, arabic, japanese, mandarin and french from common voice speech dataset. Suitable to use to finetune Speech LLM.
Afrivoice_Swahili-Voice_Instruct_FormatCantonese-Radio-Description-Instructions
Cantonese-Radio-Description-Instructions
Originally from alvanlii/cantonese-radio, we use Qwen/Qwen2.5-72B-Instruct to generate description based on the transcription.
how to prepare the dataset
huggingface-cli download \
mesolitica/Cantonese-Radio-Description-Instructions \
--include '*.zip' \
--repo-type "dataset" \
--local-dir './'
wget https://gist.githubusercontent.com/huseinzol05/2e26de4f3b29d99e993b349864ab6c10/raw/9b2251f3ff958770215d70c8d82d311f82791b78/unzip.py… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Cantonese-Radio-Description-Instructions.multimodal-expert-instruction-samples
Multimodal Expert Instruction Samples - Musical Instrument Lessons with Channel-separated Audio and Video
A music teacher and a student work through two one-on-one lessons: both voices and both instruments on separate tracks, the student on camera, with the lesson plans, the instructions given to each side and both sides' post-lesson ratings alongside.
▶ Watch the lessons · See Peer Collaboration samples · Discuss the full collection
Sister collection: Peer Collaboration… See the full description on the dataset page: https://huggingface.co/datasets/fluid-concepts/multimodal-expert-instruction-samples.openhermes_instruction_test@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and Lin, Geyu and Sun, Shuo and Liu, Zhuohan and Zhang, Wenyu and Liu, Zhengyuan and Aw, AiTi and Chen, Nancy F},
journal={NAACL},
year={2025}
}
Speaker-Diarization-Instructions
Speaker-Diarization-Instructions
Convert diarization dataset from https://huggingface.co/diarizers-community into speech instructions dataset and chunk max to 30 seconds because most of speech encoder use for LLM come from Whisper Encoder.
We highly recommend to not include AMI test set from both AMI-IHM and AMI-SDM in training set to prevent contamination. This dataset supposely to become a speech diarization benchmark.
how to prepare the dataset
huggingface-cli… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Speaker-Diarization-Instructions.Audio-Adversarial-Instructions
Audio-Adversarial-Instructions
We convert to audio boolean classifier speech instructions, randomly 50% chance for positive and other 50% chance for negative.
Data source
AudioSet from mesolitica/AudioSet-Audio-Instructions
Classification Speech Instruction from mesolitica/Classification-Speech-Instructions
EmoBox from mesolitica/Classification-Speech-Instructions
BirdCLEF_2021
how to prepare the dataset
huggingface-cli download \… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Audio-Adversarial-Instructions.llama-omni-speech-instruct
Llama3.2 Omni Speech Instruct Dataset
This dataset is created for the sole purpose of enhancing the LLM capability to become multi-modals. This dataset has speech instruction
that a model could use to learn and produce the output thus allowing the model to overcome only text input and extends it capabilities
towards processing speech command as well.
Dataset Details
Dataset Description
This dataset can be used to train an LLM model to allow adaptibility in… See the full description on the dataset page: https://huggingface.co/datasets/gruhit-patel/llama-omni-speech-instruct.CompA-R-Instructions
CompA-R-Instructions
Originally from https://huggingface.co/papers/2406.11768, we downloaded from Google Drive and converted to HuggingFace.
Ficbook-Audio-Instruct-10K
Ficbook Audio Instruct 10K
Synthetic audio instruction dataset for training Russian audio-language models.
Contains ~10K samples of fiction text voiced with OpenAI TTS and paired with diverse instruction tasks.
Dataset Description
This dataset was created for training and evaluating audio-language models on Russian fiction content.
Each sample contains:
Audio: Fiction text voiced using OpenAI's gpt-4o-mini-tts model
Text: Original text from ficbook stories
Question:… See the full description on the dataset page: https://huggingface.co/datasets/Vikhrmodels/Ficbook-Audio-Instruct-10K.cosyvoice-instructMalaysian-UltraChat-Speech-Multiturn-Instructions
Malaysian-UltraChat-Speech-Multiturn-Instructions
We filter Malaysian short user questions from mesolitica/malaysian-ultrachat that suitable to convert to voice prompt and generate synthetic voice using mesolitica/Malaysian-Dia-1.6B also verified with Force Alignment to make sure the pronunciations almost correct.
A conversation must at least have 2 audio. We follow chat template from Qwen/Qwen2-Audio-7B-Instruct.
how to prepare the dataset
huggingface-cli download \… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-UltraChat-Speech-Multiturn-Instructions.AR-Omni-Instruct-v0.1
AR-Omni-Instruct
Overview
AR-Omni-Instruct is a multimodal instruction-tuning dataset for training unified autoregressive any-to-any models.
All modalities are represented as discrete tokens in a single interleaved token stream, enabling standard next-token prediction training over multimodal sequences.
Dataset Summary
Type: multimodal instruction-tuning data
Format: discrete tokenized multimodal conversations / sequences
Use case: instruction tuning… See the full description on the dataset page: https://huggingface.co/datasets/ModalityDance/AR-Omni-Instruct-v0.1.instructtts-qwen-eval
InstructTTSEval Qwen3-TTS Evaluation Set
A ready-to-use evaluation slice of InstructTTSEval: the first 30 English samples, with TTS audio generated by Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign and instruction-following judgments produced by Qwen/Qwen2.5-Omni-3B as an audio judge.
Each of the 30 source samples has three style-instruction variants, giving 90 rows (one row per sample × instruction type):
Column
Type
Description
id
string
Source sample id in InstructTTSEval… See the full description on the dataset page: https://huggingface.co/datasets/kosw/instructtts-qwen-eval.Transcription-Instructions
Transcription Instructions
Convert ASR dataset into speech instructions.
Sample from mesolitica/pseudolabel-science-large-v3-timestamp
Sample from mesolitica/pseudolabel-malaysian-youtube-whisper-large-v3-timestamp
Sample from mesolitica/Malaysian-Emilia
Mandarin sample from mesolitica/Extra-Emilia/mandarin_v2
Cantonese sample from alvanlii/cantonese-radio
Entire dataset from mesolitica/Malaysian-STT-Whisper/extra
how to prepare the dataset
huggingface-cli download \… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Transcription-Instructions.Hindi-speech-instruct
Hindi LLaMA-Omni Instruct Dataset
A Hindi speech instruction-following dataset designed for training speech-language models such as LLaMA-Omni. Each example pairs a spoken Hindi user question (audio) with a text assistant response.
Dataset Summary
Property
Value
Language
Hindi (hi)
Total examples
~110,718
Train split
~105,000 examples (batches 001–210)
Validation split
~5,500 examples (batches 211–222)
Audio format
FLAC, 16,000 Hz mono… See the full description on the dataset page: https://huggingface.co/datasets/Pastaaaaa2003/Hindi-speech-instruct.