datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
instruction-speech-encodec-v1
Dataset Card for "Instruction Speech"
The largest open-source English speech instruction to text answer dataset
Dataset Overview
This dataset contains nearly 450,000 English speech instruction to text answer samples, using:
A subset of OpenHermes 2.5 with user's prompt length less than 64.
Audio generation using WhisperSpeech.
Tokenized using Encodec.
Usage
from datasets import load_dataset, Audio
# Load Instruction Speech dataset
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/Menlo/instruction-speech-encodec-v1.Speech-InstructionsZeroshot-Audio-Classification-Instructions
Zeroshot-Audio-Classification-Instructions
Convert audio classification dataset into zero-shot format speech instructions, support both single label and multi-label,
VGGSound
FSD50k
Nonspeech7k
urbansound8K
VocalSound
Emotion
Gender
ESD Emotion
Age
Language
TAU Urban Acoustic Scenes 2022
CochlScene
BirdCLEF_2021
EmoBox
AudioSet
We also converted huge WAV files into MP3 16k sample rate to reduce storage size.To prevent leakage, please do not include test set in training session.… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Zeroshot-Audio-Classification-Instructions.Malaysian-Speech-Instructions
Malaysian Speech Instructions
We filter Malaysian short prompts that suitable to convert to voice prompt and generate synthetic voice using mesolitica/Malaysian-F5-TTS-v2 also verified with Force Alignment to make sure the pronunciations almost correct.
We follow chat template from Qwen/Qwen2-Audio-7B-Instruct.
It contains dataset from the following sets:
Malaysian-Instructions
Critics Malaysian
Malaysian General QA
Malaysian Ultrachat
Malaysian Alpaca
Multi-turn Stupid questions… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Speech-Instructions.instruction-speech-encodec-v1.5
Dataset Card for "Instruction Speech"
The largest open-source English speech instruction to text answer dataset
Dataset Overview
This dataset contains over 332,000 English speech instruction to text answer samples, using:
A subset of jan-hq/prompt-voice-v1.5.
Audio generation using WhisperSpeech.
Tokenized using Encodec.
Usage
from datasets import load_dataset, Audio
# Load Instruction Speech dataset
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/Menlo/instruction-speech-encodec-v1.5.AudioSet-Audio-Instructions
AudioSet-Audio-Instructions
Convert AudioSet up to 527 audio labels to Speech Instruction dataset.
For Speech, we transcribe first using Whisper Large V3 after that use the transcription with the label to generate the synthetic instructions.
Classification-Speech-Instructions
Classification Speech Instructions
Speech instructions for emotion, gender, age and language audio classification.
Source code
Source code at https://github.com/mesolitica/malaysian-dataset/tree/master/llm-instruction/speech-classification-instructions
Animal-Sound-Instructions
Animal Sound Instructions
We gathered from,
Birds, birdclef-2021
Insecta, christopher/birdclef-2025
Amphibia, christopher/birdclef-2025
Mammalia, christopher/birdclef-2025
We use Qwen/Qwen2.5-72B-Instruct to generate the answers based on the metadata.
how to prepare the dataset
huggingface-cli download \
mesolitica/Animal-Sound-Instructions \
--include "*.zip" \
--repo-type "dataset" \
--local-dir './'
wget… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Animal-Sound-Instructions.CoVoST2-Instructions
CoVoST2 Instruction
Originally from https://huggingface.co/datasets/facebook/covost2, we converted to speech instruction format. We also provide test split.
We highly recommend to not include test set in training set to prevent contamination. Test set supposely to become a speech translation benchmark.
how to prepare the dataset
huggingface-cli download \
mesolitica/CoVoST2-Instructions \
--include "*.zip" \
--repo-type "dataset" \
--local-dir './'
wget… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/CoVoST2-Instructions.Speech-Translation-Instructions
Speech-Translation-Instructions
The instructions translated from 120 languages Common Voice to english, arabic, japanese, mandarin and french from common voice speech dataset. Suitable to use to finetune Speech LLM.
multimodal-expert-instruction-samples
Multimodal Expert Instruction Samples - Musical Instrument Lessons with Channel-separated Audio and Video
A music teacher and a student work through two one-on-one lessons: both voices and both instruments on separate tracks, the student on camera, with the lesson plans, the instructions given to each side and both sides' post-lesson ratings alongside.
▶ Watch the lessons · See Peer Collaboration samples · Discuss the full collection
Sister collection: Peer Collaboration… See the full description on the dataset page: https://huggingface.co/datasets/fluid-concepts/multimodal-expert-instruction-samples.Cantonese-Radio-Description-Instructions
Cantonese-Radio-Description-Instructions
Originally from alvanlii/cantonese-radio, we use Qwen/Qwen2.5-72B-Instruct to generate description based on the transcription.
how to prepare the dataset
huggingface-cli download \
mesolitica/Cantonese-Radio-Description-Instructions \
--include '*.zip' \
--repo-type "dataset" \
--local-dir './'
wget https://gist.githubusercontent.com/huseinzol05/2e26de4f3b29d99e993b349864ab6c10/raw/9b2251f3ff958770215d70c8d82d311f82791b78/unzip.py… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Cantonese-Radio-Description-Instructions.Speaker-Diarization-Instructions
Speaker-Diarization-Instructions
Convert diarization dataset from https://huggingface.co/diarizers-community into speech instructions dataset and chunk max to 30 seconds because most of speech encoder use for LLM come from Whisper Encoder.
We highly recommend to not include AMI test set from both AMI-IHM and AMI-SDM in training set to prevent contamination. This dataset supposely to become a speech diarization benchmark.
how to prepare the dataset
huggingface-cli… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Speaker-Diarization-Instructions.Audio-Adversarial-Instructions
Audio-Adversarial-Instructions
We convert to audio boolean classifier speech instructions, randomly 50% chance for positive and other 50% chance for negative.
Data source
AudioSet from mesolitica/AudioSet-Audio-Instructions
Classification Speech Instruction from mesolitica/Classification-Speech-Instructions
EmoBox from mesolitica/Classification-Speech-Instructions
BirdCLEF_2021
how to prepare the dataset
huggingface-cli download \… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Audio-Adversarial-Instructions.openhermes_instruction_test@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and Lin, Geyu and Sun, Shuo and Liu, Zhuohan and Zhang, Wenyu and Liu, Zhengyuan and Aw, AiTi and Chen, Nancy F},
journal={NAACL},
year={2025}
}
CompA-R-Instructions
CompA-R-Instructions
Originally from https://huggingface.co/papers/2406.11768, we downloaded from Google Drive and converted to HuggingFace.
Malaysian-UltraChat-Speech-Multiturn-Instructions
Malaysian-UltraChat-Speech-Multiturn-Instructions
We filter Malaysian short user questions from mesolitica/malaysian-ultrachat that suitable to convert to voice prompt and generate synthetic voice using mesolitica/Malaysian-Dia-1.6B also verified with Force Alignment to make sure the pronunciations almost correct.
A conversation must at least have 2 audio. We follow chat template from Qwen/Qwen2-Audio-7B-Instruct.
how to prepare the dataset
huggingface-cli download \… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-UltraChat-Speech-Multiturn-Instructions.Transcription-Instructions
Transcription Instructions
Convert ASR dataset into speech instructions.
Sample from mesolitica/pseudolabel-science-large-v3-timestamp
Sample from mesolitica/pseudolabel-malaysian-youtube-whisper-large-v3-timestamp
Sample from mesolitica/Malaysian-Emilia
Mandarin sample from mesolitica/Extra-Emilia/mandarin_v2
Cantonese sample from alvanlii/cantonese-radio
Entire dataset from mesolitica/Malaysian-STT-Whisper/extra
how to prepare the dataset
huggingface-cli download \… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Transcription-Instructions.Malaysian-Reasoning-Speech-Instructions
Malaysian Reasoning Speech Instructions
We filter Malaysian short prompts from mesolitica/Malaysian-Reasoning that suitable to convert to voice prompt and generate synthetic voice using mesolitica/Malaysian-Dia-1.6B also verified with Force Alignment to make sure the pronunciations almost correct.
Source code
Source code at https://github.com/mesolitica/malaysian-dataset/tree/master/speech-instruction/malaysian-reasoning
Speech-Instructions-ExtraSpeech-Instructions-MalayMMLU
Speech Instructions MalayMMLU
We filter MalayMMLU that suitable to convert to voice prompt and generate synthetic voice using mesolitica/Malaysian-Dia-1.6B also verified with Force Alignment to make sure the pronunciations almost correct.
We highly recommend to not include this dataset in training set to prevent contamination. This dataset supposely to become a speech QA benchmark.
Benchmark
gpt-4o-audio-preview-2025-06-03
Social science… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Speech-Instructions-MalayMMLU.IWSE-InstructionBasedSpeechEdit-llasa_tokenizesynth_instructionstext_multi_instruction_asropenhermes_instructionspeech_multi_instruction_asropenhermes_instructionIWSE-InstructionBasedSpeechEditinstructiontext_multi_instruction_st