datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Malaysian-Emilia
Malaysian Emilia
An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Malaysian and Singaporean Speech Generation, where originally from Emilia.
We are improving Malaysian Emilia due to https://github.com/open-mmlab/Amphion/issues/436, check out mesolitica/Malaysian-Emilia-v2
Dataset
Clone and Extract
We upload as split zip files so you can clone and extract distributedly,
huggingface-cli download --repo-type dataset \
--include… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Emilia.Malay-Dialect-Instructions
Malay dialect instruction including coding
Negeri Sembilan
QA
public transport QA,
Coding
CUDA coding,
Kedah
QA
infra QA,
Coding
Rust coding,
Kelantan
QA
Najib Razak QA,
Coding
Go coding,
Perak
QA
Anwar Ibrahim QA,
Coding
SQL coding,
Pahang
QA
Pendatang asing QA,
Coding
Typescript coding,
Terengganu… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malay-Dialect-Instructions.Malaysian-Emilia-annotated
Malaysian Emilia Annotated
Annotate Malaysian-Emilia using Data-Speech pipeline.
Malaysian Youtube
Originally from malaysia-ai/crawl-youtube
Total 3168.8 hours.
Gender prediction, filtered-24k_processed_24k_gender.zip
Language prediction, filtered-24k_processed_language.zip
Force alignment.
Post cleaned to 24k and 44k sampling rates,
24k, filtered-24k_processed_24k.zip
44k, filtered-24k_processed_44k.zip
Synthetic description… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Emilia-annotated.Malaysian-STT-Whisper
Malaysian STT Whisper format
Heavy postprocessing and post-translation to improve pseudolabeled Whisper Large V3. Also include word level timestamp.
Postprocessing
Check repetitive trigrams.
Verify Voice Activity using Silero-VAD.
Verify scores using Force Alignment.
Post-translation
We use mesolitica/nanot5-base-malaysian-translation-v2.1.
Dataset involved
Malaysian context v2
Singaporean context
Indonesian context
Mandarin audio
Tamil audio… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper.pseudolabel-malaysian-youtube-whisper-large-v3
Pseudolabel Malaysian Youtube videos using Whisper Large V3
Original dataset at https://huggingface.co/datasets/malaysia-ai/crawl-youtube, distributed pseudolabelled using 4x A100s
script at https://github.com/mesolitica/malaysian-dataset/tree/master/speech-to-text-semisupervised/pseudolabel-whisper
Each audio is 30 seconds.
Each audio saved in 16k sample rate.
Malaysian-SFT
Malaysian SFT
It contains dataset from the following sets:
mesolitica/chatgpt4-malaysian-general-qa
mesolitica/llama3-70b-social-media-qa
mesolitica/llama3-70b-qa
mesolitica/llama3-70b-non-bumi-qa
mesolitica/peribahasa-instructions
mesolitica/google-translate-camel-ai
mesolitica/synthetic-jawi-conversation
mesolitica/jawi-code-instructions
mesolitica/chatgpt4-code-instruct
mesolitica/malaysian-ultrachat
mesolitica/malay-dialect-instructions… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-SFT.Malaysian-TTS-v2
Malaysian TTS v2
Generate Malay and localize English for TTS dataset, currently only support 2 speakers, husein and idayu, where total audio is 4642.77 hours.
How to prepare the dataset
huggingface-cli download \
mesolitica/Malaysian-TTS-v2 \
--include "all-*.zip" \
--repo-type "dataset" \
--local-dir './'
huggingface-cli download \
mesolitica/STT-Normalizer \
--include "*husein*.zip" \
--exclude "*force*" \
--repo-type "dataset" \
--local-dir './'… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-TTS-v2.Malaysian-TTS
TTS
Malaysian Synthetic TTS dataset.
Generate using each Malaysian-F5-TTS-v2.
Each generation verified using esammahdi/ctc-forced-aligner.
Post-filter pitch using interactiveaudiolab/penn.
Speaker
Husein, 300 hours.
Shafiqah Idayu, 292 hours.
Anwar Ibrahim, 269 hours.
KP RTM Suhaimi Malay, 306 hours.
KP RTM Suhaimi Chinese, 192 hours.
Clean version
We trimmed start and end silents, and compressed at processed
Dataset uploaded as HuggingFace datasets… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-TTS.IMDA-STT
IMDA National Speech Corpus (NSC) Speech-to-Text
Originally from https://www.imda.gov.sg/how-we-can-help/national-speech-corpus, this repository simply a mirror. This dataset associated with Singapore Open Data Licence, https://www.sla.gov.sg/newsroom/statistics/singapore-open-data-licence
We uploaded mp3 files and compressed using 7z,
7za x part1-mp3.7z.001
All notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/speech-to-text/imda
total lengths… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/IMDA-STT.pseudolabel-malaysian-youtube-whisper-large-v3-timestamp
Pseudolabel Malaysian Youtube using Whisper Large V3 including Timestamp
how to prepare the dataset
wget https://huggingface.co/datasets/mesolitica/pseudolabel-malaysian-youtube-whisper-large-v3-timestamp/resolve/main/prepared-pseudolabel.jsonl
huggingface-cli download --repo-type dataset \
--include 'output-audio-*.zip' \
--local-dir './' \
--max-workers 20 \
mesolitica/pseudolabel-malaysian-youtube-whisper-large-v3-timestamp
wget… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/pseudolabel-malaysian-youtube-whisper-large-v3-timestamp.instructions-pair-miningpseudolabel-science-large-v3-timestamp
Pseudolabel science context audio using Whisper Large V3
Original audio from malaysia-ai/science-context-youtube, we split every 30 seconds and pseudolabelled using Whisper Large V3.
how to prepare the dataset
huggingface-cli download --repo-type dataset \
--include 'science-chunk-*.zip' \
--local-dir './' \
--max-workers 20 \
mesolitica/pseudolabel-science-large-v3-timestamp
wget… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/pseudolabel-science-large-v3-timestamp.google-image-malaysian-vehicle-dedup
Google Image Malaysian Vehicle Dedup
Original dataset https://huggingface.co/datasets/malaysia-ai/crawl-google-image-malaysian-vehicle
Source code at https://github.com/mesolitica/malaysian-dataset/tree/master/vlm/dedup-malaysian-vehicle
Dedup 70% similar
dedup-0.7.jsonl, total deduped 97598 images,
{'filename': 'train-00075-of-00165-c0ebcc169b1f62d2.parquet',
'keyword': '2021 Honda City 1.5 E',
'no': 2,
'selected_indices': [696,
702,
705,
707,
712,
716… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/google-image-malaysian-vehicle-dedup.fineweb-filter-malaysian-context
HuggingFaceFW/fineweb filter Malaysian context
What is it?
We filter the original 🍷 FineWeb dataset that consists more than 15T tokens on simple Malaysian keywords.
Total tokens for the filtered dataset is 174102784199 tokens, 174B tokens.
How we do it?
We filter rows using {'malay', 'malaysia', 'melayu', 'bursa', 'ringgit'} keywords on r5.16xlarge EC2 instance for 7 days.
We calculate total tokens using tiktoken.encoding_for_model("gpt2") on c7a.24xlarge EC2… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/fineweb-filter-malaysian-context.malaya-speech-malay-stt
Malaya-Speech Speech-to-Text dataset
This dataset combined from semisupervised Google Speech-to-Text and private datasets.
Processing script https://github.com/mesolitica/malaya-speech/blob/master/pretrained-model/prepare-stt/prepare-malay-stt-train.ipynb
This repository is to centralize the dataset for https://malaya-speech.readthedocs.io/
mesolitica-malayZeroshot-Audio-Classification-Instructions
Zeroshot-Audio-Classification-Instructions
Convert audio classification dataset into zero-shot format speech instructions, support both single label and multi-label,
VGGSound
FSD50k
Nonspeech7k
urbansound8K
VocalSound
Emotion
Gender
ESD Emotion
Age
Language
TAU Urban Acoustic Scenes 2022
CochlScene
BirdCLEF_2021
EmoBox
AudioSet
We also converted huge WAV files into MP3 16k sample rate to reduce storage size.To prevent leakage, please do not include test set in training session.… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Zeroshot-Audio-Classification-Instructions.Malaysian-Speech-Instructions
Malaysian Speech Instructions
We filter Malaysian short prompts that suitable to convert to voice prompt and generate synthetic voice using mesolitica/Malaysian-F5-TTS-v2 also verified with Force Alignment to make sure the pronunciations almost correct.
We follow chat template from Qwen/Qwen2-Audio-7B-Instruct.
It contains dataset from the following sets:
Malaysian-Instructions
Critics Malaysian
Malaysian General QA
Malaysian Ultrachat
Malaysian Alpaca
Multi-turn Stupid questions… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Speech-Instructions.Azure-TTS-annotatedMalaysian-STT-Whisper-Stage2
Malaysian STT Whisper Stage 2
Extra dataset to compliment mesolitica/Malaysian-STT-Whisper.
This dataset is stronger in confidence and suitable for second stage / annealing finetuning.
how to prepare the dataset
huggingface-cli download \
mesolitica/Malaysian-STT-Whisper-Stage2 \
--include "*.zip" \
--repo-type "dataset" \
--local-dir './'
huggingface-cli download \
mesolitica/Malaysian-Multiturn-Chat-Assistant \
--include "*.zip" \
--exclude "voice/*.zip" \
--repo-type… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper-Stage2.language-detectionpseudolabel-tamil-large-v3-timestamp
Pseudolabel Tamil using Whisper Large V3
Original dataset is newsonair_v5 from AI4Bharat, Heavy postprocessing and post-translation to improve pseudolabeled Whisper Large V3.
prepare dataset
wget https://gist.githubusercontent.com/huseinzol05/2e26de4f3b29d99e993b349864ab6c10/raw/9b2251f3ff958770215d70c8d82d311f82791b78/unzip.py
huggingface-cli download --repo-type dataset \
--include 'newsonair_v5-*.zip' \
--local-dir './' \
mesolitica/pseudolabel-tamil-large-v3-timestamp… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/pseudolabel-tamil-large-v3-timestamp.AudioSet-Audio-Instructions
AudioSet-Audio-Instructions
Convert AudioSet up to 527 audio labels to Speech Instruction dataset.
For Speech, we transcribe first using Whisper Large V3 after that use the transcription with the label to generate the synthetic instructions.
mixtral-magicoder
Mixtral Magicoder: Source Code Is All You Need on various programming languages
We sampled programming languages from https://huggingface.co/datasets/bigcode/the-stack-dedup and pushed to https://huggingface.co/datasets/malaysia-ai/starcoderdata-sample
After that, we use Magicoder: Source Code Is All You Need on various programming languages template, we target at least 10k rows for each programming languages.
C++, 10747 rows
C#, 10193 rows
CUDA, 13843 rows
Dockerfile, 13286 rows… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/mixtral-magicoder.Malaysian-Emilia-v2
Malaysian Emilia v2
This version 2 should fixed https://github.com/open-mmlab/Amphion/issues/436, an Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Malaysian and Singaporean Speech Generation. Replicating Emilia on,
Dataset
Clone and Extract
We upload as split zip files so you can clone and extract distributedly,
huggingface-cli download --repo-type dataset \
--include '*.zip' \
--local-dir './' \
--max-workers 20 \… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Emilia-v2.Classification-Speech-Instructions
Classification Speech Instructions
Speech instructions for emotion, gender, age and language audio classification.
Source code
Source code at https://github.com/mesolitica/malaysian-dataset/tree/master/llm-instruction/speech-classification-instructions
Extra-Emilia
Extra Emilia
Extra dataset to extend Tamil and Mandarin capability for Malaysian-Emilia.
Tamil
Total length is 891 hours.
Mandarin
Total length is 301 hours.
Speech-Benchmarklanguage-detection-datasetMalaysian-Ultrachat
Ultrachat like using Malaysian context
Prepare multiturn dialogue between user and assistant for malaysian context,
Astroawani, https://huggingface.co/datasets/malaysia-ai/crawl-astroawani, ultrachat-astroawani-malay.jsonl, 60198 rows, 477 MB.
Crossref melayu papers, https://huggingface.co/datasets/mesolitica/crawl-my-website/resolve/main/melayu-pdf.jsonl, ultrachat-crossref-melayu-malay.jsonl, 9959 rows, 187 MB
Epenerbitan… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Ultrachat.