meso
Datasets
All datasets matching “meso”Malaysian-Emilia
Malaysian Emilia
An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Malaysian and Singaporean Speech Generation, where originally from Emilia.
We are improving Malaysian Emilia due to https://github.com/open-mmlab/Amphion/issues/436, check out mesolitica/Malaysian-Emilia-v2
Dataset
Clone and Extract
We upload as split zip files so you can clone and extract distributedly,
huggingface-cli download --repo-type dataset \
--include… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Emilia.Malay-Dialect-Instructions
Malay dialect instruction including coding
Negeri Sembilan
QA
public transport QA,
Coding
CUDA coding,
Kedah
QA
infra QA,
Coding
Rust coding,
Kelantan
QA
Najib Razak QA,
Coding
Go coding,
Perak
QA
Anwar Ibrahim QA,
Coding
SQL coding,
Pahang
QA
Pendatang asing QA,
Coding
Typescript coding,
Terengganu… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malay-Dialect-Instructions.Malaysian-Emilia-annotated
Malaysian Emilia Annotated
Annotate Malaysian-Emilia using Data-Speech pipeline.
Malaysian Youtube
Originally from malaysia-ai/crawl-youtube
Total 3168.8 hours.
Gender prediction, filtered-24k_processed_24k_gender.zip
Language prediction, filtered-24k_processed_language.zip
Force alignment.
Post cleaned to 24k and 44k sampling rates,
24k, filtered-24k_processed_24k.zip
44k, filtered-24k_processed_44k.zip
Synthetic description… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Emilia-annotated.Malaysian-STT-Whisper
Malaysian STT Whisper format
Heavy postprocessing and post-translation to improve pseudolabeled Whisper Large V3. Also include word level timestamp.
Postprocessing
Check repetitive trigrams.
Verify Voice Activity using Silero-VAD.
Verify scores using Force Alignment.
Post-translation
We use mesolitica/nanot5-base-malaysian-translation-v2.1.
Dataset involved
Malaysian context v2
Singaporean context
Indonesian context
Mandarin audio
Tamil audio… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper.pseudolabel-malaysian-youtube-whisper-large-v3
Pseudolabel Malaysian Youtube videos using Whisper Large V3
Original dataset at https://huggingface.co/datasets/malaysia-ai/crawl-youtube, distributed pseudolabelled using 4x A100s
script at https://github.com/mesolitica/malaysian-dataset/tree/master/speech-to-text-semisupervised/pseudolabel-whisper
Each audio is 30 seconds.
Each audio saved in 16k sample rate.
Malaysian-SFT
Malaysian SFT
It contains dataset from the following sets:
mesolitica/chatgpt4-malaysian-general-qa
mesolitica/llama3-70b-social-media-qa
mesolitica/llama3-70b-qa
mesolitica/llama3-70b-non-bumi-qa
mesolitica/peribahasa-instructions
mesolitica/google-translate-camel-ai
mesolitica/synthetic-jawi-conversation
mesolitica/jawi-code-instructions
mesolitica/chatgpt4-code-instruct
mesolitica/malaysian-ultrachat
mesolitica/malay-dialect-instructions… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-SFT.