malaysian
sentiment-analysis-nanot5-small-malaysian-casedMalaysian-whisper-large-v3-turbo-v3Malaysian-gemma-3-27b-it-i1-GGUFMalaysian-Qwen2.5-1.5B-Instruct-v0.1gpt-oss-20b-Malaysian-Reasoning-SFT-v0.1-i1-GGUFMalaysian-Qwen2.5-7B-Dialect-Reasoning-GRPO-i1-GGUFMeta-Llama-3.1-70B-Instruct-Malaysian-i1-GGUFMalaysian-Qwen2.5-7B-Reasoning-SFT-i1-GGUF
malaysian-youtube
Malaysian Youtube
Malaysian and Singaporean youtube channels, total up to 60k audio files with total 18.7k hours.
URLs data at https://github.com/mesolitica/malaya-speech/tree/master/data/youtube/data
Notebooks at https://github.com/mesolitica/malaya-speech/tree/master/data/youtube
How to load the data efficiently?
import pandas as pd
import json
from datasets import Audio
from torch.utils.data import DataLoader, Dataset
chunks = 30
sr = 16000
class Train(Dataset):… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/malaysian-youtube.Malaysian-Emilia
Malaysian Emilia
An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Malaysian and Singaporean Speech Generation, where originally from Emilia.
We are improving Malaysian Emilia due to https://github.com/open-mmlab/Amphion/issues/436, check out mesolitica/Malaysian-Emilia-v2
Dataset
Clone and Extract
We upload as split zip files so you can clone and extract distributedly,
huggingface-cli download --repo-type dataset \
--include… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Emilia.Malaysian-Emilia-annotated
Malaysian Emilia Annotated
Annotate Malaysian-Emilia using Data-Speech pipeline.
Malaysian Youtube
Originally from malaysia-ai/crawl-youtube
Total 3168.8 hours.
Gender prediction, filtered-24k_processed_24k_gender.zip
Language prediction, filtered-24k_processed_language.zip
Force alignment.
Post cleaned to 24k and 44k sampling rates,
24k, filtered-24k_processed_24k.zip
44k, filtered-24k_processed_44k.zip
Synthetic description… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Emilia-annotated.Malaysian-STT-Whisper
Malaysian STT Whisper format
Heavy postprocessing and post-translation to improve pseudolabeled Whisper Large V3. Also include word level timestamp.
Postprocessing
Check repetitive trigrams.
Verify Voice Activity using Silero-VAD.
Verify scores using Force Alignment.
Post-translation
We use mesolitica/nanot5-base-malaysian-translation-v2.1.
Dataset involved
Malaysian context v2
Singaporean context
Indonesian context
Mandarin audio
Tamil audio… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper.malaysian-dialects-youtube
Malaysian dialects Youtube
Entire videos from https://www.youtube.com using 'malay dialects' keyword.
With total 398634 audio files, total 68607.6 hours.
how to download
huggingface-cli download --repo-type dataset \
--include '*.z*' \
--local-dir './' \
malaysia-ai/malaysian-dialects-youtube
https://gist.githubusercontent.com/huseinzol05/2e26de4f3b29d99e993b349864ab6c10/raw/9b2251f3ff958770215d70c8d82d311f82791b78/unzip.py
python3 unzip.py
Source code
Source… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/malaysian-dialects-youtube.Malaysian-STT
Malaysian-STT
Prepare streaming and whole mode Speech-to-Text Malaysian context dataset, suitable to train streaming LLM base or Encoder-Decoder such as Whisper.
Merged 30 seconds chunk into one audio file, can up to 10 minutes.
Segmentize based on silent at least 0.3 seconds.
Reject low score based on force alignment.
Reject timestamp anomaly based on force alignment.
Dataset involved
Dialects
IMDA
Malaysian context
Malaysia Parliament
Science context… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/Malaysian-STT.