datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
malaysian-youtube
Malaysian Youtube
Malaysian and Singaporean youtube channels, total up to 60k audio files with total 18.7k hours.
URLs data at https://github.com/mesolitica/malaya-speech/tree/master/data/youtube/data
Notebooks at https://github.com/mesolitica/malaya-speech/tree/master/data/youtube
How to load the data efficiently?
import pandas as pd
import json
from datasets import Audio
from torch.utils.data import DataLoader, Dataset
chunks = 30
sr = 16000
class Train(Dataset):… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/malaysian-youtube.Malaysian-Emilia
Malaysian Emilia
An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Malaysian and Singaporean Speech Generation, where originally from Emilia.
We are improving Malaysian Emilia due to https://github.com/open-mmlab/Amphion/issues/436, check out mesolitica/Malaysian-Emilia-v2
Dataset
Clone and Extract
We upload as split zip files so you can clone and extract distributedly,
huggingface-cli download --repo-type dataset \
--include… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Emilia.Malaysian-Emilia-annotated
Malaysian Emilia Annotated
Annotate Malaysian-Emilia using Data-Speech pipeline.
Malaysian Youtube
Originally from malaysia-ai/crawl-youtube
Total 3168.8 hours.
Gender prediction, filtered-24k_processed_24k_gender.zip
Language prediction, filtered-24k_processed_language.zip
Force alignment.
Post cleaned to 24k and 44k sampling rates,
24k, filtered-24k_processed_24k.zip
44k, filtered-24k_processed_44k.zip
Synthetic description… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Emilia-annotated.Malaysian-STT-Whisper
Malaysian STT Whisper format
Heavy postprocessing and post-translation to improve pseudolabeled Whisper Large V3. Also include word level timestamp.
Postprocessing
Check repetitive trigrams.
Verify Voice Activity using Silero-VAD.
Verify scores using Force Alignment.
Post-translation
We use mesolitica/nanot5-base-malaysian-translation-v2.1.
Dataset involved
Malaysian context v2
Singaporean context
Indonesian context
Mandarin audio
Tamil audio… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper.malaysian-dialects-youtube
Malaysian dialects Youtube
Entire videos from https://www.youtube.com using 'malay dialects' keyword.
With total 398634 audio files, total 68607.6 hours.
how to download
huggingface-cli download --repo-type dataset \
--include '*.z*' \
--local-dir './' \
malaysia-ai/malaysian-dialects-youtube
https://gist.githubusercontent.com/huseinzol05/2e26de4f3b29d99e993b349864ab6c10/raw/9b2251f3ff958770215d70c8d82d311f82791b78/unzip.py
python3 unzip.py
Source code
Source… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/malaysian-dialects-youtube.Malaysian-STT
Malaysian-STT
Prepare streaming and whole mode Speech-to-Text Malaysian context dataset, suitable to train streaming LLM base or Encoder-Decoder such as Whisper.
Merged 30 seconds chunk into one audio file, can up to 10 minutes.
Segmentize based on silent at least 0.3 seconds.
Reject low score based on force alignment.
Reject timestamp anomaly based on force alignment.
Dataset involved
Dialects
IMDA
Malaysian context
Malaysia Parliament
Science context… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/Malaysian-STT.pseudolabel-malaysian-youtube-whisper-large-v3
Pseudolabel Malaysian Youtube videos using Whisper Large V3
Original dataset at https://huggingface.co/datasets/malaysia-ai/crawl-youtube, distributed pseudolabelled using 4x A100s
script at https://github.com/mesolitica/malaysian-dataset/tree/master/speech-to-text-semisupervised/pseudolabel-whisper
Each audio is 30 seconds.
Each audio saved in 16k sample rate.
Malaysian-SFT
Malaysian SFT
It contains dataset from the following sets:
mesolitica/chatgpt4-malaysian-general-qa
mesolitica/llama3-70b-social-media-qa
mesolitica/llama3-70b-qa
mesolitica/llama3-70b-non-bumi-qa
mesolitica/peribahasa-instructions
mesolitica/google-translate-camel-ai
mesolitica/synthetic-jawi-conversation
mesolitica/jawi-code-instructions
mesolitica/chatgpt4-code-instruct
mesolitica/malaysian-ultrachat
mesolitica/malay-dialect-instructions… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-SFT.Malaysian-TTS-v2
Malaysian TTS v2
Generate Malay and localize English for TTS dataset, currently only support 2 speakers, husein and idayu, where total audio is 4642.77 hours.
How to prepare the dataset
huggingface-cli download \
mesolitica/Malaysian-TTS-v2 \
--include "all-*.zip" \
--repo-type "dataset" \
--local-dir './'
huggingface-cli download \
mesolitica/STT-Normalizer \
--include "*husein*.zip" \
--exclude "*force*" \
--repo-type "dataset" \
--local-dir './'… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-TTS-v2.Malaysian-TTS
TTS
Malaysian Synthetic TTS dataset.
Generate using each Malaysian-F5-TTS-v2.
Each generation verified using esammahdi/ctc-forced-aligner.
Post-filter pitch using interactiveaudiolab/penn.
Speaker
Husein, 300 hours.
Shafiqah Idayu, 292 hours.
Anwar Ibrahim, 269 hours.
KP RTM Suhaimi Malay, 306 hours.
KP RTM Suhaimi Chinese, 192 hours.
Clean version
We trimmed start and end silents, and compressed at processed
Dataset uploaded as HuggingFace datasets… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-TTS.pseudolabel-malaysian-youtube-whisper-large-v3-timestamp
Pseudolabel Malaysian Youtube using Whisper Large V3 including Timestamp
how to prepare the dataset
wget https://huggingface.co/datasets/mesolitica/pseudolabel-malaysian-youtube-whisper-large-v3-timestamp/resolve/main/prepared-pseudolabel.jsonl
huggingface-cli download --repo-type dataset \
--include 'output-audio-*.zip' \
--local-dir './' \
--max-workers 20 \
mesolitica/pseudolabel-malaysian-youtube-whisper-large-v3-timestamp
wget… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/pseudolabel-malaysian-youtube-whisper-large-v3-timestamp.google-image-malaysian-vehicle-dedup
Google Image Malaysian Vehicle Dedup
Original dataset https://huggingface.co/datasets/malaysia-ai/crawl-google-image-malaysian-vehicle
Source code at https://github.com/mesolitica/malaysian-dataset/tree/master/vlm/dedup-malaysian-vehicle
Dedup 70% similar
dedup-0.7.jsonl, total deduped 97598 images,
{'filename': 'train-00075-of-00165-c0ebcc169b1f62d2.parquet',
'keyword': '2021 Honda City 1.5 E',
'no': 2,
'selected_indices': [696,
702,
705,
707,
712,
716… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/google-image-malaysian-vehicle-dedup.fineweb-filter-malaysian-context
HuggingFaceFW/fineweb filter Malaysian context
What is it?
We filter the original 🍷 FineWeb dataset that consists more than 15T tokens on simple Malaysian keywords.
Total tokens for the filtered dataset is 174102784199 tokens, 174B tokens.
How we do it?
We filter rows using {'malay', 'malaysia', 'melayu', 'bursa', 'ringgit'} keywords on r5.16xlarge EC2 instance for 7 days.
We calculate total tokens using tiktoken.encoding_for_model("gpt2") on c7a.24xlarge EC2… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/fineweb-filter-malaysian-context.malaysian-cartoons-youtube
Crawl Youtube Malaysian cartoons
With total 45544 audio files, total 20766 hours.
how to download
huggingface-cli download --repo-type dataset \
--include '*.zip' \
--local-dir './' \
malaysia-ai/malaysian-cartoons-youtube
wget https://gist.githubusercontent.com/huseinzol05/2e26de4f3b29d99e993b349864ab6c10/raw/9b2251f3ff958770215d70c8d82d311f82791b78/unzip.py
python3 unzip.py
Source code
Source code at… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/malaysian-cartoons-youtube.Malaysian-Emilia
Malaysian Emilia
Gather Malaysian Emilia from,
https://huggingface.co/datasets/mesolitica/Malaysian-Emilia-v2
https://huggingface.co/datasets/Scicom-intl/Malaysian-Chinese-Emilia
https://huggingface.co/datasets/mesolitica/Malaysian-Emilia#malaysian-dialect
And do,
Trim silent.
Permutation for Voice Conversion include post-filtering during permutation.
Convert to Neucodec speech tokens.
Malaysian-Chinese-Emilia
Malaysian-Chinese-Emilia
Use https://github.com/mesolitica/Emilia to pseudo-label Malaysian Chinese audio.
Total rows: 605169
Total hours: 1857.611445057867 hours
Permutation for Voice Conversion
Also we already calculated speaker permutation to prepare for voice conversion.
Malaysian-Speech-Instructions
Malaysian Speech Instructions
We filter Malaysian short prompts that suitable to convert to voice prompt and generate synthetic voice using mesolitica/Malaysian-F5-TTS-v2 also verified with Force Alignment to make sure the pronunciations almost correct.
We follow chat template from Qwen/Qwen2-Audio-7B-Instruct.
It contains dataset from the following sets:
Malaysian-Instructions
Critics Malaysian
Malaysian General QA
Malaysian Ultrachat
Malaysian Alpaca
Multi-turn Stupid questions… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Speech-Instructions.malaysian-cartoons-youtube-filtered-24k
Filtered Youtube Malaysian Cartoons Youtube
Originally from https://huggingface.co/datasets/malaysia-ai/malaysian-cartoons-youtube, we filtered audio less than 4 hours and converted to 24k sampling rate for audio processing.
how to download
huggingface-cli download --repo-type dataset \
--include '*.z*' \
--local-dir './' \
malaysia-ai/malaysian-cartoons-youtube-filtered-24k
wget https://www.7-zip.org/a/7z2301-linux-x64.tar.xz
tar -xf 7z2301-linux-x64.tar.xz
~/7zz x… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/malaysian-cartoons-youtube-filtered-24k.Malaysian-STT-Whisper-Stage2
Malaysian STT Whisper Stage 2
Extra dataset to compliment mesolitica/Malaysian-STT-Whisper.
This dataset is stronger in confidence and suitable for second stage / annealing finetuning.
how to prepare the dataset
huggingface-cli download \
mesolitica/Malaysian-STT-Whisper-Stage2 \
--include "*.zip" \
--repo-type "dataset" \
--local-dir './'
huggingface-cli download \
mesolitica/Malaysian-Multiturn-Chat-Assistant \
--include "*.zip" \
--exclude "voice/*.zip" \
--repo-type… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper-Stage2.malaysian-youtube-filtered-24k
Filtered Malaysian Youtube
Originally from https://huggingface.co/datasets/malaysia-ai/malaysian-youtube, we filtered audio less than 4 hours and converted to 24k sampling rate for audio processing.
how to download
huggingface-cli download --repo-type dataset \
--include '*.z*' \
--local-dir './' \
malaysia-ai/malaysian-youtube-filtered-24k
wget https://www.7-zip.org/a/7z2301-linux-x64.tar.xz
tar -xf 7z2301-linux-x64.tar.xz
~/7zz x filtered-24k.zip -y -mmt40
Malaysian-Emilia-v2
Malaysian Emilia v2
This version 2 should fixed https://github.com/open-mmlab/Amphion/issues/436, an Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Malaysian and Singaporean Speech Generation. Replicating Emilia on,
Dataset
Clone and Extract
We upload as split zip files so you can clone and extract distributedly,
huggingface-cli download --repo-type dataset \
--include '*.zip' \
--local-dir './' \
--max-workers 20 \… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Emilia-v2.malaysian-podcast-youtube
Crawl Youtube Malaysian Podcast
With total 19092 audio files, total 2233.8 hours.
how to download
huggingface-cli download --repo-type dataset \
--include '*.z*' \
--local-dir './' \
malaysia-ai/malaysian-podcast-youtube
wget https://www.7-zip.org/a/7z2301-linux-x64.tar.xz
tar -xf 7z2301-linux-x64.tar.xz
~/7zz x malaysian-podcast.zip -y -mmt40
Licensing
All the videos, songs, images, and graphics used in the video belong to their respective owners and I does… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/malaysian-podcast-youtube.Malaysian-Ultrachat
Ultrachat like using Malaysian context
Prepare multiturn dialogue between user and assistant for malaysian context,
Astroawani, https://huggingface.co/datasets/malaysia-ai/crawl-astroawani, ultrachat-astroawani-malay.jsonl, 60198 rows, 477 MB.
Crossref melayu papers, https://huggingface.co/datasets/mesolitica/crawl-my-website/resolve/main/melayu-pdf.jsonl, ultrachat-crossref-melayu-malay.jsonl, 9959 rows, 187 MB
Epenerbitan… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-Ultrachat.malaysian-rerankercrawl-google-image-malaysian-vehicle
Crawl Google Image Malaysian Car
Crawl Google Image using Malaysian car keywords.
Source code at https://github.com/mesolitica/malaysian-dataset/tree/master/crawl/google-image
Malaysian-SFT
Malaysian SFT
It contains dataset from the following sets:
mesolitica/chatgpt4-malaysian-general-qa
mesolitica/llama3-70b-social-media-qa
mesolitica/llama3-70b-qa
mesolitica/llama3-70b-non-bumi-qa
mesolitica/peribahasa-instructions
mesolitica/google-translate-camel-ai
mesolitica/synthetic-jawi-conversation
mesolitica/jawi-code-instructions
mesolitica/chatgpt4-code-instruct
mesolitica/malaysian-ultrachat
mesolitica/malay-dialect-instructions… See the full description on the dataset page: https://huggingface.co/datasets/skilledu/Malaysian-SFT.Malaysian-Translation
Malaysian-Translation
Noisy translation dataset to train translation models from us, we provided 2 stages of training, first stage is much more noisy but second stage should be cleaner.
mixtral-malaysian-general-qa
Mixtral Malaysian Chat
Simulate conversation between a user and an assistant on various topics. Generated using Mixtral Instructions.
Notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/chatbot/mixtral-malaysian-chat
Multi-turn Bad things
Multiturn of the user is saying bad things to the assistant.
mixtral-conversation-badthings.jsonl, 57798 rows, 163 MB.
Example data
[{'role': 'user',
'content': "Hey bot, you're really dumb."… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/mixtral-malaysian-general-qa.dedup-Malaysian-Emilia
Dedup Malaysian Emilia
Dedup mesolitica/Malaysian-Emilia at 95% similarity threshold.
You can use this for voice profiling or public voice synthetic generation.
download the dataset
huggingface-cli download \
mesolitica/dedup-Malaysian-Emilia \
--include "*.zip" \
--repo-type "dataset" \
--local-dir './'
wget https://gist.githubusercontent.com/huseinzol05/2e26de4f3b29d99e993b349864ab6c10/raw/9b2251f3ff958770215d70c8d82d311f82791b78/unzip.py
python3 unzip.py… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/dedup-Malaysian-Emilia.chatgpt4-malaysian-general-qa
Synthetic Malaysian QA
Generated common QA using ChatGPT4 based on Malaysia topics, notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/question-answer/chatgpt4-synthetic-malaysian-qa
General Malaysia topics
malaysian-general-qa.jsonl, 20396 rows, 28.6 MB.
malaysian-general-qa-v2.jsonl, 5294 rows, 8.05 MB.
malaysian-general-qa-v3.jsonl, 1368 rows, 5.09 MB.
malaysian-general-qa-v4.jsonl, 7733 rows. 36.2 MB.
malaysian-general-qa-v5.jsonl, 6363 rows… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/chatgpt4-malaysian-general-qa.