Datasetpublic

mesolitica/Malaysian-STT-Whisper-Stage2

Malaysian STT Whisper Stage 2 Extra dataset to compliment mesolitica/Malaysian-STT-Whisper. This dataset is stronger in confidence and suitable for second stage / annealing finetuning. how to prepare the dataset huggingface-cli download \ mesolitica/Malaysian-STT-Whisper-Stage2 \ --include "*.zip" \ --repo-type "dataset" \ --local-dir './' huggingface-cli download \ mesolitica/Malaysian-Multiturn-Chat-Assistant \ --include "*.zip" \ --exclude "voice/*.zip" \… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper-Stage2.

sourceHugging Faceupdated 1y agoView on Hugging Face

2likes380downloads

Dataset Card

Malaysian STT Whisper Stage 2

Extra dataset to compliment mesolitica/Malaysian-STT-Whisper.

This dataset is stronger in confidence and suitable for second stage / annealing finetuning.

how to prepare the dataset

bash

huggingface-cli download \
mesolitica/Malaysian-STT-Whisper-Stage2 \
--include "*.zip" \
--repo-type "dataset" \
--local-dir './'

huggingface-cli download \
mesolitica/Malaysian-Multiturn-Chat-Assistant \
--include "*.zip" \
--exclude "voice/*.zip" \
--repo-type "dataset" \
--local-dir './'

huggingface-cli download \
mesolitica/Malaysian-UltraChat-Speech-Multiturn-Instructions \
--include "ultrachat-speech-*.zip" \
--repo-type "dataset" \
--local-dir './'

huggingface-cli download \
mesolitica/STT-Normalizer \
--include "prepare-dataset-normalizer-text-*.zip" \
--repo-type "dataset" \
--local-dir './'

huggingface-cli download \
mesolitica/STT-Normalizer \
--include "prepare-dataset-normalizer-text-chat-assistant-*.zip" \
--exclude "*force-alignment.zip" \
--repo-type "dataset" \
--local-dir './'

huggingface-cli download \
mesolitica/STT-Normalizer \
--include "prepare-dataset-normalizer-text-combine-*.zip" \
--exclude "*force-alignment.zip" \
--repo-type "dataset" \
--local-dir './'

huggingface-cli download \
mesolitica/STT-Normalizer \
--include "prepare-dataset-normalizer-text-english-news-*.zip" \
--exclude "*force-alignment.zip" \
--repo-type "dataset" \
--local-dir './'

huggingface-cli download \
mesolitica/STT-Normalizer \
--include "prepare-dataset-normalizer-text-malay-news-*.zip" \
--exclude "*force-alignment.zip" \
--repo-type "dataset" \
--local-dir './'

huggingface-cli download \
mesolitica/STT-Normalizer \
--include "prepare-dataset-normalizer-text-malay-news-part2-*.zip" \
--exclude "*force-alignment.zip" \
--repo-type "dataset" \
--local-dir './'

wget https://gist.githubusercontent.com/huseinzol05/2e26de4f3b29d99e993b349864ab6c10/raw/9b2251f3ff958770215d70c8d82d311f82791b78/unzip.py
python3 unzip.py

Test set

We uploaded Malaysia Hansard test set at mesolitica/Speech-Benchmark/malaysia-hansard

Source code

Source code at https://github.com/malaysia-ai/dataset/tree/main/stt-whisper