datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Malay-Dialect-Instructions
Malay dialect instruction including coding
Negeri Sembilan
QA
public transport QA,
Coding
CUDA coding,
Kedah
QA
infra QA,
Coding
Rust coding,
Kelantan
QA
Najib Razak QA,
Coding
Go coding,
Perak
QA
Anwar Ibrahim QA,
Coding
SQL coding,
Pahang
QA
Pendatang asing QA,
Coding
Typescript coding,
Terengganu… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/Malay-Dialect-Instructions.instructions-pair-miningmixtral-magicoder
Mixtral Magicoder: Source Code Is All You Need on various programming languages
We sampled programming languages from https://huggingface.co/datasets/bigcode/the-stack-dedup and pushed to https://huggingface.co/datasets/malaysia-ai/starcoderdata-sample
After that, we use Magicoder: Source Code Is All You Need on various programming languages template, we target at least 10k rows for each programming languages.
C++, 10747 rows
C#, 10193 rows
CUDA, 13843 rows
Dockerfile, 13286 rows… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/mixtral-magicoder.malaysian-rerankerMesoMathematics
MesoMathematics
Frozen data artifacts for the paper “Mathematical Knowledge at the
Mesoscale: Organization after the Formal Mathematics Revolution” by Andrea
E. V. Ferrari, Benjy Firester, Xinze Li, Simone Severini, and Patrick Shafto.
The corresponding source code, exact commands, and manuscript live in the
MathNetwork/MesoMathematics
repository.
This release fixes Mathlib at v4.33.0, commit
db584cd6d46c92f209a44c0f1c829460d327499d, with Lean v4.33.0. The full
commit, not the… See the full description on the dataset page: https://huggingface.co/datasets/CloKTech/MesoMathematics.mixtral-factual-QA
Mixtral Factual QA
Generate questions and answers based on context provided. We use contexts from,
maktabahalbakri.com
muftiwp.gov.my
asklegal.my
dewanbahasa-jdbp
gov.my
patriots
rootofscience
majalahsains
nasilemaktech
alhijrahnews
https://huggingface.co/datasets/open-phi/textbooks
notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/question-answer/mixtral-factual
factually-wrong-qa-coding.jsonl, 31253 rows, 425 MB
factually-wrong-qa.jsonl, 1108037 rows, 10… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/mixtral-factual-QA.standard-malay-translation-instructionstranslated-xwikischatgpt4-commonsense-qa
Synthetic CommonSense
Generated using ChatGPT4, originally from https://huggingface.co/datasets/commonsense_qa
Notebook at https://github.com/mesolitica/malaysian-dataset/tree/master/question-answer/chatgpt4-commonsense
synthetic-commonsense.jsonl, 36332 rows, 7.34 MB.
Example data
{'question': '1. Seseorang yang bersara mungkin perlu kembali bekerja jika mereka apa?\n A. mempunyai hutang\n B. mencari pendapatan\n C. meninggalkan pekerjaan\n D. memerlukan… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/chatgpt4-commonsense-qa.translated-cnn-dailymailmixtral-malaysian-abstractive-summarization
Mixtral Malaysian Abstractive Summarization
Use https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1 to generate abstractive summarization on Malaysian dataset, notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/summarization/mixtral
Example data
{'source': 'gamerbraves.com.jsonl',
'text': 'Hunter x Hunter USJ Collaboration Announced\n\n\nUniversal Studios Japan ( USJ ) has announced a collaboration with popular Shonen anime Hunter x… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/mixtral-malaysian-abstractive-summarization.pseudolabel-malaya-speech-stt-train-whisper-large-v3translated-glaive-function-callOpenAI-embedding-ada-002google-translate-camel-aitranslated-LLaVA-Instruct-150Kembedding-pair-miningchatgpt4-code-instruct
ChatGPT4 Code Instruct
Originally from https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1, translate and answer using ChatGPT4.
Notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/chatbot/chatgpt4-code-instruct
synthetic-codealpaca-v1-chatgpt4.jsonl, 43482 rows, 274 MB.
Example data
{'instruction': "Harap ubah skrip Python berikut agar ia memasukkan pengulangan 'while' daripada pengulangan 'for' yang sedia ada, yang meneruskan… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/chatgpt4-code-instruct.Azure-TTS-Yasmin-WikipediaAzure-TTS-Osman-Wikipediachatgpt4-noisy-translation-twitter-dialect
ChatGPT 4 Noisy Translation Twitter to local dialect
Notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/translation/chatgpt4-twitter-dialect
translated-MMLU
Translated MMLU
Originally from https://huggingface.co/datasets/cais/mmlu, translated to Malay using Google Translate.
Precaution
We found out some translated answers not really coherent with original English answers, so it is better to skip translated answers.
rumi-jawi-instructionszeroshot-NERbge-large-en-embeddingdependency-parsing-instructionssynthetic-word-switching-translation
synthetic-word-switching-translation
Notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/translation/synthetic-word-switching-translation
maksud-instructionschatgpt-malaysian-extractive-qa
Synthetic Malaysian Extractive QA
Generated using ChatGPT3.5, notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/question-answer/chatgpt3.5-extractive-news
translated-LLaVA-Pretrain