datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
mixtral-factual-QA
Mixtral Factual QA
Generate questions and answers based on context provided. We use contexts from,
maktabahalbakri.com
muftiwp.gov.my
asklegal.my
dewanbahasa-jdbp
gov.my
patriots
rootofscience
majalahsains
nasilemaktech
alhijrahnews
https://huggingface.co/datasets/open-phi/textbooks
notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/question-answer/mixtral-factual
factually-wrong-qa-coding.jsonl, 31253 rows, 425 MB
factually-wrong-qa.jsonl, 1108037 rows, 10… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/mixtral-factual-QA.chatgpt4-malaysian-general-qa
Synthetic Malaysian QA
Generated common QA using ChatGPT4 based on Malaysia topics, notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/question-answer/chatgpt4-synthetic-malaysian-qa
General Malaysia topics
malaysian-general-qa.jsonl, 20396 rows, 28.6 MB.
malaysian-general-qa-v2.jsonl, 5294 rows, 8.05 MB.
malaysian-general-qa-v3.jsonl, 1368 rows, 5.09 MB.
malaysian-general-qa-v4.jsonl, 7733 rows. 36.2 MB.
malaysian-general-qa-v5.jsonl, 6363 rows… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/chatgpt4-malaysian-general-qa.chatgpt-malaysian-open-qa
Synthetic Malaysian Open QA
Generated using ChatGPT3.5 on MS Wikipedia, MS Common Crawl and Malaysia Hansard, notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/question-answer/chatgpt3.5-wikipedia
common-crawl-qa.jsonl, 69829 rows, 291 MB.
hansard-qa.jsonl, 42368 rows, 344 MB.
wikipedia-qa.jsonl, 44923 rows, 238 MB.
Example data
{'paragraph': "PANDAN JAYA SITI AISHAH KUALA LUMPUR SUHANI KUALA LUMPUR SUMI BANDAR TUN RAZAK SYAHIDA TAMAN SEPAKAT (AU… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/chatgpt-malaysian-open-qa.chatgpt4-commonsense-qa
Synthetic CommonSense
Generated using ChatGPT4, originally from https://huggingface.co/datasets/commonsense_qa
Notebook at https://github.com/mesolitica/malaysian-dataset/tree/master/question-answer/chatgpt4-commonsense
synthetic-commonsense.jsonl, 36332 rows, 7.34 MB.
Example data
{'question': '1. Seseorang yang bersara mungkin perlu kembali bekerja jika mereka apa?\n A. mempunyai hutang\n B. mencari pendapatan\n C. meninggalkan pekerjaan\n D. memerlukan… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/chatgpt4-commonsense-qa.chatgpt-malaysian-qa-choice
Synthetic QA Choice
Generated using ChatGPT3.5,
https://huggingface.co/datasets/malaysia-ai/dedup-text-dataset/resolve/main/dewanbahasa-jdbp.jsonl
https://huggingface.co/datasets/malaysia-ai/dedup-text-dataset/resolve/main/majalahsains.jsonl
https://huggingface.co/datasets/malaysia-ai/dedup-text-dataset/resolve/main/wikipedia-2023-10-01.jsonl
Notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/question-answer/chatgpt3.5-qa-choice
qa-dewanbahasa-jdbp.jsonl… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/chatgpt-malaysian-qa-choice.chatgpt-malaysian-extractive-qa
Synthetic Malaysian Extractive QA
Generated using ChatGPT3.5, notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/question-answer/chatgpt3.5-extractive-news