Datasetpublic

aisingapore/WangchanLION-Curated

Citation @misc{phatthiyaphaibun2025mangosteenopenthaicorpus, title={Mangosteen: An Open Thai Corpus for Language Model Pretraining}, author={Wannaphong Phatthiyaphaibun and Can Udomcharoenchaikit and Pakpoom Singkorapoom and Kunat Pipatanakul and Ekapol Chuangsuwanich and Peerat Limkonchotiwat and Sarana Nutanong}, year={2025}, eprint={2507.14664}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2507.14664}, }… See the full description on the dataset page: https://huggingface.co/datasets/aisingapore/WangchanLION-Curated.

sourceHugging Faceodc-byupdated 1y agoView on Hugging Face

4likes239downloads

Dataset Card

Citation

@misc{phatthiyaphaibun2025mangosteenopenthaicorpus,
      title={Mangosteen: An Open Thai Corpus for Language Model Pretraining}, 
      author={Wannaphong Phatthiyaphaibun and Can Udomcharoenchaikit and Pakpoom Singkorapoom and Kunat Pipatanakul and Ekapol Chuangsuwanich and Peerat Limkonchotiwat and Sarana Nutanong},
      year={2025},
      eprint={2507.14664},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.14664}, 
}

We have collected additional Thai text that is unlikely to be included in the common crawl from various sources. The total number of documents collected is as follows:425,304 documents, we deduplication these noncc documents to later divide them into train sets for further web data and validation set.

The following table shows the analysis of the data after deduplication.

type	Source	Number of documents	Number of words	Cultural?	Human-craft?	Evaluation topics?
encyclopedic	th.wikibooks.org	1,179	756,175	No		Common safety
	th.wikipedia.org	162,189	74,151,797	No		Common safety
	th.wikiquote.org	929	156,910	No		Common safety
	th.wikisource.org	1,890	5,601,908	No		Common safety
finance	airesearch/cmdf_vistec	86,813	348,189,390	No		Common safety
government document	lift catalog data	1	6,931	No	Yes/No	Common safety
	data.go.th	9,488	2,766,950	No	No	Common safety
	envilink.go.th	1	10,659
	government data catalog smart plus	427	5,290,950	Yes	Yes/No	Common safety/ Cross-lingual
	https://ratchakitcha.soc.go.th	59,744	175,031,218	Yes	Yes	Culture safety/ Cross-lingual
	nakhoratchasima data catalog	1	68,000	Yes	Yes	Country safety/ Cross-lingual
	opdc data portal	3,148	598,829
	open-d	7	284,100
	royal thai government	1	41,356
	National Economic and Social Development Board	1	37,663
	pythainlp/thailand-policy-statements	60	226,087
legal	pythainlp/thai-cc-license	6	50,727
	pythainlp/thai-constitution-corpus	20	444,313
	pythainlp/thailaw-v1.0	52,317	79,715,118
academic literature	government data catalog smart plus	427	5,290,950
	openbase.in.th	4,173	165,909,425
	platform for social empowerment and transformation	90	209,716
	pythainlp/thai-it-books	7	174,644
	pythainlp/thai-tnhc2-books	353	22,002,703
	pythainlp/tlcv2.0_oa	361	2,970,463
	TDRI	25	2,801,106
	Bangkok Open Data	10	2,334
	Open educational resources repository	14	47,951
	CMU Journal of Law and Social Sciences	47	37,976
	E-journal of education studies, Burapha University	68	59,137
	Chulalongkorn University Law Journal	64	46,425
	Lanna Journal of Health Promotion and Environmental Health	53	52,320
	Journal of Educational Studies, Burapha University	64	56,320
	Journal of Yanasangwon Research Institute	65	47,286
	Journal of Food and Drug Administration	79	115,541
	https://github.com/kongruksiamza/ebook-for-education	8	83,432
	social technology institute	3	11,152
youtube	youtube	17,826	46,613,632

Documents from some sources cannot be directly used or easily processed. It is necessary to use text extraction technology (OCR) to extract the text due to the document format in PDF. The table below shows the number and proportion of documents that required OCR.

Source	Number of documents	Number of documents required for PCR	Percentage of documents requiring OCR
data.go.th	9488	18	0.189713
government data catalog smart plus	427	198	46.370023
ebook construction	8	8	100
openbase.in.th	4173	3443	82.50659
opendata.nesdc.go.th	7	3	42.857143
royal thai government	1	1	100
Open educational resources repository	14	2	14.285714

The model used for text extraction ishttps://github.com/VikParuchuri/marker In the future, you should try VLM, such as:https://olmocr.allenai.org/Or Typhoon2 Vision

Data from various sources can be classified into 6 types as shown in the table below.

Domain	count	proportion
Encyclopedic	166187	41.34
Finance	86813	21.59
Government	72,879	18.13
Legal	52,343	13.02
YouTube	17,837	4.43
Education	5,911	1.47

The additional documents we collect are confirmed to be open source and have a license to allow for redistribution, with the copyright share as shown in the table below.

license	count	proportion
CC BY-SA 4.0	166187	41.388233
CC0	112871	28.110088
CC BY 4.0	112407	27.994531
CC BY-NC-SA 4.0	4173	1.03927
ODC-BY	3769	0.938655
CC BY-NC 4.0	1853	0.461483
CC BY-NC-ND 4.0	250	0.062262
CC BY 3.0	13	0.003238
GFDL	6	0.001494
OGL	3	0.000747

Resources

—Pre-training data (web): https://huggingface.co/datasets/aisingapore/WangchanLION-Web
—Pre-training data (curated): https://huggingface.co/datasets/aisingapore/WangchanLION-Curated
—Pre-training model: https://huggingface.co/aisingapore/WangchanLION-v3
—SFT model: https://huggingface.co/aisingapore/WangchanLION-v3-IT
—Paper: https://arxiv.org/abs/2507.14664
—Blog: https://sea-lion.ai/sea-lion-wangchanlionv3/
—Github: https://github.com/vistec-AI/Mangosteen