DatarrX/myanmar-Wikipedia

1---2license: cc-by-4.03task_categories:4- text-generation5- fill-mask6- token-classification7language:8- my9tags:10- burmese11- wikipedia12- nlp13- dataset14- low-resource15size_categories:16- 100K<n<1M17---18 19# Myanmar Wikipedia Dataset (20260501)20 21This dataset contains a cleaned, processed, and high-quality collection of Burmese Wikipedia articles, curated to serve as a robust foundation for Natural Language Processing (NLP) and Artificial Intelligence development in the Burmese language.22 23## 🏛️ About DatarrX24**DatarrX (Burmese: ဒေတာအက်စ်)** is a non-profit open-source foundation dedicated to building a robust digital foundation for the Burmese language in the AI era. We believe that high-quality data is the core of AI innovation. Our mission is to empower the Burmese AI ecosystem by building high-quality open-source datasets, essential AI tools, and localized technical resources.25 26---27 28## 🏗️ Dataset Creation Pipeline29 30This dataset was developed through a rigorous multi-stage cleaning and preparation process to ensure maximum utility for researchers and developers:31 321.  **Data Extraction:** The source material was derived from the official [Burmese Wikipedia Database Dump](https://dumps.wikimedia.org/mywiki/) (2026-05-01).332.  **Markup Sanitization:** We systematically removed MediaWiki markup, HTML tags, template artifacts (`{{...}}`), and internal links (`[[...]]`).343.  **Filtering & Noise Reduction:** * Applied strict Zawgyi-to-Unicode verification to ensure data consistency.35    * Removed tabular metadata, non-informative CSS/module snippets, and malformed rows.36    * Filtered out rows containing non-Burmese characters (where applicable) and low-information symbols.374.  **Sentence Segmentation & Normalization:** * Articles were split into individual sentences. 38    * We enforced a standard Burmese full-stop (`။`) at the end of every sentence to ensure structural consistency.395.  **Syllable Tokenization:** Using `mm-syllable`, every sentence was segmented into its constituent Burmese syllables, providing a secondary column (`syllable`) for syllable-level model training.40 41---42 43## 📊 Dataset Statistics44* **Total Sentences:** 546,04545* **Average Sentence Length:** 122.98 characters46* **Average Syllable Count:** 39.1147* **Structural Consistency:** 99.90% of sentences end with the standard Burmese full-stop (`။`).48 49### Data Structure50| Column | Description |51| :--- | :--- |52| `text` | The clean, normalized Burmese sentence. |53| `syllable` | The same sentence segmented into syllables with spaces. |54 55**Sample Entry:**56```json57{58  "text": "အကြောင်းအရာပေါင်း ခုကို ဖြင့် ဖတ်ရှုနိုင်ပါသည်။",59  "syllable": "အ ကြောင်း အ ရာ ပေါင်း ခု ကို ဖြင့် ဖတ် ရှု နိုင် ပါ သည် ။",60  "text_len": 47,61  "syllable_count": 1462}63```64 65---66 67## 👨‍💻 Creator & Contributors68 69This dataset is curated and maintained by **Khant Sint Heinn (Burmese: ခန့်ဆင့်ဟိဏ်း, Kalix Louis)**.70 71Khant Sint Heinn is a Machine Learning Engineer specializing in NLP and open-source AI development. As the Lead Developer at **DatarrX**, he focuses on building scalable data pipelines and practical infrastructure to improve support for low-resource languages. His work aims to transform limited language resources into practical opportunities through clean data and community-driven innovation.72 73---74 75## ⚖️ License & Citation76 77This work is released under the **Creative Commons Attribution 4.0 International (CC-BY-4.0)** license, ensuring openness and collaboration for the global AI research community.78 79If you find this dataset useful for your research or applications, please cite it as:80 81```bibtex82@misc{datarrx_my_wiki_2026,83  author       = {Khant Sint Heinn},84  title        = {yanmar Wikipedia Dataset (20260501)},85  year         = {2026},86  publisher    = {Hugging Face},87  organization = {DatarrX},88  howpublished = {https://huggingface.co/datasets/DatarrX/myanmar-Wikipedia},89  note         = {Published under DatarrX. Open-source community asset released under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)}90}91 92```