paodigitalhub/pao-sentences-dataset

Pa'O Sentences Dataset is a text corpus for Pa'O language (ပအိုဝ်ႏ) containing structured line-by-line sentences designed for NLP, LLM pre-training, and machine translation.

📝 Pa'O Sentences Dataset (ပအိုဝ်ႏ လိက်လာႏငေါဝ်းရဲဉ်ႏ ရွမ်ခြွဉ်းဗူႏ)

📌 Project Summary (ထာꩻမာꩻခြပ်ရဲဉ်ႏ နပ်ထွားရဲပ်အအဲဉ်ႏ)

The Pa'O Sentences Dataset is an open-source textual corpus developed to support Natural Language Processing (NLP), Large Language Model (LLM) pre-training, Machine Translation, and speech text corpus development for the Pa'O language (ပအိုဝ်ႏဘာႏသာႏ).

As a low-resource language primarily spoken in Shan State and other regions of Myanmar, this dataset provides structured line-by-line sentences suitable for language modeling, Text-to-Speech (TTS) prompt selection, and general AI research.

Dataset ယိုနောဝ်ꩻ ပအိုဝ်ႏဘာႏသာႏငေါဝ်းငွါအတာႏ Natural Language Processing (NLP)၊ AI / LLM မော်ဒယ်များ လေ့ကျင့်တည်ဆောက်ခြင်းနှင့် စာသားဆိုင်ရာ သုတေသနများတွင် အသုံးပြုနိုင်ရန် စနစ်တကျ စုဆောင်းထားသော မူပိုင်ခွင့်လွတ် (Open-Source) Text Dataset ဖြစ်ပါသည်။

💡 How to Use (သွုံꩻဗာႏနုဲင်းမုဲင်ꩻ)

1. Install Dependencies

pip install -U datasets pandas pyarrow

2. Load Dataset in Python (Option A: Using Hugging Face Datasets)

from datasets import load_dataset

# Load Pa'O Sentences Dataset
dataset = load_dataset("paodigitalhub/pao-sentences-dataset")

# Print dataset split info
print(dataset)

# Inspect first 5 sentences
for i in range(5):
    print(f"[{i+1}]", dataset["train"][i]["text"])

3. Load Dataset in Python (Option B: Using Pandas)

import pandas as pd

# Direct load via Pandas from Hugging Face Parquet file
df = pd.read_parquet("hf://datasets/paodigitalhub/pao-sentences-dataset/data/pao-sentences-001.parquet")

# Inspect first 5 rows
print(df.head())

Load All Sentences (ထူႏလို့ လိတ်ရဲဉ်ႏ)

from datasets import load_dataset

# Load the complete Pa'O Sentences Dataset
dataset = load_dataset("paodigitalhub/pao-sentences-dataset")

# Get all sentences from the train split
sentences = dataset["train"]["text"]

print(f"Total sentences: {len(sentences)}")

# Print all sentences
for i, sentence in enumerate(sentences, 1):
    print(f"[{i}] {sentence}")

📊 Data Fields (အမုဲင်ထွော့သေꩻ)

The dataset consists of a single text field containing line-by-line Pa'O sentences:

🎯 Intended Use Cases (ကသွုံꩻနွောင်ꩻဒါႏ အဖဲ့ꩻထွောင်းဖုံႏ)

• —Language Modeling & LLM Pre-training: Training Tokenizers, Masked Language Models (e.g., BERT, RoBERTa), and LLM fine-tuning for Pa'O.
• —Text-to-Speech (TTS) Corpus: Selecting prompt sentences for Pa'O voice recording and speech synthesis.
• —Computational Linguistics: Monolingual text corpus research, grammar analysis, and machine translation models.

📁 Repository Structure (တဲ့ꩻဆို့ꩻဖုဲင်လာႏ အရွီးခိုႏပွုံႏ)

To ensure automatic parsing by Hugging Face datasets loaders, all sentence files are structured inside the data/ directory:

pao-sentences-dataset/
├── data/
│   ├── pao-sentences-001.parquet
│   └── pao-sentences-002.parquet
├── .gitattributes
└── README.md

👨‍💻 Key Contributors & Acknowledgments (ကေားကတဲမ်းရွမ်သားယို ကေႏဇူꩻတင်ႏခရာႏ)

This textual dataset has been collected, curated, and maintained by:

• —KhunPhanDuae (ခွန်ဖန်ဒွဲ့)
• —KhunRorNa (ခွန်ရောန)

Pa'O Digital Hub is a project being organized and carried out by SuccessImprove under RYPAK.

🌐 External Resources & Useful Links (ထောင်ထွားခင်ႏလမ်းထန်ႏဖဲ့ꩻတာႏ လိဉ့်ဖြွီꩻ)

• —Pa'O Language Documentation & Resource Hub: KhunPhanDuae - Pa-O-Language Website

📜 License

This dataset is released under the Creative Commons Attribution 4.0 International (CC-BY-4.0) license.