Datasetpublic

SThandarTint/myanmar-written-corpus

Myanmar Written Corpus The Myanmar Written Corpus is a comprehensive collection of high-quality, but not fully CLEAN, written Myanmar text, designed to address the lack of large-scale, openly accessible resources for Myanmar Natural Language Processing (NLP). It is tailored to support various tasks such as text-to-speech (TTS), automatic speech recognition (ASR), translation, text generation, and more. This dataset serves as a critical resource for researchers and developers… See the full description on the dataset page: https://huggingface.co/datasets/SThandarTint/myanmar-written-corpus.

sourceHugging Facecc0-1.0updated 1mo agoView on Hugging Face

0likes17downloads

Dataset Card

Myanmar Written Corpus

The Myanmar Written Corpus is a comprehensive collection of high-quality, but not fully CLEAN, written Myanmar text, designed to address the lack of large-scale, openly accessible resources for Myanmar Natural Language Processing (NLP). It is tailored to support various tasks such as text-to-speech (TTS), automatic speech recognition (ASR), translation, text generation, and more.

This dataset serves as a critical resource for researchers and developers aiming to advance Myanmar language technologies.

Dataset Overview

—Language: Myanmar
—Format: Parquet
—Size: 10 million sentences
—License: CC0 1.0
—Purpose: To support research and development in Myanmar NLP tasks.

Features

Field Name	Description
`sentence_id`	A unique numeric ID for each sentence.
`text`	The original written Myanmar sentence.
`string_length`	The number of characters in the sentence.
`length_category`	Categorization of the sentence length (e.g., `<100`, `<200`, etc.).

Dataset Statistics

—Total Sentences: 10 million.
—Character Length Ranges:
—<100: To be analyzed.
—<200: To be analyzed.
—<300: To be analyzed.
—... (Expand as needed).

Example Usage

The dataset can be loaded using the datasets library from Hugging Face:

python

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("freococo/myanmar-written-corpus")

# Access features
print(dataset["train"][0])  # Print the first example
print(dataset["train"][:5])  # Print the first 5 examples

# Access specific columns
texts = dataset["train"]["text"]  # Extract all sentences
lengths = dataset["train"]["string_length"]  # Extract string lengths

Applications

This dataset is suitable for:

—Training NLP models: Enables fine-tuning transformers for Myanmar language tasks.
—Text-to-Speech (TTS): Provides diverse text for TTS synthesis models.
—Automatic Speech Recognition (ASR): Aids in building language models for ASR systems.
—Machine Translation (MT): Offers a resource for Myanmar-to-other-languages translation models.
—Text Generation: Supports training for Myanmar text generation models.

License

This dataset is licensed under the CC0 1.0 License. You are free to use, modify, and distribute the dataset without restrictions.

Acknowledgments

This dataset is derived from publicly available sources, including:

—The Hugging Face FineWeb2 dataset, specifically its Myanmar subset.
—Original writers, speakers, and creators of the sentences.

The dataset has undergone extensive manual and automated processing to ensure quality and utility. These efforts included text cleaning, deduplication, and categorization.

Citation

If you use this dataset, please cite it as follows:


@dataset{myanmar_written_corpus,
  title = {Myanmar Written Corpus},
  author = {freococo},
  year = {2024},
  url = {https://huggingface.co/datasets/freococo/myanmar-written-corpus}
}