chuuhtetnaing/myanmar-cc100-dataset

Please visit to the [GitHub repository](https://github.com/chuuhtetnaing/myanmar-language-dataset-collection) for other Myanmar Langauge datasets.

Myanmar CC100 Dataset

A preprocessed subset of the CC100 dataset containing only Myanmar language text, with consistent Unicode encoding.

Dataset Description

This dataset is derived from the statmt/cc100 created by "Statistical and Neural Machine Translation". It contains only the Myanmar language portion of the original CC100 dataset, with additional preprocessing to standardize text encoding.

Preprocessing

The main preprocessing step applied to this dataset was:

• —Zawgyi to Unicode conversion: Myanmar text can be encoded in two different ways - Zawgyi and Unicode. We detected Zawgyi-encoded text and converted it to Unicode for consistency, ensuring all text in the dataset uses the same encoding standard.

The conversion was performed using Myanmar Tools for detection and ICU for transliteration:

from myanmartools import ZawgyiDetector
from icu import Transliterator

# Initialize the detector and converter
detector = ZawgyiDetector()
converter = Transliterator.createInstance('Zawgyi-my')

# Example conversion function
def zawgyi_to_unicode(text):
    score = detector.get_zawgyi_probability(text)
    if score > 0.5:  # If likely Zawgyi
        return converter.transliterate(text)
    return text  # Already Unicode

Dataset Structure

The dataset contains only one field:

• —text: The main text content in Myanmar language (Unicode encoded)

Usage

You can load this dataset using the Hugging Face datasets library:

from datasets import load_dataset

dataset = load_dataset("chuuhtetnaing/myanmar-cc100-dataset")

Dataset Creation

This dataset was created by:

1. 1.Extracting the Myanmar language content from the original statmt/cc100 dataset, which includes both Unicode ('my') and Zawgyi ('my_zaw') encoded text
2. 2.Detecting Zawgyi-encoded text using Google's Myanmar Tools probabilistic detector
3. 3.Converting Zawgyi text to Unicode using ICU's transliteration converter
4. 4.Creating a 90-10 train-test split of the combined data

Dependencies

The preprocessing of this dataset relied on:

• —Myanmar Tools - For Zawgyi detection
• —PyICU - For Zawgyi to Unicode conversion
• —Documentation - Documentation from Myanmar Tools

License

This dataset follows the same license as the original CulturaX dataset. Please refer to the original dataset page for licensing information.

Myanmar Tools is released under the Apache License 2.0.