botisan-ai/cantonese-mandarin-translations

Dataset Card for cantonese-mandarin-translations

Table of Contents

• —Table of Contents
• —Dataset Description
• —Dataset Summary
• —Supported Tasks and Leaderboards
• —Languages
• —Dataset Structure
• —Data Instances
• —Data Fields
• —Data Splits
• —Dataset Creation
• —Curation Rationale
• —Source Data
• —Annotations
• —Personal and Sensitive Information
• —Considerations for Using the Data
• —Social Impact of Dataset
• —Discussion of Biases
• —Other Known Limitations
• —Additional Information
• —Dataset Curators
• —Licensing Information
• —Citation Information
• —Contributions

Dataset Description

• —Homepage:
• —Repository:
• —Paper:
• —Leaderboard:
• —Point of Contact:

Dataset Summary

This is a machine-translated parallel corpus between Cantonese (a Chinese dialect that is mainly spoken by Guangdong (province of China), Hong Kong, Macau and part of Malaysia) and Chinese (written form, in Simplified Chinese).

Supported Tasks and Leaderboards

N/A

Languages

• —Cantonese (yue)
• —Simplified Chinese (zh-CN)

Dataset Structure

JSON lines with yue field and zh field for the parallel corpus.

Data Instances

N/A

Data Fields

• —yue: Cantonese corpus
• —zh: translated Chinese corpus

Data Splits

No data splitting is done as of yet.

Dataset Creation

The dataset is produced by doing the following:

• —Download HKCancor Cantonese Corpus and CommonVoice Cantonese (Hong Kong Chinese `yue`) text corpus
• —Extract text corpus and merge datasets
• —Run text against Microsoft's Translator API from yue to zh-Hans

Curation Rationale

Currently no such corpus exists, and it is hard to find such a corpus, so we tried to generate a reasonable batch of samples using machine translation for research purposes.

Source Data

• —HKCancor
• —CommonVoice 7.0 Chinese (Hong Kong)

Initial Data Collection and Normalization

Normalization scripts will be included soon.

Who are the source language producers?

• —HKCancor
• —CommonVoice 7.0 Chinese (Hong Kong)

Annotations

Annotation process

We run the Cantonese text corpus against Microsoft's Translator API.

Who are the annotators?

• —Microsoft's Translator API

Personal and Sensitive Information

N/A

Considerations for Using the Data

Social Impact of Dataset

We would like to share this parallel corpus and welcome contributions to preserve the Cantonese dialect.

Discussion of Biases

N/A

Other Known Limitations

This parallel corpus is machine-translated, it is not 100% accurate.

Additional Information

Dataset Curators

• —Botisan AI
• —Haoran (Simon) Liang

Licensing Information

CC BY-NC-SA 4.0

Citation Information

@misc {botisanAiCantoneseMandarinTranslationsDatasets,
    author = {Liang, H.},
    title = {Cantonese Mandarin Translations Dataset},
    year = {2021},
    url = {https://huggingface.co/datasets/botisan-ai/cantonese-mandarin-translations},
}

Contributions

Thanks to @lhr0909 for adding this dataset.