guiti999/cantonese-traditional-chinese-parallel-corpus-gen3
Cantonese-Written Chinese Parallel Dataset (3rd Generation) About the Dataset Data Splits Training Data (train): 160,000 Sentence Pairs Validation Data (validation): 20,000 Sentence Pairs Test Data (test): 5,461 Sentence Pairs Languages Cantonese (yue) Traditional Chinese (zh-TW) Original Data Structure JSON lines consisting of yue, zh and ref fields. Data Source LIHKG HKCancor Cantonse-Mandarin… See the full description on the dataset page: https://huggingface.co/datasets/guiti999/cantonese-traditional-chinese-parallel-corpus-gen3.
Cantonese-Written Chinese Parallel Dataset (3rd Generation)
About the Dataset
Data Splits
- Training Data (
train): 160,000 Sentence Pairs - Validation Data (
validation): 20,000 Sentence Pairs - Test Data (
test): 5,461 Sentence Pairs
Languages
- Cantonese (
yue) - Traditional Chinese (
zh-TW)
Original Data Structure
- JSON lines consisting of
yue,zhandreffields.
Data Source
- LIHKG
- HKCancor
- Cantonse-Mandarin Translations
- and various online public datasets
Written Chinese are translated by Microsoft Bing and DeepSeek V3, then translated to Traditional Chinese. Test Set's reference data are translated by a group of professional human translators.
Considerations for Using the Data
Social Impact of Dataset
This dataset is served as a way to preserve Cantonese language.
Discussion of Biases
As the data mostly scrapped from LIHKG, there may be biases on the contents.
Known Limitations
Most contents are machine-translated by Microsoft Bing and DeepSeek V3, the translation may not be 100% accurate.
Other Information
Dataset Curator
License
Citation
@inproceedings{10.1145/3639233.3639332,
author = {Kwok, Raptor Yick-Kan and Au Yeung, Siu-Kei and Li, Zongxi and Hung, Kevin},
title = {Cantonese to Written Chinese Translation via HuggingFace Translation Pipeline},
year = {2024},
isbn = {9798400709227},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3639233.3639332},
doi = {10.1145/3639233.3639332},
abstract = {Cantonese, a low-resource language [5] that has been used in Southeastern China for hundreds of years, with over 85 million native speakers worldwide, is poorly supported in the mainstream language model for existing translation platforms such as Baidu, Google and Bing. This paper presents a large parallel corpus of 130 thousand Cantonese and Written Chinese pairs. The data are used to train a translation model using the translation pipeline of the Hugging Face Transformers architecture, a dominant architecture for natural language processing nowadays [18]. The BLEU score and manual assessment evaluate the performance. The translation results achieve a BLEU score of 41.35and chrF++ score of 44.88on the entire validation set. The model also works reasonably well with long sentences of over 20 Chinese characters. It achieves a BLEU score of 48.61and chrF++ score of 39.87on long sentences. Those results are comparable with the existing Baidu Fanyi and Bing Translate. We also establish a Cantonese sentence evaluation metric to classify the quality of the source Cantonese sentence by professional translators. We then compare the BLEU and chrF++ scores with the corresponding evaluation score and found that the better the quality of the source sentence, the higher the BLEU and chrF++ scores. Last, we proved that our corpus enabled the Cantonese translation capability of the Chinese BART pre-trained model.},
booktitle = {Proceedings of the 2023 7th International Conference on Natural Language Processing and Information Retrieval},
pages = {77–84},
numpages = {8},
keywords = {Cantonese, Written Chinese, neural networks, translation},
location = {Seoul, Republic of Korea},
series = {NLPIR '23}
}
@misc {cantoneseWrittenChineseTranslationsGen3,
author = {Kwok, R.},
title = {Cantonese Written Chinese Parallel Corpus (3rd Generation)},
year = {2024},
url = {https://huggingface.co/datasets/raptorkwok/cantonese-traditional-chinese-parallel-corpus-gen3},
}