IKMLab-team/hk_content_corpus

HK Content Corpus (Cantonese \& Traditional Chinese)

Dataset Description

• —Language: Hong Kong Cantonese, Traditional Chinese
• —Size: 9.44GB
• —Source: public web sources (news sites, online forums, encyclopedia and restaurant reviews).

This dataset contains eight cleaned source-specific corpora of Hong Kong Cantonese and Traditional Chinese text, crawled from public websites and platforms.

It was initially created for the experiments reported in https://doi.org/10.1145/3744341 which study the effect of diglossia on Hong Kong language modeling.

Each file stores plain UTF-8 text, where each record occupies one line, and blank lines serve as separators.

This dataset is also available at Zenodo: https://doi.org/10.5281/zenodo.16882351

We only change file extension from .corpus to .csv and add header row here for HuggingFace's dataset viewer function

👉 This cleaned corpus is derived from a larger MySQL database used to store raw text during the data collection stage. If you need the original database for reprocessing or reproduction, please refer to: https://huggingface.co/datasets/IKMLab-team/hkcontentcorpus_mysql

Files

Intended Uses

• —Language model pretraining and finetuning
• —Hong Kong Cantonese/Traditional Chinese linguistic modeling
• —Downstream tasks such as classification or generation

NOTE: HKNSL became effective since 2020-6-30, which can create bias on user content created afterwards. Those portion of data should be used with caution.

Citation

If you use this dataset, please cite the following paper:

@article{Yung2025HKDiglossia,
  author    = {Yung, Yiu Cheong and Lin, Ying-Jia and Kao, Hung-Yu},
  title     = {Exploring the Effectiveness of Pre-training Language Models with Incorporation of Diglossia for Hong Kong Content},
  journal   = {ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP)},
  volume    = {24},
  number    = {7},
  pages     = {71:1--71:16},
  year      = {2025},
  publisher = {Association for Computing Machinery},
  doi       = {10.1145/3744341}
}

and optionally also cite the dataset DOI:

@dataset{yung_2025_16882351,
  author       = {Yung, Yiu Cheong},
  title        = {HK Content Corpus (Cantonese \& Traditional Chinese)},
  month        = aug,
  year         = 2025,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.16882351},
  url          = {https://doi.org/10.5281/zenodo.16882351},
}