jed351/Chinese-Common-Crawl-Filtered
Traditional Chinese C4 Dataset Summary Data obtained from 2025-18 and 2025-13 Common Crawl. Downloaded and processed using code based on another project attempting to recreate the C4 dataset. The resultant dataset contains both simplified and traditional Chinese. It was then filtered using a modified list of simplified Chinese characters to obtain another traditional Chinese dataset. I am still ironning out the process of filtering. The 2025-13 dataset was… See the full description on the dataset page: https://huggingface.co/datasets/jed351/Chinese-Common-Crawl-Filtered.
Traditional Chinese C4
Dataset Summary
Data obtained from 2025-18 and 2025-13 Common Crawl.
Downloaded and processed using code based on another project attempting to recreate the C4 dataset.
The resultant dataset contains both simplified and traditional Chinese. It was then filtered using a modified list of simplified Chinese characters to obtain another traditional Chinese dataset.
I am still ironning out the process of filtering. The 2025-13 dataset was deduplicated without splitting into shards resulting in a smaller dataset than the 2025-18 dataset.
Unfortunately, this process takes 2 CPU nodes for 4 hours. In addition, other people would have done a better job at cleaning the common crawl.
To preserver time, money and resources, I will not work on this dataset anymore but instead shift my focus to the Traditional Chinese Common Crawl dataset.