Chinese
Datasets
All datasets matching “Chinese”Fineweb-Edu-Chinese-V2.1
Chinese Fineweb Edu Dataset V2.1 [中文] [English]
[OpenCSG Community] [👾github] [wechat] [Twitter]
📖Technical Report
The Chinese Fineweb Edu Dataset V2.1 is an enhanced version of the V2 dataset, designed specifically for natural language processing (NLP) tasks in the education sector. This version introduces two new data sources, map-cc and opencsg-cc, and retains data with scores ranging from 2 to 3. The dataset entries are organized into different… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/Fineweb-Edu-Chinese-V2.1.chinese-fineweb-edu
This version is deprecated. We recommend you to use the newest version Fineweb-edu-chinese-v2.1 !
Chinese Fineweb Edu Dataset [中文] [English]
[OpenCSG Community] [👾github] [wechat] [Twitter]
📖Technical Report
Chinese Fineweb Edu dataset is a meticulously constructed high-quality Chinese pre-training corpus, specifically designed for natural language processing tasks in the education domain. This dataset undergoes a rigorous selection and… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/chinese-fineweb-edu.Fineweb-Edu-Chinese-V2.2
Chinese Fineweb Edu Dataset V2.2 (Instruct & Pre-train)
[[中文]] | [[English]]
OpenCSG Community | 👾 GitHub | 📖 Technical Report
Dataset Introduction: Filling the Data Puzzle for Chinese Education LLMs
Chinese Fineweb Edu Dataset V2.2is a rare high-quality dataset in the open-source community that covers the full process from Pre-training to Supervised Fine-Tuning (SFT) for the Chinese education domain.
This project aims to solve the core pain point of… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/Fineweb-Edu-Chinese-V2.2.MMLU_ChineseChinese version of MMLU dataset tranlasted by gpt-3.5-turbo.The dataset is used in the research related to MultilingualSIFT.
Traditional-Chinese-Common-Crawl-NOT-CleanedCommon Crawl Dumps that were briefly filtered by keywords to remove bad words and simplified Chinese.
The hash based cleaned dataset can be found here.
Files here are for future usage (downloading from Common Crawl and keyword filtering are very slow)
Traditional-Chinese-Common-Crawl-Filtered
Traditional Chinese C4
Dataset Summary
Data obtained from 2013～2025 Common Crawl.
Downloaded and processed using code based on another project attempting to recreate the C4 dataset.
The resultant dataset contains both simplified and traditional Chinese, which could be found here.
It was then filtered using a modified list of simplified Chinese characters to obtain this traditional Chinese dataset.
Unfortunately, I don't have enough funding to run a deduplication across… See the full description on the dataset page: https://huggingface.co/datasets/jed351/Traditional-Chinese-Common-Crawl-Filtered.