datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
alpaca-data-gpt4-chineseChineseWebText
ChineseWebText: Large-Scale High-quality Chinese Web Text Extracted with Effective Evaluation Model
This directory contains the ChineseWebText dataset, and the EvalWeb tool-chain to process CommonCrawl Data. Our EvalWeb tool is publicly available on github https://github.com/CASIA-LM/ChineseWebText.
ChineseWebText
Dataset Overview
We release the latest and largest Chinese dataset ChineseWebText, which consists of 1.42 TB data and each text is assigned a… See the full description on the dataset page: https://huggingface.co/datasets/CASIA-LM/ChineseWebText.ChineseWebText2.0
ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information
This directory contains the ChineseWebText2.0 dataset, and a new tool-chain called MDFG-tool for constructing large-scale and high-quality Chinese datasets with multi-dimensional and fine-grained information. Our ChineseWebText2.0 code is publicly available on github (here).
ChineseWebText2.0
Dataset Overview
We have released the latest… See the full description on the dataset page: https://huggingface.co/datasets/CASIA-LM/ChineseWebText2.0.Chinese-SimpleQA
Overview
🌐 Website • 🤗 Hugging Face • ⏬ Data • 📃 Paper • 📊 Leaderboard
Chinese SimpleQA is the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions, and Chinese SimpleQA mainly has five properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate). Specifically, our benchmark covers 6 major topics with 99 diverse subtopics.
Please visit our website or check our paper for more details.… See the full description on the dataset page: https://huggingface.co/datasets/OpenStellarTeam/Chinese-SimpleQA.chinese-c4
Introduction
Chinese-C4 is a clean Chinese internet dataset based on Common Crawl. The dataset is 46.29GB and has undergone multiple cleaning strategies, including Chinese filtering, heuristic cleaning based on punctuation, line-based hashing for deduplication, and repetition removal.
The dataset is open source and free for commercial use, and you are welcome to use the data and the cleaning strategies provided and contribute your cleaning strategies.
You can find the cleaning… See the full description on the dataset page: https://huggingface.co/datasets/shjwudp/chinese-c4.alpaca-gpt4-chineseThe dataset is used in the research related to MultilingualSIFT.
Vript_Chinese
🎬 Vript: Refine Video Captioning into Video Scripting [Github Repo]
We construct a fine-grained video-text dataset with 44.7K annotated high-resolution videos (~293k clips) in Chinese. The annotation of this dataset is inspired by the video script. If we want to make a video, we have to first write a script to organize how to shoot the scenes in the videos. To shoot a scene, we need to decide the content, shot type (medium shot, close-up, etc), and how the camera moves (panning… See the full description on the dataset page: https://huggingface.co/datasets/Mutonix/Vript_Chinese.webnovel-chinese
简介
搜集网络上的网文小说，清洗，分割后，用于训练大语言模型，共计9000本左右，大约9B左右token。
使用
格式说明
采用jsonl格式存储，分为三个字段：
title ：小说名称
chapter：章节
text：正文内容
示例：
{"title": "斗破苍穹", "chapter": " 第一章 陨落的天才", "text": "“斗之力，三段！”\n望着测验魔石碑上面闪亮得甚至有些刺眼的五个大字，少年面无表情，唇角有着一抹自嘲，紧握的手掌，因为大力，而导致略微尖锐的指甲深深的刺进了掌心之中，带来一阵阵钻心的疼痛……\n“萧炎，斗之力，三段！级别：低级！”测验魔石碑之旁，一位中年男子，看了一眼碑上所显示出来的信息，语气漠然的将之公布了出来……\n"}
ChineseWebText
ChineseWebText: Large-Scale High-quality Chinese Web Text Extracted with Effective Evaluation Model
This directory contains the ChineseWebText dataset, and the EvalWeb tool-chain to process CommonCrawl Data. Our EvalWeb tool is publicly available on github https://github.com/CASIA-LM/ChineseWebText.
ChineseWebText
Dataset Overview
We release the latest and largest Chinese dataset ChineseWebText, which consists of 1.42 TB data and each text is assigned a… See the full description on the dataset page: https://huggingface.co/datasets/alkaid0618/ChineseWebText.Chinese-DeepSeek-R1-Distill-data-110k
中文基于满血DeepSeek-R1蒸馏数据集（Chinese-Data-Distill-From-R1）
🤗 Hugging Face | 🤖 ModelScope | 🚀 Github | 📑 Blog
注意：提供了直接SFT使用的版本，点击下载。将数据中的思考和答案整合成output字段，大部分SFT代码框架均可直接直接加载训练。
本数据集为中文开源蒸馏满血R1的数据集，数据集中不仅包含math数据，还包括大量的通用类型数据，总数量为110K。
为什么开源这个数据？
R1的效果十分强大，并且基于R1蒸馏数据SFT的小模型也展现出了强大的效果，但检索发现，大部分开源的R1蒸馏数据集均为英文数据集。 同时，R1的报告中展示，蒸馏模型中同时也使用了部分通用场景数据集。
为了帮助大家更好地复现R1蒸馏模型的效果，特此开源中文数据集。
该中文数据集中的数据分布如下：… See the full description on the dataset page: https://huggingface.co/datasets/Congliu/Chinese-DeepSeek-R1-Distill-data-110k.Wizard-LM-Chinese-instruct-evolWizard-LM-Chinese是在MSRA的Wizard-LM数据集上，对指令进行翻译，然后再调用GPT获得答案的数据集
Wizard-LM包含了很多难度超过Alpaca的指令。
中文的问题翻译会有少量指令注入导致翻译失败的情况
中文回答是根据中文问题再进行问询得到的。
我们会陆续将更多数据集发布到hf，包括
Coco Caption的中文翻译
CoQA的中文翻译
CNewSum的Embedding数据
增广的开放QA数据
WizardLM的中文翻译
如果你也在做这些数据集的筹备，欢迎来联系我们，避免重复花钱。
骆驼(Luotuo): 开源中文大语言模型
https://github.com/LC1332/Luotuo-Chinese-LLM
骆驼(Luotuo)项目是由冷子昂 @ 商汤科技, 陈启源 @ 华中师范大学 以及 李鲁鲁 @ 商汤科技 发起的中文大语言模型开源项目，包含了一系列语言模型。
( 注意: 陈启源 正在寻找2024推免导师，欢迎联系 )
骆驼项目不是商汤科技的官方产品。
Citation… See the full description on the dataset page: https://huggingface.co/datasets/silk-road/Wizard-LM-Chinese-instruct-evol.Chinese-Instruct
中文指令微调数据集
💻 Github Repo
本项目旨在构建一个高质量、多领域、大规模的中文指令微调数据集。
本项目将会持续更新。更多数据集欢迎访问 Github Repo。
[!TIP]
如果您想要一个可用于学习的简化版中文指令数据集，可以访问：Mxode/Chinese-Instruct-Lite
具体构成
dpsk-r1-distil：中文 DeepSeek-R1 蒸馏数据集，来自 Congliu/Chinese-DeepSeek-R1-Distill-data-110k，根据打分质量做了筛选，提取了最终的回答，未包含思考过程。
chinese-reasoning-distil：中文推理蒸馏数据集，来自 Mxode/Chinese-Reasoning-Distil-Data，提取了最终的回答，未包含思考过程。
firefly：中文通用指令微调数据集，指令取自 Mxode/Firefly-1.1M-Rephrased，其本身已经相较于原 Firefly… See the full description on the dataset page: https://huggingface.co/datasets/Mxode/Chinese-Instruct.ChineseSafe
ChineseSafe
Dataset for ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models
Usage
from datasets import load_dataset
dataset = load_dataset("SUSTech/ChineseSafe", split="test")
Citation
If you find our dataset useful, please cite:
@article{zhang2024chinesesafe,
title={ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models},
author={Zhang, Hengxiang and Gao, Hongfu and Hu, Qiang and Chen, Guanhua and Yang… See the full description on the dataset page: https://huggingface.co/datasets/SUSTech/ChineseSafe.alpaca_data_chinese_51kzhoblimpchinese_exam_train_dataChineseHarm-bench ChineseHarm-bench
A Chinese Harmful Content Detection Benchmark
⚠️ WARNING: This project and associated data contain content that may be toxic, offensive, or disturbing. Use responsibly and with discretion.
Project •
Paper •
Hugging Face
🌟Benchmark
This folder contains the ChineseHarm-Bench.
bench.json is the full benchmark combining all categories.
The other files (e.g., 低俗色情.json, 欺诈.json) are category-specific subsets.
Each file is a list of… See the full description on the dataset page: https://huggingface.co/datasets/zjunlp/ChineseHarm-bench.chinese-classical-corpus
Chinese Classical Corpus
🔗 源码 & 构建脚本: github.com/gujilab/chinese-classical-corpus — 完整抽取 pipeline、14 个 Python 脚本、验证套件
🎯 配套评测基准: gujilab/chinese-classical-bench — 500 道题 × 5 任务，测 LLM 古典文献能力（题目均从本语料抽样）
中国古典文献结构化语料集，含完整十三经 + 说文解字 + 资治通鉴 + 二十四史前 15 部，以及 197 万条古译今/今译古/断句指令对。
全部 CC0 公有领域，可商用、可改用、无附加限制。
为什么做这个
中文（尤其文言文）常被说成"高密度优势"。本语料集 + 配套评测想把这个论点变成可验证的数字 —— 包括它在哪些场景成立、在哪些场景不成立。
Tokenizer 层面 —— 真成立
7 个主流 tokenizer 横评（tokenizer_study）：
DeepSeek-V3 /… See the full description on the dataset page: https://huggingface.co/datasets/gujilab/chinese-classical-corpus.Chinese-DeepSeek-R1-Distill-data-110k-SFT
中文基于满血DeepSeek-R1蒸馏数据集（Chinese-Data-Distill-From-R1）
🤗 Hugging Face | 🤖 ModelScope | 🚀 Github | 📑 Blog
注意：该版本为，可以直接SFT使用的版本，将原始数据中的思考和答案整合成output字段，大部分SFT代码框架均可直接直接加载训练。
本数据集为中文开源蒸馏满血R1的数据集，数据集中不仅包含math数据，还包括大量的通用类型数据，总数量为110K。
为什么开源这个数据？
R1的效果十分强大，并且基于R1蒸馏数据SFT的小模型也展现出了强大的效果，但检索发现，大部分开源的R1蒸馏数据集均为英文数据集。 同时，R1的报告中展示，蒸馏模型中同时也使用了部分通用场景数据集。
为了帮助大家更好地复现R1蒸馏模型的效果，特此开源中文数据集。该中文数据集中的数据分布如下：
Math：共计36568个样本，
Exam：共计2432个样本，
STEM：共计12648个样本，… See the full description on the dataset page: https://huggingface.co/datasets/Congliu/Chinese-DeepSeek-R1-Distill-data-110k-SFT.chinese-materials-science-open-intelligence
🔬 Chinese Materials Science & Metallurgy Open Intelligence Dataset
Curated open intelligence dataset providing English research briefs, authoritative DOIs, executive summaries, and high-resolution micrographs of breakthrough Chinese scientific research in Materials Science, Metallurgy, Advanced Alloys, and Mining Engineering.
[!IMPORTANT]
Data Completeness & Research Authenticity Notice:
Included in this Hugging Face Open Dataset: English structured abstracts, core… See the full description on the dataset page: https://huggingface.co/datasets/simpleG2023/chinese-materials-science-open-intelligence.Fineweb-Edu-Chinese-V2.3
Chinese Fineweb Edu Dataset V2.3
中文 | English
OpenCSG 社区 | GitHub | 数据集许可协议
数据集简介
Chinese Fineweb Edu Dataset V2.3 是 OpenCSG 面向中文教育、知识问答、指令微调和文本生成场景构建的高质量中文教育 SFT 数据集。
该版本包含 23.04 万条高质量中文教育 QA pairs，并将同一批问答对发布为 Alpaca、Messages、Messages-no-system 三种训练格式。三种格式面向不同训练模板，建议训练时按模型和框架选择其中一种格式使用，而不是将不同格式简单相加作为独立知识规模。
V2.3 是在 V2.2 基础上的质量升级版本。针对 V2.2 社区反馈和内部质量审计中出现的重复模式、异常中英文混入、噪声片段、弱证据支撑回答和低质量合成输出等问题，V2.3 提高了源文本进入生成环节的门槛，并优化了问答生成与过滤逻辑。
在数据构建上，V2.3 从约 2.3T… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/Fineweb-Edu-Chinese-V2.3.sharegpt-chineseChinese ShareGPT data translated by gpt-3.5-turbo.The dataset is used in the research related to MultilingualSIFT.
chinese-meme-description-dataset
Describe image information using the following LLM Models
gpt4o
Claude-3.5-sonnet-20240620
gemini-1.5-pro
gemini-1.5-flash
gemini-1.0-pro-vision
yi-vision
Gemini Code
# -*- coding: gbk -*-
import google.generativeai as genai
import PIL.Image
import os
import json
import shutil
from tqdm import tqdm
from concurrent.futures import ThreadPoolExecutor, as_completed
genai.configure(api_key='')
model = genai.GenerativeModel(
'gemini-1.5-pro-latest'… See the full description on the dataset page: https://huggingface.co/datasets/REILX/chinese-meme-description-dataset.alpaca_chinese_datasetThanks for hikariming.
This dataset was created by merging "翻译后的中文数据" and "其他中文问题补充" from hikariming/alpaca_chinese_dataset in a certain format.
chinese-clean-energy-battery-open-intelligence
🔬 Chinese Clean Energy, Battery Chemistry & Smart Grid Open Intelligence Dataset
Curated open intelligence dataset tracking authentic Chinese scientific breakthroughs in Solid-State Battery chemistry, Perovskite Solar cells, Ultra-High Voltage (UHV) power grids, and industrial decarbonization.
[!IMPORTANT]
Data Completeness & Research Authenticity Notice:
Included in this Hugging Face Open Dataset: English structured abstracts, core quantitative takeaways, author… See the full description on the dataset page: https://huggingface.co/datasets/simpleG2023/chinese-clean-energy-battery-open-intelligence.TinyBooks-QA-Chinese本数据集已停止更新，请移步https://huggingface.co/datasets/fzmnm/TinyStoriesAdv-zh
TinyBooks-QA-Chinese
Inspired by the (TinyStories)[https://arxiv.org/abs/2305.07759] paper, where a small language model exhibits strong capabilities when trained on high-quality, 🍼baby-friendly stories synthesized by AI, I present an AI-generated Encyclopedia suitable for kindergarten and grade school levels.
This AI-synthesized dataset converts classical literature into a question-answer style curriculum with… See the full description on the dataset page: https://huggingface.co/datasets/fzmnm/TinyBooks-QA-Chinese.Chinese-pretraining-datasetData source: https://github.com/CVI-SZU/Linly/wiki/Linly-OpenLLaMA
webnovel-chinese
简介
搜集网络上的网文小说，清洗，分割后，用于训练大语言模型，共计9000本左右，大约9B左右token。
使用
格式说明
采用jsonl格式存储，分为三个字段：
title ：小说名称
chapter：章节
text：正文内容
示例：
{"title": "斗破苍穹", "chapter": " 第一章 陨落的天才", "text": "“斗之力，三段！”\n望着测验魔石碑上面闪亮得甚至有些刺眼的五个大字，少年面无表情，唇角有着一抹自嘲，紧握的手掌，因为大力，而导致略微尖锐的指甲深深的刺进了掌心之中，带来一阵阵钻心的疼痛……\n“萧炎，斗之力，三段！级别：低级！”测验魔石碑之旁，一位中年男子，看了一眼碑上所显示出来的信息，语气漠然的将之公布了出来……\n"}
chinese_modern_poetry
简介
数据集包括了近现代的中国诗人及外国诗人（中译版）作品，所有作品著作权归原作者所有，侵删请联系aa531811820@gmail.com
chinese_poems.jsonl为原数据，training_imagery2-5_maxlen256.json 分别是根据2-5个关键意象生成诗歌的相关数据集
数据来源于网络，包括但不限于
https://github.com/sheepzh/poetry
https://bedtimepoem.com/
https://poemwiki.org/
baidu、google、zhihu等
一些作品
使用此数据集训练ChatGLM、LLaMA7b模型生成的诗歌，更多诗歌查看poems目录
chinese-ai-and-robotics-open-intelligence
🔬 Chinese AI, Humanoid Robotics & Neural Systems Open Intelligence Dataset
Curated open intelligence dataset tracking Chinese frontier developments in Large Language Models (LLMs), Humanoid Dynamic Locomotion, 3D Computer Vision, and Neuromorphic edge processors.
[!IMPORTANT]
Data Completeness & Research Authenticity Notice:
Included in this Hugging Face Open Dataset: English structured abstracts, core quantitative takeaways, author institutional affiliations, and… See the full description on the dataset page: https://huggingface.co/datasets/simpleG2023/chinese-ai-and-robotics-open-intelligence.