datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
MultiArithchi-bench
Clinical Healthcare In-Situ Environment
Task fixtures for a long-horizon, policy-rich healthcare-workflow agent benchmark
What is in this dataset
CHI-Bench evaluates AI agents on end-to-end U.S. healthcare workflows across three long-horizon domains: provider prior authorization, payer utilization management, and population care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed over MCP, with a 1… See the full description on the dataset page: https://huggingface.co/datasets/actava/chi-bench.alpaca-data-gpt4-chineseChineseWebText
ChineseWebText: Large-Scale High-quality Chinese Web Text Extracted with Effective Evaluation Model
This directory contains the ChineseWebText dataset, and the EvalWeb tool-chain to process CommonCrawl Data. Our EvalWeb tool is publicly available on github https://github.com/CASIA-LM/ChineseWebText.
ChineseWebText
Dataset Overview
We release the latest and largest Chinese dataset ChineseWebText, which consists of 1.42 TB data and each text is assigned a… See the full description on the dataset page: https://huggingface.co/datasets/CASIA-LM/ChineseWebText.ChineseWebText2.0
ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information
This directory contains the ChineseWebText2.0 dataset, and a new tool-chain called MDFG-tool for constructing large-scale and high-quality Chinese datasets with multi-dimensional and fine-grained information. Our ChineseWebText2.0 code is publicly available on github (here).
ChineseWebText2.0
Dataset Overview
We have released the latest… See the full description on the dataset page: https://huggingface.co/datasets/CASIA-LM/ChineseWebText2.0.sounio-code-examples
Sounio Curated Code Examples
Curated compile-clean .sio examples for training and evaluating code models on
Sounio, a self-hosted systems and scientific programming language for epistemic
computing, uncertainty propagation, and algebraic effects.
This directory is the Cx-1 expansion lane for
chiuratto-AIgourakis/sounio-code-examples.
Current batch
Examples: 5,000
Metadata files: 5,000
Compiler gate: bin/souc check pass rate 5,000/5,000
Utility layer: 5,000… See the full description on the dataset page: https://huggingface.co/datasets/chiuratto-AIgourakis/sounio-code-examples.Chinese-SimpleQA
Overview
🌐 Website • 🤗 Hugging Face • ⏬ Data • 📃 Paper • 📊 Leaderboard
Chinese SimpleQA is the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions, and Chinese SimpleQA mainly has five properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate). Specifically, our benchmark covers 6 major topics with 99 diverse subtopics.
Please visit our website or check our paper for more details.… See the full description on the dataset page: https://huggingface.co/datasets/OpenStellarTeam/Chinese-SimpleQA.chinese-c4
Introduction
Chinese-C4 is a clean Chinese internet dataset based on Common Crawl. The dataset is 46.29GB and has undergone multiple cleaning strategies, including Chinese filtering, heuristic cleaning based on punctuation, line-based hashing for deduplication, and repetition removal.
The dataset is open source and free for commercial use, and you are welcome to use the data and the cleaning strategies provided and contribute your cleaning strategies.
You can find the cleaning… See the full description on the dataset page: https://huggingface.co/datasets/shjwudp/chinese-c4.Vript_Chinese
🎬 Vript: Refine Video Captioning into Video Scripting [Github Repo]
We construct a fine-grained video-text dataset with 44.7K annotated high-resolution videos (~293k clips) in Chinese. The annotation of this dataset is inspired by the video script. If we want to make a video, we have to first write a script to organize how to shoot the scenes in the videos. To shoot a scene, we need to decide the content, shot type (medium shot, close-up, etc), and how the camera moves (panning… See the full description on the dataset page: https://huggingface.co/datasets/Mutonix/Vript_Chinese.alpaca-gpt4-chineseThe dataset is used in the research related to MultilingualSIFT.
webnovel-chinese
简介
搜集网络上的网文小说，清洗，分割后，用于训练大语言模型，共计9000本左右，大约9B左右token。
使用
格式说明
采用jsonl格式存储，分为三个字段：
title ：小说名称
chapter：章节
text：正文内容
示例：
{"title": "斗破苍穹", "chapter": " 第一章 陨落的天才", "text": "“斗之力，三段！”\n望着测验魔石碑上面闪亮得甚至有些刺眼的五个大字，少年面无表情，唇角有着一抹自嘲，紧握的手掌，因为大力，而导致略微尖锐的指甲深深的刺进了掌心之中，带来一阵阵钻心的疼痛……\n“萧炎，斗之力，三段！级别：低级！”测验魔石碑之旁，一位中年男子，看了一眼碑上所显示出来的信息，语气漠然的将之公布了出来……\n"}
ChiPBench-D
ChiPBench-D
ChiPBench:Benchmarking End-to-End Performance of AI-based Chip Placement Algorithms
Chip placement is a critical step in the Electronic Design Automation (EDA) workflow, which aims to arrange chip modules on the canvas to optimize the performance, power, and area (PPA) metrics of final designs.Recent advances show great potential of AI-based algorithms in chip placement.However, due to the lengthy EDA workflow, evaluations of these algorithms often focus on intermediate… See the full description on the dataset page: https://huggingface.co/datasets/MIRA-Lab/ChiPBench-D.ChineseWebText
ChineseWebText: Large-Scale High-quality Chinese Web Text Extracted with Effective Evaluation Model
This directory contains the ChineseWebText dataset, and the EvalWeb tool-chain to process CommonCrawl Data. Our EvalWeb tool is publicly available on github https://github.com/CASIA-LM/ChineseWebText.
ChineseWebText
Dataset Overview
We release the latest and largest Chinese dataset ChineseWebText, which consists of 1.42 TB data and each text is assigned a… See the full description on the dataset page: https://huggingface.co/datasets/alkaid0618/ChineseWebText.RoadSocial
About Dataset
This dataset accompanies the paper RoadSocial: A Diverse Dataset and Benchmark for Road Event Understanding from Social Video Narratives.
RoadSocial is a large-scale, diverse VideoQA dataset designed for generic road event understanding from social media narratives. It can help enhance the road event comprehension capabilities of general-purpose Video LLMs and improve their performance in traffic scene understanding, planning, and other autonomous vehicle (AV) related… See the full description on the dataset page: https://huggingface.co/datasets/chiragp26/RoadSocial.medical-prescription-datasetChinese-DeepSeek-R1-Distill-data-110k
中文基于满血DeepSeek-R1蒸馏数据集（Chinese-Data-Distill-From-R1）
🤗 Hugging Face | 🤖 ModelScope | 🚀 Github | 📑 Blog
注意：提供了直接SFT使用的版本，点击下载。将数据中的思考和答案整合成output字段，大部分SFT代码框架均可直接直接加载训练。
本数据集为中文开源蒸馏满血R1的数据集，数据集中不仅包含math数据，还包括大量的通用类型数据，总数量为110K。
为什么开源这个数据？
R1的效果十分强大，并且基于R1蒸馏数据SFT的小模型也展现出了强大的效果，但检索发现，大部分开源的R1蒸馏数据集均为英文数据集。 同时，R1的报告中展示，蒸馏模型中同时也使用了部分通用场景数据集。
为了帮助大家更好地复现R1蒸馏模型的效果，特此开源中文数据集。
该中文数据集中的数据分布如下：… See the full description on the dataset page: https://huggingface.co/datasets/Congliu/Chinese-DeepSeek-R1-Distill-data-110k.Wizard-LM-Chinese-instruct-evolWizard-LM-Chinese是在MSRA的Wizard-LM数据集上，对指令进行翻译，然后再调用GPT获得答案的数据集
Wizard-LM包含了很多难度超过Alpaca的指令。
中文的问题翻译会有少量指令注入导致翻译失败的情况
中文回答是根据中文问题再进行问询得到的。
我们会陆续将更多数据集发布到hf，包括
Coco Caption的中文翻译
CoQA的中文翻译
CNewSum的Embedding数据
增广的开放QA数据
WizardLM的中文翻译
如果你也在做这些数据集的筹备，欢迎来联系我们，避免重复花钱。
骆驼(Luotuo): 开源中文大语言模型
https://github.com/LC1332/Luotuo-Chinese-LLM
骆驼(Luotuo)项目是由冷子昂 @ 商汤科技, 陈启源 @ 华中师范大学 以及 李鲁鲁 @ 商汤科技 发起的中文大语言模型开源项目，包含了一系列语言模型。
( 注意: 陈启源 正在寻找2024推免导师，欢迎联系 )
骆驼项目不是商汤科技的官方产品。
Citation… See the full description on the dataset page: https://huggingface.co/datasets/silk-road/Wizard-LM-Chinese-instruct-evol.Chinese-Instruct
中文指令微调数据集
💻 Github Repo
本项目旨在构建一个高质量、多领域、大规模的中文指令微调数据集。
本项目将会持续更新。更多数据集欢迎访问 Github Repo。
[!TIP]
如果您想要一个可用于学习的简化版中文指令数据集，可以访问：Mxode/Chinese-Instruct-Lite
具体构成
dpsk-r1-distil：中文 DeepSeek-R1 蒸馏数据集，来自 Congliu/Chinese-DeepSeek-R1-Distill-data-110k，根据打分质量做了筛选，提取了最终的回答，未包含思考过程。
chinese-reasoning-distil：中文推理蒸馏数据集，来自 Mxode/Chinese-Reasoning-Distil-Data，提取了最终的回答，未包含思考过程。
firefly：中文通用指令微调数据集，指令取自 Mxode/Firefly-1.1M-Rephrased，其本身已经相较于原 Firefly… See the full description on the dataset page: https://huggingface.co/datasets/Mxode/Chinese-Instruct.ChineseSafe
ChineseSafe
Dataset for ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models
Usage
from datasets import load_dataset
dataset = load_dataset("SUSTech/ChineseSafe", split="test")
Citation
If you find our dataset useful, please cite:
@article{zhang2024chinesesafe,
title={ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models},
author={Zhang, Hengxiang and Gao, Hongfu and Hu, Qiang and Chen, Guanhua and Yang… See the full description on the dataset page: https://huggingface.co/datasets/SUSTech/ChineseSafe.chinese_exam_train_datathe-un-laion-templeAll files uploaded. Enjoy!
Dataset Card for The Unlaion Temple
Dataset Details
Dataset Description
Laion-5B is still not public, so we decided to create our own dataset.
The Unlaion Temple is a raw dataset of CommonCrawl images (Estimated to be a total of 2 Billion urls). We haven't verified whether the links in this dataset are functional.
You are responsible for handling the data.
We've made some improvements to the dataset based on user feedback:
All… See the full description on the dataset page: https://huggingface.co/datasets/DSULT-Chiharu/the-un-laion-temple.alpaca_data_chinese_51kzhoblimpChildren-Stories-CollectionChildren Stories Collection
A great synthetic datasets consists of around 0.9 million stories especially meant for Young Children. You can directly use these datasets for training large models.
Total 10 datasets are available for download. You can use any one or all the json files for training purpose.
These datasets are in "prompt" and "text" format. Total token length is also available.
Thank you for your love & support.
ChineseHarm-bench ChineseHarm-bench
A Chinese Harmful Content Detection Benchmark
⚠️ WARNING: This project and associated data contain content that may be toxic, offensive, or disturbing. Use responsibly and with discretion.
Project •
Paper •
Hugging Face
🌟Benchmark
This folder contains the ChineseHarm-Bench.
bench.json is the full benchmark combining all categories.
The other files (e.g., 低俗色情.json, 欺诈.json) are category-specific subsets.
Each file is a list of… See the full description on the dataset page: https://huggingface.co/datasets/zjunlp/ChineseHarm-bench.chinese-classical-corpus
Chinese Classical Corpus
🔗 源码 & 构建脚本: github.com/gujilab/chinese-classical-corpus — 完整抽取 pipeline、14 个 Python 脚本、验证套件
🎯 配套评测基准: gujilab/chinese-classical-bench — 500 道题 × 5 任务，测 LLM 古典文献能力（题目均从本语料抽样）
中国古典文献结构化语料集，含完整十三经 + 说文解字 + 资治通鉴 + 二十四史前 15 部，以及 197 万条古译今/今译古/断句指令对。
全部 CC0 公有领域，可商用、可改用、无附加限制。
为什么做这个
中文（尤其文言文）常被说成"高密度优势"。本语料集 + 配套评测想把这个论点变成可验证的数字 —— 包括它在哪些场景成立、在哪些场景不成立。
Tokenizer 层面 —— 真成立
7 个主流 tokenizer 横评（tokenizer_study）：
DeepSeek-V3 /… See the full description on the dataset page: https://huggingface.co/datasets/gujilab/chinese-classical-corpus.chinese-materials-science-open-intelligence
🔬 Chinese Materials Science & Metallurgy Open Intelligence Dataset
Curated open intelligence dataset providing English research briefs, authoritative DOIs, executive summaries, and high-resolution micrographs of breakthrough Chinese scientific research in Materials Science, Metallurgy, Advanced Alloys, and Mining Engineering.
[!IMPORTANT]
Data Completeness & Research Authenticity Notice:
Included in this Hugging Face Open Dataset: English structured abstracts, core… See the full description on the dataset page: https://huggingface.co/datasets/simpleG2023/chinese-materials-science-open-intelligence.Chinese-DeepSeek-R1-Distill-data-110k-SFT
中文基于满血DeepSeek-R1蒸馏数据集（Chinese-Data-Distill-From-R1）
🤗 Hugging Face | 🤖 ModelScope | 🚀 Github | 📑 Blog
注意：该版本为，可以直接SFT使用的版本，将原始数据中的思考和答案整合成output字段，大部分SFT代码框架均可直接直接加载训练。
本数据集为中文开源蒸馏满血R1的数据集，数据集中不仅包含math数据，还包括大量的通用类型数据，总数量为110K。
为什么开源这个数据？
R1的效果十分强大，并且基于R1蒸馏数据SFT的小模型也展现出了强大的效果，但检索发现，大部分开源的R1蒸馏数据集均为英文数据集。 同时，R1的报告中展示，蒸馏模型中同时也使用了部分通用场景数据集。
为了帮助大家更好地复现R1蒸馏模型的效果，特此开源中文数据集。该中文数据集中的数据分布如下：
Math：共计36568个样本，
Exam：共计2432个样本，
STEM：共计12648个样本，… See the full description on the dataset page: https://huggingface.co/datasets/Congliu/Chinese-DeepSeek-R1-Distill-data-110k-SFT.sharegpt-chineseChinese ShareGPT data translated by gpt-3.5-turbo.The dataset is used in the research related to MultilingualSIFT.
Fineweb-Edu-Chinese-V2.3
Chinese Fineweb Edu Dataset V2.3
中文 | English
OpenCSG 社区 | GitHub | 数据集许可协议
数据集简介
Chinese Fineweb Edu Dataset V2.3 是 OpenCSG 面向中文教育、知识问答、指令微调和文本生成场景构建的高质量中文教育 SFT 数据集。
该版本包含 23.04 万条高质量中文教育 QA pairs，并将同一批问答对发布为 Alpaca、Messages、Messages-no-system 三种训练格式。三种格式面向不同训练模板，建议训练时按模型和框架选择其中一种格式使用，而不是将不同格式简单相加作为独立知识规模。
V2.3 是在 V2.2 基础上的质量升级版本。针对 V2.2 社区反馈和内部质量审计中出现的重复模式、异常中英文混入、噪声片段、弱证据支撑回答和低质量合成输出等问题，V2.3 提高了源文本进入生成环节的门槛，并优化了问答生成与过滤逻辑。
在数据构建上，V2.3 从约 2.3T… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/Fineweb-Edu-Chinese-V2.3.