datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
stackv2_edu_filtered
Stack V2 Edu
Description
We filter the Stack V2 to only include code from openly licensed repositories, based on the license detection performed by the creators of Stack V2. When multiple licenses are detected in a single repository, we ensure that all of the licenses are on the Blue Oak Council certified license list. Per-document license information is available in the license entry of the metadata field of each example. Code for collecting, processing, and preparing… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/stackv2_edu_filtered.IndustryCorpus_education[中文主页]
Industry models play a crucial role in driving enterprise intelligence transformation and innovative development. High-quality industry data is key to improving the performance of large models and realizing industry applications. However, datasets currently used for industry model training generally suffer from issues such as insufficient data volume, low quality, and lack of domain expertise.
To address these problems, we constructed and applied 22 industry data processing operators to… See the full description on the dataset page: https://huggingface.co/datasets/BAAI/IndustryCorpus_education.fineweb-edu-zh-chengyu-cpt
Fineweb-Edu Chinese — Chengyu-Tagged Continued-Pretraining Corpus
A 3.74M-document Chinese corpus (~7.8B tokens) for continued pretraining on
cultural knowledge in figurative language, built from the highest-quality
tier of opencsg/Fineweb-Edu-Chinese-V2.1.
Each document is educational Chinese text containing at least one culturally
vetted chengyu, with an appended 【成语注释】 knowledge block listing every
matched idiom's figurative meaning(s) and classical source citation.
This is a… See the full description on the dataset page: https://huggingface.co/datasets/jiviteshjn/fineweb-edu-zh-chengyu-cpt.DeepSWEGym2-Edu
Dataset Description
This dataset is a filtered and deduplicated version of a merge containing many high quality SWE datasets, it aims to improve benchmark results on DeepSWE-style problems, benchmarks, and general coding skills.
It is specifically filtered for rows with complex/long code problems in the original datasets, having an average row size of 291.74kb, a total uncompressed size of 14.93GB, and a total of 53649 examples.
Dataset Details
Curated by:… See the full description on the dataset page: https://huggingface.co/datasets/MoreThought/DeepSWEGym2-Edu.Omni-Edu
Omni-Edu — Core V6 SFT mixture
69,999 supervised instruction examples (~158M characters) covering K-12 subject
competence, curriculum grounding, diagnostic reasoning, pedagogical action and
general-purpose instruction. 12,146 rows (17.4%) are multimodal; every image
referenced by the JSONL ships in this repository under images/.
This is the system-prompted assembly of the v6 core mixture: every row carries
an explicit system message, and the non-system turns are byte-identical… See the full description on the dataset page: https://huggingface.co/datasets/lhpku20010120/Omni-Edu.DeepSWEGym-Edu
Dataset Description
This dataset is a heavily filtered version of all the SWE-bench/SWE-smith-lang datasets (expect php) merged together (originally 88k rows total), it aims to improve benchmark results on DeepSWE-style problems, benchmarks, and general coding skills.
It is specifically filtered for rows with very complex/long code problems in the original datasets, having an average row size of 101.81kb, a total uncompressed size of 4.29GB, and 42529 examples total.… See the full description on the dataset page: https://huggingface.co/datasets/MoreThought/DeepSWEGym-Edu.Fineweb-Edu-Chinese-V2.3
Chinese Fineweb Edu Dataset V2.3
中文 | English
OpenCSG 社区 | GitHub | 数据集许可协议
数据集简介
Chinese Fineweb Edu Dataset V2.3 是 OpenCSG 面向中文教育、知识问答、指令微调和文本生成场景构建的高质量中文教育 SFT 数据集。
该版本包含 23.04 万条高质量中文教育 QA pairs，并将同一批问答对发布为 Alpaca、Messages、Messages-no-system 三种训练格式。三种格式面向不同训练模板，建议训练时按模型和框架选择其中一种格式使用，而不是将不同格式简单相加作为独立知识规模。
V2.3 是在 V2.2 基础上的质量升级版本。针对 V2.2 社区反馈和内部质量审计中出现的重复模式、异常中英文混入、噪声片段、弱证据支撑回答和低质量合成输出等问题，V2.3 提高了源文本进入生成环节的门槛，并优化了问答生成与过滤逻辑。
在数据构建上，V2.3 从约 2.3T… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/Fineweb-Edu-Chinese-V2.3.smollm-corpus-fineweb-edu-enPurified-openai-messages
📖 smollm-corpus-fineweb-edu-enPurified-openai-messages
smollm-corpus-fineweb-edu-enPurified is a highly curated, "prose-first" subset of the fineweb-edu-dedup subset found in HuggingFaceTB/smollm-corpus.
The enPurified collection is built on a specific philosophy: Specialization. While the original dataset is excellent for general pre-training, high-quality fluent English prose often gets diluted when mixed with syntax-heavy code, rigid math formulas, or low-information web junk.… See the full description on the dataset page: https://huggingface.co/datasets/enPurified/smollm-corpus-fineweb-edu-enPurified-openai-messages.multilingual_tokenizer_benchmark
Multilingual Tokenizer Benchmark
More details of each subset like word count, character count, original sources, etc, can be found in the dataset_meta.yaml file in the repository root.
Natural language word count functions
Download spacy models
pip install ntlk spacy pygments underthesea camel-tools
python -m spacy download ko_core_news_sm
python -m spacy download ja_core_news_sm
python -m spacy download zh_core_web_sm
import nltk
nltk.download('punkt_tab')… See the full description on the dataset page: https://huggingface.co/datasets/eduagarcia/multilingual_tokenizer_benchmark.massive-yt-edu-queue
Massive YouTube Educational Video Queue
Full metadata and content classification for 4,489,228 YouTube educational videos totaling 3,975,157 hours.
Description
This dataset contains metadata, content categorization, and license risk assessment for ~4.5M YouTube videos identified as potentially educational. It serves as the discovery and processing queue for the massive-yt-edu-transcriptions project, which aims to create the world's largest open educational transcript… See the full description on the dataset page: https://huggingface.co/datasets/thepowerfuldeez/massive-yt-edu-queue.InstructGpt-educational
LuminaSFT
LuminaSFT is a synthetic SFT dataset suite specifically designed to improve both general-purpose and task-specific SLMs. LuminaSFT consists of multiple curated splits that target diverse capabilities:
UltraChat200K-DeepSeek - A regenerated base SFT dataset for broad instruction following.
InstructGPT-NaturalQA and InstructGPT-TriviaQA - Factual question answering datasets to strengthen knowledge recall and answer accuracy.
CoT-Drop - A reading comprehension dataset with… See the full description on the dataset page: https://huggingface.co/datasets/amd/InstructGpt-educational.eduagentbench
EduAgentBench
EduAgentBench is a 150-task benchmark release for evaluating AI tutor agents. It targets three complementary capabilities: professional pedagogical judgment, situated multi-turn tutoring, and Canvas-style teaching workflow execution.
This release is prepared for anonymous research review. It contains transformed benchmark task definitions, synthetic/mock course state, compact metadata, and machine-readable dataset documentation. It does not redistribute raw source PDFs… See the full description on the dataset page: https://huggingface.co/datasets/eduagentbench/eduagentbench.RusLang-edu-1000
RusLang-Edu-1000 — an educational Russian-language QA dataset
RusLang-Edu-1000 is an expert-curated dataset of 1,000 instruction-format records ("question — detailed educational answer") covering the Russian language and linguistics: from phonetics and orthography to dialectology and theoretical linguistics. Every record contains a detailed answer (on average ≈1,100 characters), a short reference answer, a concise statement of the rule, and rich annotation (subject area, task… See the full description on the dataset page: https://huggingface.co/datasets/DatasetsEval/RusLang-edu-1000.kenyan-Edu-blindspot-eval
Kenyan Educational Blind Spot Evaluation
Model: Qwen/Qwen2.5-1.5B-Instruct (1.54B)Author: Walter OnyangoContext: Fatima Institute for Global AI Research Fellowship application
1. The Blind Spot
From my experience studying Computer Science in Kenya and building tools such as UAMAS (an AI-assisted assessment system) and a text-simplification extension used by real learners, I repeatedly observed that current models struggle with:
East-African local knowledge… See the full description on the dataset page: https://huggingface.co/datasets/waltertaya/kenyan-Edu-blindspot-eval.adaption-urdu-edu-cultural-reasoning
This dataset is a remastered version prepared using Adaption's Adaptive Data platform.
adaption-urdu_edu_cultural_reasoning
This dataset contains a mixed collection of question-answer pairs and linguistic tasks presented in both English and Urdu. The content spans multiple domains including history, biology, geography, and Urdu literature, featuring multiple-choice questions, translation exercises, and poetic composition prompts. Samples include historical treaty analysis… See the full description on the dataset page: https://huggingface.co/datasets/abdullah693/adaption-urdu-edu-cultural-reasoning.Multidisciplinary-Educational-Summaries
Knowledge Summarization Dataset
Overview
100 structured knowledge summaries across STEM, social sciences, and humanities. Features 70% Indian-centric content, 25% European perspectives, and 5% other Asian contexts for balanced representation.
Dataset Structure
{
"input": "Long-form text",
"output": {
"type": "summary",
"topic": "Subject name",
"difficulty": "beginner/intermediate/advanced",
"points": ["Key point 1", "Key point 2"]
}
}… See the full description on the dataset page: https://huggingface.co/datasets/Srinivasmec26/Multidisciplinary-Educational-Summaries.aihub-korean-education-instruct-sample
Korean Education Instruction Dataset (Sample)
Note: 이 데이터셋은 전체 데이터셋의 샘플 버전입니다 (카테고리별 최대 1,000건).
개요
AI Hub의 한국어 교육 데이터셋 13종을 sLLM 지시학습(Instruction Tuning)용으로 변환한 데이터셋입니다.
초등학교부터 고등학교까지의 다양한 교육 콘텐츠를 포함합니다.
데이터셋 통계
카테고리
데이터 수
math (수학)
1000
korean (국어)
1000
writing (글쓰기)
1000
career (진로)
1000
curriculum (교과)
1000
tutor (튜터링)
1000
총계
6000
사용 방법
from datasets import load_dataset
# 데이터셋 로드
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/neuralfoundry-coder/aihub-korean-education-instruct-sample.educational_domain_dataset
Nepali Grounded Education QA (OpenHermes-format)
A small, fact-grounded Nepali instruction-tuning dataset of question–answer pairs about
student enrollment statistics from Nepal's Ministry of Education. Every answer is
anchored to a real numeric value pulled from government open data — nothing in the
answers is model-hallucinated.
Dataset Summary
Rows
611
Language
Nepali (Devanagari script)
Format
ShareGPT / hermes-instruction-response… See the full description on the dataset page: https://huggingface.co/datasets/Somtharu181coder/educational_domain_dataset.Omni-Edu
Omni-Edu — instruction-tuning mixture
69,999 supervised instruction examples (~158M characters) covering K-12 subject
competence, curriculum grounding, diagnostic reasoning, pedagogical action and
general-purpose instruction. 12,146 rows (17.4%) are multimodal; every image
referenced by the JSONL ships in this repository under images/.
Composition
Capability family
Examples
Subject competence
31,855
Pedagogical action and scaffolding
14,226… See the full description on the dataset page: https://huggingface.co/datasets/OmniEdu/Omni-Edu.FineWeb-Edu-Spanish
High Quality Spanish Corpus
This dataset contains a sample of a large collection of high-quality Spanish text data with their metadata.
To access the full data please visit Token Haven
Creation
The dataset was created by filtering all English common crawl data for high-quality text using the FineWeb-Edu classifier with education score of 4 or higher over 5.
The data is source from the v1.0.0 of the HuggingFaceFW/fineweb-edu dataset which corresponds to… See the full description on the dataset page: https://huggingface.co/datasets/TokenHaven/FineWeb-Edu-Spanish.FineWeb-Edu-Arabic
High Quality Arabic Corpus
This dataset contains a large collection of high-quality Arabic text data with their metadata.
To access the full data please visit Token Haven
Creation
The dataset was created by filtering all English common crawl data for high-quality text using the FineWeb-Edu classifier with education score of 4 or higher over 5.
The data is source from the v1.0.0 of the HuggingFaceFW/fineweb-edu dataset which corresponds to CC-MAIN-2024-10 from… See the full description on the dataset page: https://huggingface.co/datasets/TokenHaven/FineWeb-Edu-Arabic.TinyStories-korean-eduTinyStories와 TinyStories-Korean, fineweb-edu-classifier를 이용하여 만든 데이터입니다.
TinyStories를 fineweb-edu-classifier로 평가를 한 뒤 int_score가 3 이상이면 이에 해당되는 TinyStories-Korean을 데이터에 추가하는 방식으로 제작하였습니다.
TinyStories-Korean에서 데이터 300개가 누락되어 있어서 데이터 누락이 되지 않음을 확인한 1832000번째 데이터까지만 평가하였습니다.
score 비율
3: 99.4%(501,651개)
4: 0.6%(2,931개)
FineWeb-Edu-Norwegian
High Quality Norwegian Corpus
This dataset contains a large collection of high-quality Norwegian text data with their metadata.
To access the full data please visit Token Haven
Creation
The dataset was created by filtering all English common crawl data for high-quality text using the FineWeb-Edu classifier with education score of 4 or higher over 5.
The data is source from the v1.0.0 of the HuggingFaceFW/fineweb-edu dataset which corresponds to CC-MAIN-2024-10… See the full description on the dataset page: https://huggingface.co/datasets/TokenHaven/FineWeb-Edu-Norwegian.