datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
general-instruction-augmented-corpora
Instruction Pre-Training: Language Models are Supervised Multitask Learners (EMNLP 2024)
This repo contains the general instruction-augmented corpora (containing 200M instruction-response pairs covering 40+ task categories) used in our paper Instruction Pre-Training: Language Models are Supervised Multitask Learners.
We explore supervised multitask pre-training by proposing Instruction Pre-Training, a framework that scalably augments massive raw corpora with instruction-response… See the full description on the dataset page: https://huggingface.co/datasets/instruction-pretrain/general-instruction-augmented-corpora.General-Bench-Closeset
On Path to Multimodal Generalist: General-Level and General-Bench
[📖 Project]
[🏆 Leaderboard]
[📄 Paper]
[🤗 Paper-HF]
[🤗 Dataset-HF]
[📝 Dataset-Github]
Close Set of General-Bench
We divide our General-Bench into two settings: open and close.
This is the Close Set, where we release only the sample inputs—without ground-truth answers—for 🏆 Leaderboard purpose.
To participate the leaderboard, please follow the detailed instructions to submit the evaluation results (submission).… See the full description on the dataset page: https://huggingface.co/datasets/General-Level/General-Bench-Closeset.shofo-tiktok-general-small
Shofo TikTok General (Small)
Overview
Shofo TikTok General (Small) is a dataset containing 50,000 TikTok videos with comprehensive metadata, transcripts, comments, and engagement metrics. This is a curated subset of Shofo's larger TikTok index, which contains hundreds of millions of indexed videos.
Size: ~50K videos (~500GB)
Modality: Video + Audio + Text (transcripts, comments, captions)
Source: TikTok
Schema
Column
Type
Description
file_name… See the full description on the dataset page: https://huggingface.co/datasets/Shofo/shofo-tiktok-general-small.SlideChat
Introduction
This repository provides the dataset resources used for training and evaluating SlideChat, a multimodal large language model for whole-slide pathology image understanding.
The dataset includes both instruction-following training data and VQA/Caption evaluation benchmarks across multiple pathology cohorts and tasks.
Contents
Training Instruction Data
SlideInstruct_train_stage1_caption.json: Slide-level caption instruction data used for Stage-1… See the full description on the dataset page: https://huggingface.co/datasets/General-Medical-AI/SlideChat.GeneralThought-430K-filteredData from https://huggingface.co/datasets/GeneralReasoning/GeneralThought-430K, removed prompts with non commercial data
RADAR-auxiliary-data
RADAR: Preprocessed Anatomical Masks for Merlin CT Data
This dataset provides preprocessed anatomical segmentation masks for the Merlin abdominal CT training set, generated by TotalSegmentator and post-processed for use with the RADAR framework. These masks enable anatomy-aware vision–language pretraining without any additional manual annotation.
Overview
RADAR is a generalist vision–language model trained on over 400,000 contrast-enhanced abdominal CT… See the full description on the dataset page: https://huggingface.co/datasets/radar-generalist/RADAR-auxiliary-data.GMAI-VL-5.5M
GMAI-VL-5.5M Dataset
GMAI-VL-5.5M is a comprehensive, large-scale medical General Medical AI Vision-Language (GMAI-VL) dataset built specifically for training multimodal foundation models in the medical domain. It contains an extraordinary scale of high-quality instructions encompassing over 5.5 million multimodal question-answering pairs, carefully constructed based on hundreds of medical classification, segmentation, and detection datasets.
This repository… See the full description on the dataset page: https://huggingface.co/datasets/General-Medical-AI/GMAI-VL-5.5M.GeneralThoughtArchive
GeneralThought-430K
Thought wants to be free
Open reasoning data for March 14 2025. This dataset was part of a side-project in the weeks following the R1 release by Chengxi and Ross - we are no longer maintaining this
dataset but are archiving it here.
The dataset contains questions, reference answers, reasoning traces, final answers and other metadata from several popular reasoning models including DeepSeek-R1, DeepSeek-R1-Zero, OpenThoughts-32B, LIMO… See the full description on the dataset page: https://huggingface.co/datasets/RJT1990/GeneralThoughtArchive.General-Bench-Openset
On Path to Multimodal Generalist: General-Level and General-Bench
[📖 Project]
[🏆 Leaderboard]
[📄 Paper]
[🤗 Paper-HF]
[🤗 Dataset-HF (Close-Set)]
[🤗 Dataset-HF (Open-Set)]
[📝 Github]
Open Set of General-Bench
We divide our General-Bench into two settings: Open and Close.
This is the Open Set repo, where we release the full ground-truth annotations for all datasets, allowing to train and evaluate models for open research purpose.
If you wish to rank on our 🏆 leaderboard, please… See the full description on the dataset page: https://huggingface.co/datasets/General-Level/General-Bench-Openset.general-reasoning-ift-pairs
Reasoning-IFT Pairs (General Domain)
This dataset provides the largest set of IFT and Reasoning answers pairs for a set of general domain queries (cf: math-domain).It is based on the Infinity-Instruct dataset, an extensive and high-quality collection of instruction fine-tuning data.
We curated 900k queries from the 7M_core subset of Infinity-Instruct, which covers multiple domains including general knowledge, commonsense Q&A, coding, and math.For each query… See the full description on the dataset page: https://huggingface.co/datasets/Scale-or-Reason/general-reasoning-ift-pairs.GMAI-Reasoning10K
GMAI-Reasoning10K
Medical Reasoning dataset used in GMAI-VL-R1
Data description
GMAI-Reasoning10K is a high-quality medical image reasoning dataset containing 10,000 carefully selected samples. The data was collected from 95 medical datasets from reliable sources such as Kaggle, GrandChallenge, and Open-Release, covering 12 imaging modalities including X-ray, CT, and MRI.
Data preprocessing followed the standardization methods from SAMed-20M: 3D data (CT/MRI) had… See the full description on the dataset page: https://huggingface.co/datasets/General-Medical-AI/GMAI-Reasoning10K.OpenOneRec-General-Pretrain
通用文本数据集
本目录包含 OpenOneRec 项目使用的通用文本数据集信息。这些数据集均来自 HuggingFace，经过清洗处理和对齐到项目统一的数据格式，并转换为 Parquet 格式用于训练。
数据格式说明
所有数据集均已转换为统一的 Parquet 格式，符合项目的数据格式规范（参考 ../README.md）。数据格式支持：
Segments 格式：用于普通文本数据，使用 segments 字段存储文本段落列表
Chat 格式：用于对话数据，使用 messages 字段存储对话消息列表
每个 Parquet 文件包含以下核心字段：
uuid: 唯一标识符
source: 数据来源标识
metadata: JSON 格式的元数据字典
segments 或 messages: 文本内容（根据数据类型选择）
详细的数据格式规范请参考 ../README.md。
数据集列表
数据集名称
样本数量
HuggingFace 仓库
reasoning_v1_20m
1,666… See the full description on the dataset page: https://huggingface.co/datasets/OpenOneRec/OpenOneRec-General-Pretrain.steady-rans-generalization
Steady-RANS cross-family generalization dataset
Data for the paper "Towards generalized flow field prediction: one model across unseen
object families" (under double blind review; this account is anonymous for that reason).
Trained checkpoints and evaluation code are in the companion model repo:
steady-rans-surrogates.
Steady incompressible k-omega SST (OpenFOAM simpleFoam) external flow around 855 distinct
shapes (17 scripted parametric families plus 40 ModelNet object… See the full description on the dataset page: https://huggingface.co/datasets/BlidReview/steady-rans-generalization.AITW_GeneralVisionThink-General-Train
VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning
Senqiao/VisionThink-General-Train
This is the training dataset used for our Reasoning VLM on general VQA tasks.
VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning [Paper]
Senqiao Yang,
Junyi Li,
Xin Lai,
Bei Yu,
Hengshuang Zhao,
Jiaya Jia
Highlights
Our VisionThink leverages reinforcement learning to autonomously learn whether to… See the full description on the dataset page: https://huggingface.co/datasets/Senqiao/VisionThink-General-Train.general-master-en-202608
General · Master · English · 2026-08
English pretraining text, assembled from three public sources, cleaned with one
character-level cleaner, and filtered for repetition.
109,337,531 documents and 468,064,046,462 characters.
Composition
Config
Documents
Characters
What it is
fineweb-edu-dedup
65,010,430
297,544,916,118
Web text an educational classifier kept
cosmopedia-v2
38,591,146
144,011,993,012
Synthetic prose from a seeded generator… See the full description on the dataset page: https://huggingface.co/datasets/Brainquiver/general-master-en-202608.general-sft-utk-10B
Dataset: general-sft-utk-10B
This dataset was uploaded from /mnt/yulan_pretrain/mount/data_final_train/general-sft-utk-10B/no-curriculum/tmp.
Objaverse-General-Find3DThis dataset contains benchmarks for open-world object part segmentation proposed by Find Any Part in 3D (ICCV 2025).
This dataset includes two human-annotated benchmarks: Objaverse-General (of 100 object categories) and ShapeNetPart-Objaverse (of the same categories of ShapeNetPart, but with objects source from Objaverse to study distribution shift).
Usage
Inside both objaverse-general and objaverse-shapanetepart directories, the benchmark has the following directory structure:… See the full description on the dataset page: https://huggingface.co/datasets/ziqima/Objaverse-General-Find3D.shadow-dance-artifacts
Shadow-Dance private runtime artifacts
marqo-general-ecommerce-evalgeneral-reasoning-ift-pairs
Reasoning-IFT Pairs (General Domain)
This dataset provides the largest set of IFT and Reasoning answers pairs for a set of general domain queries (cf: math-domain).It is based on the Infinity-Instruct dataset, an extensive and high-quality collection of instruction fine-tuning data.
We curated 900k queries from the 7M_core subset of Infinity-Instruct, which covers multiple domains including general knowledge, commonsense Q&A, coding, and math.For each query, we… See the full description on the dataset page: https://huggingface.co/datasets/Sidsidney/general-reasoning-ift-pairs.OpenOneRec-General-SFT
SFT 通用文本数据集
本目录包含 OpenOneRec 项目 SFT（Supervised Fine-Tuning）阶段使用的通用文本数据集信息。这些数据集均来自 HuggingFace，经过清洗处理和对齐到项目统一的数据格式，并转换为 Parquet 格式用于训练。
数据格式说明
所有数据集均已转换为统一的 Parquet 格式，符合项目的数据格式规范（参考 ../README.md）。SFT 数据主要使用 Chat 格式，使用 messages 字段存储对话消息列表。
每个 Parquet 文件包含以下核心字段：
uuid: 唯一标识符
source: 数据来源标识
metadata: JSON 格式的元数据字典
messages: 对话消息列表（JSON 格式），包含 role（user/assistant/system）和 content 字段
详细的数据格式规范请参考 ../README.md。
数据集列表
数据集名称
样本数量
HuggingFace 仓库… See the full description on the dataset page: https://huggingface.co/datasets/OpenOneRec/OpenOneRec-General-SFT.moda-general-capability-rollouts
MODA General Capability Retention Rollouts
This dataset contains the raw model generations and evaluation results for the
MODA general-capability retention experiments. It covers 16 models, seven
benchmarks, 260,592 prompt records, and 2,605,920 stored generations.
The evaluation code is pinned to source commit
12ea99b2a57a354f2b7d6792f62a3d9313192fa7.
Evaluation protocol
Benchmarks: GSM8K, MMLU abstract_algebra, GPQA Diamond, BoolQ,
HellaSwag, TruthfulQA, and… See the full description on the dataset page: https://huggingface.co/datasets/Hkang/moda-general-capability-rollouts.General-Knowledge
Dataset Card for Dataset Name
Dataset Summary
The dataset is a collection of questions and answers themed on general facts and reasoning. The dataset is divided into two features - 'Question' and 'Answer'.
It is meant to be used for training a model to be good at general knowledge and reasoning. This dataset is inspired from the Alpaca dataset, and infact contains a subset of the alpaca dataset in itself.
Distribution
The distribution of the… See the full description on the dataset page: https://huggingface.co/datasets/MuskumPillerum/General-Knowledge.general-web-it-202608
General · Web · Italian · 2026-08
Italian pretraining text, built from the Italian portion of EPFL's FineWeb2-HQ, which is the
high quality slice of FineWeb-2. Every document passes one character-level cleaner and a
repetition filter.
21,065,052 documents and 66,158,573,443 characters of Italian prose.
Contents
Config
Documents
Characters
Upstream
fineweb2-hq-ita_Latn
21,065,052
66,158,573,443
epfml/FineWeb2-HQ, ita_Latn
The character count is… See the full description on the dataset page: https://huggingface.co/datasets/Brainquiver/general-web-it-202608.general-web-fr-202608
General · Web · French · 2026-08
French pretraining text, built from the French portion of EPFL's FineWeb2-HQ, which is the
high quality slice of FineWeb-2. Every document passes one character-level cleaner and a
repetition filter.
31,999,309 documents and 118,346,333,763 characters of French prose.
Contents
Config
Documents
Characters
Upstream
fineweb2-hq-fra_Latn
31,999,309
118,346,333,763
epfml/FineWeb2-HQ, fra_Latn
The character count is exact.… See the full description on the dataset page: https://huggingface.co/datasets/Brainquiver/general-web-fr-202608.db-sfw-512px-general-filter
Danbooru SFW 512px General Filter
This dataset is meant to be used for training a simple binary classifier that can filter the
Danbooru SFW 2021 dataset. There are two classes: "accepted" and "rejected", with "accepted"
representing samples that should pass through the filter and "rejected" representing samples
that should not.
Generallly, to be accepted a sample should meet the following criteria:
No horizontal/vertical bars, or bars that can be removed easily by filling in… See the full description on the dataset page: https://huggingface.co/datasets/hayden-donnelly/db-sfw-512px-general-filter.google-shopping-general-eval
Marqo Ecommerce Embedding Models
In this work, we introduce the GoogleShopping-1m dataset for evaluation. This dataset comes with the release of our state-of-the-art embedding models for ecommerce products: Marqo-Ecommerce-B and Marqo-Ecommerce-L.
Released Content:
Marqo-Ecommerce-B and Marqo-Ecommerce-L embedding models
GoogleShopping-1m and AmazonProducts-3m for evaluation
Evaluation Code
The benchmarking results show that the… See the full description on the dataset page: https://huggingface.co/datasets/Marqo/google-shopping-general-eval.SFT_Chinese_GeneralBenchMAX_General_Translation
Dataset Sources
Paper: BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models
Link: https://huggingface.co/papers/2502.07346
Repository: https://github.com/CONE-MT/BenchMAX
Dataset Description
BenchMAX_General_Translation is a dataset of BenchMAX, which evaluates the translation capability on the general domain.
We collect parallel test data from Flore-200, TED-talk, and WMT24.
Usage
Run the following commands to generate… See the full description on the dataset page: https://huggingface.co/datasets/LLaMAX/BenchMAX_General_Translation.