datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
propagator-multimodal-pretraining-data
Propagator Multimodal Pretraining Data
This public dataset contains tokenized multimodal pretraining data prepared for the Propagator model family. It combines language, image-grounded, and speech/audio-token examples into a single training format.
This is not a raw text or image browsing dataset. The examples have already been converted into compact binary token frames for model training, with a manifest that records the source groups and file layout.
Source Code… See the full description on the dataset page: https://huggingface.co/datasets/ken-sungmin/propagator-multimodal-pretraining-data.TableBench
Dataset Card for TableBench
📚 Paper
🏆 Leaderboard
💻 Code
Dataset Summary
TableBench is a comprehensive and complex
benchmark designed to evaluate Table
Question Answering (TableQA) capabilities, aligning closely with the "Reasoning Complexity of
Questions" dimension in real-world Table QA scenarios. It covers 18 question
categories
across 4 major ategories—including… See the full description on the dataset page: https://huggingface.co/datasets/Multilingual-Multimodal-NLP/TableBench.Multi-modal-Self-instruct
Dataset Description
Paper Information
Dataset Examples
Leaderboard
Dataset Usage
Data Downloading
Data Format
Evaluation
Citation
You can download the zip dataset directly, and both train and test subsets are collected in Multi-modal-Self-instruct.zip.
Dataset Description
Multi-Modal Self-Instruct dataset utilizes large language models and their code capabilities to synthesize massive abstract images and visual reasoning instructions across daily scenarios. This benchmark… See the full description on the dataset page: https://huggingface.co/datasets/zwq2018/Multi-modal-Self-instruct.Kairos-Multimodal-Reasoning
A dataset for training models in multimodal reasoning tasks
Usage
from datasets import load_dataset
ds = load_dataset("Aquiles-ai/Kairos-Multimodal-Reasoning")
print(ds.features)
print(ds["train"]["source"])
Preview of dataset examples
We've built a playground so you can see some of the examples included in the dataset.
Link: https://kairos-example.vercel.app/
Dataset used in the blog post: Kairos: Building a Multimodal Model with LFM2.5 and… See the full description on the dataset page: https://huggingface.co/datasets/Aquiles-ai/Kairos-Multimodal-Reasoning.Multimodal-Robustness-BenchmarkMedical_Multimodal_Evaluation_Data
Evaluation Guide
This dataset is used to evaluate medical multimodal LLMs, as used in HuatuoGPT-Vision. It includes benchmarks such as VQA-RAD, SLAKE, PathVQA, PMC-VQA, OmniMedVQA, and MMMU-Medical-Tracks.
To get started:
Download the dataset and extract the images.zip file.
Find evaluation code on our GitHub: HuatuoGPT-Vision.
This open-source release aims to simplify the evaluation of medical multimodal capabilities in large models. Please cite the relevant benchmark… See the full description on the dataset page: https://huggingface.co/datasets/FreedomIntelligence/Medical_Multimodal_Evaluation_Data.FinanceComplexQA
Finance-ComplexQA
Finance-ComplexQA is a bilingual Chinese-English benchmark for complex question answering in the financial domain. It is designed to evaluate whether large language models and agent systems can answer finance questions by grounding their reasoning in reference documents rather than relying only on parametric knowledge.
The dataset covers multiple financial document domains and reasoning skills, including retrieval, multi-hop reasoning, numerical calculation… See the full description on the dataset page: https://huggingface.co/datasets/Multilingual-Multimodal-NLP/FinanceComplexQA.STXBP1_PubMed_Central_Multimodal_Dataset
STXBP1 PubMed Central Multimodal Dataset v2 (12-13-2025)
A comprehensive multimodal dataset for training vision-language models on biomedical scientific literature, with focus on STXBP1-related neurological research.
🆕 Version 2 Updates (December 2025)
497,360 training examples (up from ~31K)
170,591 matched figure-image pairs (99.7% match rate)
Full captions preserved (no truncation)
Multiple training formats for different use cases
Validated response lengths for… See the full description on the dataset page: https://huggingface.co/datasets/SkyWhal3/STXBP1_PubMed_Central_Multimodal_Dataset.IndustryBench
IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs
💻Github | 📝Paper
IndustryBench is a multi-lingual benchmark for evaluating the industrial domain knowledge of large language models. It comprises 2,049 expert-curated QA pairs spanning 12 industrial sectors, with human-reviewed translations in Chinese, English, Russian, and Vietnamese.
Overview
Dimension
Details
Total questions
2,049
Languages
Chinese (zh), English (en), Russian (ru)… See the full description on the dataset page: https://huggingface.co/datasets/alibaba-multimodal-industrial-ai/IndustryBench.GeoperceptionEuclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions
Dataset Card for Geoperception
A Benchmark for Low-level Geometric Perception
Dataset Details
Dataset Description
Geoperception is a benchmark focused specifically on accessing model's low-level visual perception ability in 2D geometry.
It is sourced from the Geometry-3K corpus, which offers precise logical forms for geometric diagrams, compiled from popular high-school… See the full description on the dataset page: https://huggingface.co/datasets/euclid-multimodal/Geoperception.Multimodal-STEM-HLE-plus-plus
multimodal-STEM-HLE++
A high-value multimodal STEM dataset designed and empirically proven to push state-of-the-art LLMs beyond their current limits.
Explore the full multimodal-STEM-HLE++ dataset: https://go.turing.com/mm-stem-hle
Why This Dataset
Post-training with RL is now the primary driver of frontier model improvement. The bottleneck is finding data at the right difficulty for current SOTA models. MMLU is saturated (>90%). HLE, once considered unsolvable, is now… See the full description on the dataset page: https://huggingface.co/datasets/TuringEnterprises/Multimodal-STEM-HLE-plus-plus.AutoMemoryBench
AutoMemoryBench
State-Contract Evaluation for Auditable Agent Memory
AutoMemoryBench evaluates whether an agent uses the right memory, and only
the admissible memory, under a query-time state contract. Each executable
contract partitions memory into required, admissible, and
prohibited sets. Prohibited memories are typed as superseded, deleted,
restricted, cross-namespace, or stale-tool.
Relevance is not enough: remembered evidence must also be… See the full description on the dataset page: https://huggingface.co/datasets/Multilingual-Multimodal-NLP/AutoMemoryBench.multimodal-vision-language-video-models-2026
👁️ Multimodal Vision-Language & Video Foundation Models Dataset (2026 Edition)
A structured research dataset featuring 1,000 domain-verified research papers and code repositories focused on Multimodal Vision-Language Models (VLM), Video Foundation Models, Diffusion Transformers (DiT), Visual Grounding, and World Simulators.
Built with Universal Scientific Engine V15.1 Gold, providing 47 schema attributes with verified repository attribution, modality capability matrix, vision… See the full description on the dataset page: https://huggingface.co/datasets/beatsprom/multimodal-vision-language-video-models-2026.Standard-Multimodal-Explanation
Dataset Card for Standard Multimodal Explanation (SME)
This is a dataset for Multimodal Explanation for Visual Question Answering (MEVQA).
Dataset Details
Dataset Description
This dataset contains questions, images, answers, and the multimodal explanations of the underlying reasoning process.
The explanations are in standard English with additional [BOX] for visual grounding.
Language(s) (NLP): English
License: apache-2.0
Modality:
Language… See the full description on the dataset page: https://huggingface.co/datasets/LivXue/Standard-Multimodal-Explanation.hse-multimodal-rag-corpus
HSE Multimodal RAG Corpus
Chunks, labeled QA (including out-of-scope abstention), and published retrieval metrics.
chunks.jsonl
qa_pairs.jsonl
eval_results.json
benchmark_report.json
Iranian_olympiad_of_informatics_multimodal_questionsmultimodal-python-copilot-training-overview
Multimodal Datasets for Training Python Copilots from Source Code Analysis
Welcome to the matlok multimodal python copilot training datasets. This is an overview for our training and fine-tuning datasets found below:
~2.3M unique source coding rows
1.1M+ instruct alpaca yaml text rows updated bi-weekly
~923K png knowledge graph images with alpaca text description
~334K mp3s over ~2 years of continuous audio playtime
requires 1.5 TB storage on disk
Please reach out if you find an… See the full description on the dataset page: https://huggingface.co/datasets/matlok/multimodal-python-copilot-training-overview.lumos_multimodal_ground_iterative
🪄 Agent Lumos: Unified and Modular Training for Open-Source Language Agents
🌐[Website]
📝[Paper]
🤗[Data]
🤗[Model]
🤗[Demo]
We introduce 🪄Lumos, Language Agents with Unified Formats, Modular Design, and Open-Source LLMs. Lumos unifies a suite of complex interactive tasks and achieves competitive performance with GPT-4/3.5-based and larger open-source agents.
Lumos has following features:
🧩 Modular Architecture:
🧩 Lumos consists of planning, grounding… See the full description on the dataset page: https://huggingface.co/datasets/ai2lumos/lumos_multimodal_ground_iterative.ai-code-multimodal-fr
Dataset : IA Generation de Code, IA Multimodale & Small Language Models (FR)
Description
Dataset francophone couvrant les outils d'assistance au codage par IA, l'IA multimodale, les Small Language Models (SLM) et GraphRAG.
Ce dataset est concu pour la recherche, la formation et le developpement d'applications dans le domaine de l'IA appliquee au developpement logiciel et a la cybersecurite.
Articles couverts
IA pour la Generation de Code : Copilot, Cursor… See the full description on the dataset page: https://huggingface.co/datasets/AYI-NEDJIMI/ai-code-multimodal-fr.solarhive-community-solar-multimodal
SolarHive Community Solar Dataset
Canonical training corpus for the SolarHive family of fine-tuned Gemma 4 models. 1,727 rows (1,713 text + 14 image-grounded).
A combined text + sky-image training corpus for community solar energy intelligence. Built to fine-tune Gemma 4 into an AI energy advisor for residential solar microgrids — answering questions about production, storage, grid mix, weather impact, maintenance scheduling, and cross-source planning, with native… See the full description on the dataset page: https://huggingface.co/datasets/Truthseeker87/solarhive-community-solar-multimodal.ck12-tqa-multimodal
CK-12 TQA Multimodal: Textbook Question Answering with Images
Dataset Description
Dataset Summary
CK-12 TQA Multimodal is a comprehensive multimodal dataset for science education, containing 26,260 questions paired with 6,206 images from middle school science textbooks. This dataset is sourced from CK-12 Foundation's open educational resources and includes both text-only questions and diagram-based visual reasoning questions.
This is the complete multimodal… See the full description on the dataset page: https://huggingface.co/datasets/notefill/ck12-tqa-multimodal.ai-code-multimodal-en
Dataset: AI Code Generation, Multimodal AI & Small Language Models (EN)
Description
English dataset covering AI coding assistants, multimodal AI, Small Language Models (SLMs), and GraphRAG.
This dataset is designed for research, training, and application development in the field of AI applied to software development and cybersecurity.
Articles Covered
AI Code Generation: Copilot, Cursor, Claude Code - Comparison of 12 leading AI coding assistants
Computer… See the full description on the dataset page: https://huggingface.co/datasets/AYI-NEDJIMI/ai-code-multimodal-en.evidence-grounded-multimodal-kg-multi-lecture-reasoning
Evidence-Grounded Multimodal KG for Multi-Lecture Reasoning
This dataset contains the artifacts from an evidence-grounded multimodal knowledge graph construction pipeline over three neural-network lectures.
Contents
Raw videos, audio, and 1 FPS frames.
Faster-Whisper transcripts.
High-recall semantic anchors with transcript windows.
EasyOCR outputs for anchor frames.
Qwen2.5-VL raw extractions.
Validated concept and relationship mentions.
Canonical concepts and… See the full description on the dataset page: https://huggingface.co/datasets/sahilfarib/evidence-grounded-multimodal-kg-multi-lecture-reasoning.lumos_multimodal_plan_iterative
🪄 Agent Lumos: Unified and Modular Training for Open-Source Language Agents
🌐[Website]
📝[Paper]
🤗[Data]
🤗[Model]
🤗[Demo]
We introduce 🪄Lumos, Language Agents with Unified Formats, Modular Design, and Open-Source LLMs. Lumos unifies a suite of complex interactive tasks and achieves competitive performance with GPT-4/3.5-based and larger open-source agents.
Lumos has following features:
🧩 Modular Architecture:
🧩 Lumos consists of planning, grounding… See the full description on the dataset page: https://huggingface.co/datasets/ai2lumos/lumos_multimodal_plan_iterative.dutch-central-exam-mcq-multimodal-subsetMultimodal Multiple Choice Questions of the Dutch Central Exam 1999-2024
What?
This dataset contains only multimodal multiple choice questions from the Dutch Central Exam (High School level). From Wikipedia:
The Eindexamen (Dutch pronunciation: [ˈɛi̯ntɛksamən]) or centraal examen (CE) is the matriculation exam in the Netherlands, which takes place in a student's final year of high school education (voortgezet onderwijs; "continued education"). The exam is regulated by the Dutch Secondary… See the full description on the dataset page: https://huggingface.co/datasets/jjzha/dutch-central-exam-mcq-multimodal-subset.multimodal-privacy
Auditing M-LLMs for Privacy Risks: A Synthetic Benchmark and Evaluation Framework
Recent advances in multi-modal Large Language Models (M-LLMs) have demonstrated a powerful ability to synthesize implicit information from disparate sources, including images and text. These resourceful data from social media also introduce a significant and underexplored privacy risk: the inference of sensitive personal attributes from seemingly daily media content. However, the lack of benchmarks and… See the full description on the dataset page: https://huggingface.co/datasets/xaddh/multimodal-privacy.Driving_License_Nepali_MultimodalThis dataset is curated as part of the Cohere4AI project called "Multimodal-Multilingual Exam Collection".
GATE_2022_Multimodal
GATE 2022 MULTIMODAL
This dataset has been curated as part of Cohere For AI's multimodal examination benchmark creation.
PSC_MultimodalThis dataset is curated as part of the Cohere4AI project called "Multimodal-Multilingual Exam Collection".
multimodal_mcq_greek_physics