datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
propagator-multimodal-pretraining-data
Propagator Multimodal Pretraining Data
This public dataset contains tokenized multimodal pretraining data prepared for the Propagator model family. It combines language, image-grounded, and speech/audio-token examples into a single training format.
This is not a raw text or image browsing dataset. The examples have already been converted into compact binary token frames for model training, with a manifest that records the source groups and file layout.
Source Code… See the full description on the dataset page: https://huggingface.co/datasets/ken-sungmin/propagator-multimodal-pretraining-data.MultiChartQA
MultiChartQA
This repository contains the questions and answers for our Multi-chart Benchmark. At present, only the data is available, but the test code will be provided soon. We welcome everyone to use and explore our benchmark!
Introduction
MultiChartQA is an extensive and demanding benchmark that features real-world charts. We source charts from various places to ensure both diversity and completeness. Each multi-chart group includes 2 or 3 charts, and each group is… See the full description on the dataset page: https://huggingface.co/datasets/Zifeng618/MultiChartQA.Multi-modal-Self-instruct
Dataset Description
Paper Information
Dataset Examples
Leaderboard
Dataset Usage
Data Downloading
Data Format
Evaluation
Citation
You can download the zip dataset directly, and both train and test subsets are collected in Multi-modal-Self-instruct.zip.
Dataset Description
Multi-Modal Self-Instruct dataset utilizes large language models and their code capabilities to synthesize massive abstract images and visual reasoning instructions across daily scenarios. This benchmark… See the full description on the dataset page: https://huggingface.co/datasets/zwq2018/Multi-modal-Self-instruct.remote_sensing_VQA_multilingual
Remote Sensing VQA — Multilingual
A multilingual counterfactual MCQ dataset built from remote sensing / satellite imagery.
Each row contains a satellite image, two captions (original vs counterfactual), and a multiple-choice question probing whether a VLM follows the image or the misleading text.
Languages
Language
Code
Rows
English
en
50
Hindi
hi
50
Urdu
ur
50
Telugu
te
50
Bahasa Indonesia
id
50
Columns
Column
Type… See the full description on the dataset page: https://huggingface.co/datasets/apart-global-south-hack/remote_sensing_VQA_multilingual.Kairos-Multimodal-Reasoning
A dataset for training models in multimodal reasoning tasks
Usage
from datasets import load_dataset
ds = load_dataset("Aquiles-ai/Kairos-Multimodal-Reasoning")
print(ds.features)
print(ds["train"]["source"])
Preview of dataset examples
We've built a playground so you can see some of the examples included in the dataset.
Link: https://kairos-example.vercel.app/
Dataset used in the blog post: Kairos: Building a Multimodal Model with LFM2.5 and… See the full description on the dataset page: https://huggingface.co/datasets/Aquiles-ai/Kairos-Multimodal-Reasoning.vlm-projects-multi-lang-final-v2
My Final Multilingual Medical VQA Dataset
This dataset is organized into multiple configurations (subsets), one for each language.
You can load a specific language subset like this:
from datasets import load_dataset
# Load the Vietnamese training data
vi_train = load_dataset("tungvu3196/vlm-projects-multi-lang-final-v2", "Vietnamese", split="train")
# Load the English testing data
en_test = load_dataset("tungvu3196/vlm-projects-multi-lang-final-v2", "English", split="test")
counterfactual-pendulum-multilingual
📌 Dataset Summary
When a Vision-Language Model (VLM) is given an image along with a text prompt containing contradictory or misleading information, how does it react? Does it rely on the visual evidence, succumb to textual bias, or honestly abstain when faced with unresolvable conflict?
This dataset adapts the Counterfactual Pendulum scenario across two visual conflict dimensions:
Angular (Angle): Conflict in the pendulum's angle of inclination.
Light: Conflict in the light… See the full description on the dataset page: https://huggingface.co/datasets/apart-global-south-hack/counterfactual-pendulum-multilingual.MultiUI
MulitUI
Dataset for the paper: Harnessing Webpage Uis For Text Rich Visual Understanding
🌐 Homepage | 🐍 GitHub | 📖 arXiv
Introduction
We introduce MultiUI, a dataset containing 7.3 million samples from 1 million websites, covering diverse multi- modal tasks and UI layouts. Models trained on MultiUI not only excel in web UI tasks—achieving up to a 48% improvement on VisualWebBench and a 19.1% boost in action accuracy on a web agent dataset Mind2Web—but also… See the full description on the dataset page: https://huggingface.co/datasets/neulab/MultiUI.Multimodal-Robustness-BenchmarkMedical_Multimodal_Evaluation_Data
Evaluation Guide
This dataset is used to evaluate medical multimodal LLMs, as used in HuatuoGPT-Vision. It includes benchmarks such as VQA-RAD, SLAKE, PathVQA, PMC-VQA, OmniMedVQA, and MMMU-Medical-Tracks.
To get started:
Download the dataset and extract the images.zip file.
Find evaluation code on our GitHub: HuatuoGPT-Vision.
This open-source release aims to simplify the evaluation of medical multimodal capabilities in large models. Please cite the relevant benchmark… See the full description on the dataset page: https://huggingface.co/datasets/FreedomIntelligence/Medical_Multimodal_Evaluation_Data.MultiBBQ
MultiBBQ: A Fairness Benchmark for Multimodal LLMs
Controllable diagnosis of social bias in multimodal LLMs with synthetic images.
MultiBBQ is a fairness evaluation benchmark for multimodal large language models (MLLMs).
It extends the language-only BBQ benchmark into the visual
domain: each attested social bias is paired with an AI-generated photorealistic image of two
people who differ only in the target demographic, so a model's fairness can be… See the full description on the dataset page: https://huggingface.co/datasets/MLL-Lab/MultiBBQ.GeoperceptionEuclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions
Dataset Card for Geoperception
A Benchmark for Low-level Geometric Perception
Dataset Details
Dataset Description
Geoperception is a benchmark focused specifically on accessing model's low-level visual perception ability in 2D geometry.
It is sourced from the Geometry-3K corpus, which offers precise logical forms for geometric diagrams, compiled from popular high-school… See the full description on the dataset page: https://huggingface.co/datasets/euclid-multimodal/Geoperception.code-conflict
Code Conflict Dataset
A dataset of 100 visual Python code conflict samples designed to evaluate Vision-Language Models (VLMs) under cross-modal conflicts (discrepancy between code screenshots and caption text).
Dataset Statistics
Total Rows: 100 samples
Language: English (english)
Categories: 5 distinct Python code conflict_types (20 samples per category):
operator_substitution (Rows 1–20): Swapping math or logic operators (e.g., + to -, == to !=, or to and).… See the full description on the dataset page: https://huggingface.co/datasets/multilingual-vlm-conflict/code-conflict.Multimodal-STEM-HLE-plus-plus
multimodal-STEM-HLE++
A high-value multimodal STEM dataset designed and empirically proven to push state-of-the-art LLMs beyond their current limits.
Explore the full multimodal-STEM-HLE++ dataset: https://go.turing.com/mm-stem-hle
Why This Dataset
Post-training with RL is now the primary driver of frontier model improvement. The bottleneck is finding data at the right difficulty for current SOTA models. MMLU is saturated (>90%). HLE, once considered unsolvable, is now… See the full description on the dataset page: https://huggingface.co/datasets/TuringEnterprises/Multimodal-STEM-HLE-plus-plus.rpg-conflict
RPG Fantasy Battle Conflict Dataset
A dataset of 100 visual RPG combat conflict samples designed to evaluate Vision-Language Models (VLMs) under cross-modal conflicts (discrepancy between battle screenshots and caption text). Derived from the rcannizzaro/rpg_fantasy_battle_counterfactual_v2 dataset.
Dataset Statistics
This dataset consists of a single train split containing 100 perfectly isolated conflict samples derived from the… See the full description on the dataset page: https://huggingface.co/datasets/multilingual-vlm-conflict/rpg-conflict.ChinaHeritaQA
Images
This folder contains visual data for the ChinaHeritaQA benchmark: https://arxiv.org/abs/2606.08959
Contents
Folder
Description
Image_data/
Chinese UNESCO World Heritage Site images (2,279 images from 51 sites)
worlds_data/
Non-Chinese World Heritage Site images (133 images from 23 sites)
Overview
The image dataset includes a comprehensive collection of photographs from both Chinese and international UNESCO World Heritage… See the full description on the dataset page: https://huggingface.co/datasets/Multilingual-NLP/ChinaHeritaQA.adaption-multilingual-doc-qa
This dataset is a remastered version of Reubencf/magazines-multilingual-vqa prepared using Adaption's Adaptive Data platform.
multilingual_doc_qa
This dataset contains multilingual question-answer pairs focused on extracting specific factual details from documents (page numbers, names, ages, dates, counts, titles, etc.). Each entry consists of a prompt asking for a specific detail and a completion providing the precise answer grounded in the source page text. Cross-lingual: the… See the full description on the dataset page: https://huggingface.co/datasets/Reubencf/adaption-multilingual-doc-qa.counterfactual-pendulum-multilingual
📌 Dataset Summary
When a Vision-Language Model (VLM) is given an image along with a text prompt containing contradictory or misleading information, how does it react? Does it rely on the visual evidence, succumb to textual bias, or honestly abstain when faced with unresolvable conflict?
This dataset adapts the Counterfactual Pendulum scenario across two visual conflict dimensions:
Angular (Angle): Conflict in the pendulum's angle of inclination.
Light: Conflict in the light… See the full description on the dataset page: https://huggingface.co/datasets/akanshjain37/counterfactual-pendulum-multilingual.3D-Object-ConflictIranian_olympiad_of_informatics_multimodal_questionsMulti-CritarXiv-AI-papers-multi-vector
Overview
This is a dataset containing individual pages from the top-40 most cited AI papers on arXiv](https://arxiv.org/abs/2412.12121) from the period 2023-01-01 to 2024-09-30.
Only the first 10 pages from each paper is included.
The dataset includes an image of each page as well as a multi-vector embedding using vidore/colqwen2-v1.0.
vlm-projects-multi-lang-final
My Final Multilingual Medical VQA Dataset
This dataset is organized into multiple configurations (subsets), one for each language.
You can load a specific language subset like this:
from datasets import load_dataset
vi_train = load_dataset("tungvu3196/vlm-projects-multi-lang-final", "Vietnamese", split="train")
multimodel_llava_med_zh_instruct_60kBorrowed from https://huggingface.co/datasets/BUAADreamer/llava-med-zh-instruct-60k
Fix the <image> placeholder issue, which will cause error during training:
raise ValueError(f"The number of images does not match the number of {IMAGE_PLACEHOLDER} tokens.")
ck12-tqa-multimodal
CK-12 TQA Multimodal: Textbook Question Answering with Images
Dataset Description
Dataset Summary
CK-12 TQA Multimodal is a comprehensive multimodal dataset for science education, containing 26,260 questions paired with 6,206 images from middle school science textbooks. This dataset is sourced from CK-12 Foundation's open educational resources and includes both text-only questions and diagram-based visual reasoning questions.
This is the complete multimodal… See the full description on the dataset page: https://huggingface.co/datasets/notefill/ck12-tqa-multimodal.dutch-central-exam-mcq-multimodal-subsetMultimodal Multiple Choice Questions of the Dutch Central Exam 1999-2024
What?
This dataset contains only multimodal multiple choice questions from the Dutch Central Exam (High School level). From Wikipedia:
The Eindexamen (Dutch pronunciation: [ˈɛi̯ntɛksamən]) or centraal examen (CE) is the matriculation exam in the Netherlands, which takes place in a student's final year of high school education (voortgezet onderwijs; "continued education"). The exam is regulated by the Dutch Secondary… See the full description on the dataset page: https://huggingface.co/datasets/jjzha/dutch-central-exam-mcq-multimodal-subset.MultiTaskNLP-TestDataset
MultiTaskNLP-Dataset
1. Introduction
The MultiTaskNLP-Dataset has undergone significant quality improvements through iterative data curation. In the latest version, we have substantially enhanced the data completeness and label accuracy by implementing rigorous annotation protocols and multi-stage quality assurance mechanisms. The dataset demonstrates outstanding quality metrics across various dimensions, including completeness, accuracy… See the full description on the dataset page: https://huggingface.co/datasets/toolevalxm/MultiTaskNLP-TestDataset.solarhive-community-solar-multimodal
SolarHive Community Solar Dataset
Canonical training corpus for the SolarHive family of fine-tuned Gemma 4 models. 1,727 rows (1,713 text + 14 image-grounded).
A combined text + sky-image training corpus for community solar energy intelligence. Built to fine-tune Gemma 4 into an AI energy advisor for residential solar microgrids — answering questions about production, storage, grid mix, weather impact, maintenance scheduling, and cross-source planning, with native… See the full description on the dataset page: https://huggingface.co/datasets/Truthseeker87/solarhive-community-solar-multimodal.multimodal-privacy
Auditing M-LLMs for Privacy Risks: A Synthetic Benchmark and Evaluation Framework
Recent advances in multi-modal Large Language Models (M-LLMs) have demonstrated a powerful ability to synthesize implicit information from disparate sources, including images and text. These resourceful data from social media also introduce a significant and underexplored privacy risk: the inference of sensitive personal attributes from seemingly daily media content. However, the lack of benchmarks and… See the full description on the dataset page: https://huggingface.co/datasets/xaddh/multimodal-privacy.GATE_2022_Multimodal
GATE 2022 MULTIMODAL
This dataset has been curated as part of Cohere For AI's multimodal examination benchmark creation.