datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
NEXUS-temporal_hierarchical_multi-modal
NEXUS: Neural Evolution for eXtensible Universal Semantics Dataset
(Temporal Multimodal Slices)
This dataset is a multi-modal, hierarchical, temporal representation derived from HuggingFaceFV/finevideo. It is designed for streaming training where the primary unit is a 10 ms "slice" that aggregates upward into moments (100 ms), seconds (1 s), experiences (10 s), and minutes (60 s).
It is meant to represent an extensible stream of "experience" as there are… See the full description on the dataset page: https://huggingface.co/datasets/Ardea/NEXUS-temporal_hierarchical_multi-modal.Multi-modal-Self-instruct
Dataset Description
Paper Information
Dataset Examples
Leaderboard
Dataset Usage
Data Downloading
Data Format
Evaluation
Citation
You can download the zip dataset directly, and both train and test subsets are collected in Multi-modal-Self-instruct.zip.
Dataset Description
Multi-Modal Self-Instruct dataset utilizes large language models and their code capabilities to synthesize massive abstract images and visual reasoning instructions across daily scenarios. This benchmark… See the full description on the dataset page: https://huggingface.co/datasets/zwq2018/Multi-modal-Self-instruct.MixBench2026
MixBench: A Benchmark for Mixed Modality Retrieval
MixBench is a benchmark for evaluating retrieval across text, images, and multimodal documents. It is designed to test how well retrieval models handle queries and documents that span different modalities, such as pure text, pure images, and combined image+text inputs.
MixBench includes four subsets, each curated from a different data source:
MSCOCO
Google_WIT
VisualNews
OVEN
Each subset contains:
queries.jsonl: each entry… See the full description on the dataset page: https://huggingface.co/datasets/mixed-modality-search/MixBench2026.coco-modality-equivalencecoco-modality-equivalenceOptical-Reasoning-4k
Optical Reasoning
Overview
Optical Reasoning contains 3,907 rendered visual rationales used in "Optical Reasoning: Rethinking Images as an Expressive Reasoning Medium Beyond Text". It covers 5 benchmarks, including typographic rationales for all benchmarks and graphical rationales for AQuA-RAT.
AQuA-RAT: Multiple-choice algebra and quantitative reasoning problems with five answer options.
GPQA Diamond: Graduate-level multiple-choice science questions spanning… See the full description on the dataset page: https://huggingface.co/datasets/ModalityDance/Optical-Reasoning-4k.gsm8k-rendered-vlm-v2
GSM8K Rendered-VL v2
1319 rendered GSM8K test problems for the VLM modality study (Phase 1).
Contributors
Rodela Ghosh — study design, pilot (v1), dataset packaging and Hugging Face release (scripts/prepare_hf_v2_release.py)
Aviral Gupta — benchmark infrastructure (src/), v2 rendering protocol (src/rendering.py), Phase 1 model runs
Code: https://github.com/Ro-netizen004/vlm-modality-research
Not interchangeable with v1: RodelaG/gsm8k-rendered-vlm
v1
v2… See the full description on the dataset page: https://huggingface.co/datasets/vlm-modality-research/gsm8k-rendered-vlm-v2.AR-Omni-Instruct-v0.1
AR-Omni-Instruct
Overview
AR-Omni-Instruct is a multimodal instruction-tuning dataset for training unified autoregressive any-to-any models.
All modalities are represented as discrete tokens in a single interleaved token stream, enabling standard next-token prediction training over multimodal sequences.
Dataset Summary
Type: multimodal instruction-tuning data
Format: discrete tokenized multimodal conversations / sequences
Use case: instruction tuning… See the full description on the dataset page: https://huggingface.co/datasets/ModalityDance/AR-Omni-Instruct-v0.1.multi-modal-image-editmodality-conflict-arbitration-v2
Modality-Conflict Arbitration Benchmark (v2)
A controlled benchmark for studying how a vision-language model arbitrates between
its two input channels when they disagree — and whether that choice tracks the
reliability of each channel.
Each row is a single conflict trial: an image of one math problem paired with the
text of a different problem. Because the two ground-truth answers are carried side by
side, the model's output alone tells you which modality it followed — no… See the full description on the dataset page: https://huggingface.co/datasets/vlm-modality-research/modality-conflict-arbitration-v2.brain_modalitytrue_imgs_modalities
Dataset Summary
ESmike/true_imgs_modalities is a multimodal dataset containing 4,149 real images at a uniform resolution of 512 × 512, along with multiple derived modalities for each image.
This dataset is intended for research and experimentation in computer vision, generative modeling, and multimodal learning.
All images originate from real photographs curated and processed. Each base image is provided alongside additional modality representations (details in the Data Fields… See the full description on the dataset page: https://huggingface.co/datasets/ESmike/true_imgs_modalities.chartqa-evidence-conflict-v2
ChartQA-Conflict v2
This dataset contains 230 reviewed conflicts between a native ChartQA chart and
an evidence-bearing textual report. The original chart supports one answer,
while counterfactual facts in the report support a distinct answer to the same
question. Neither source is privileged in the evaluation prompt.
Each row contains the original chart, shared question, chart-supported answer,
report-supported answer, evidence-bearing report, unit and counterfactual
strategy… See the full description on the dataset page: https://huggingface.co/datasets/vlm-modality-research/chartqa-evidence-conflict-v2.Modality-Aligntranslate-Multi-modal-Self-instruct
Translated https://huggingface.co/datasets/zwq2018/Multi-modal-Self-instruct
Translate to Malay using https://mesolitica.com/translation Base model, a nice dataset for visual QA charts, tables, simulated maps, dashboards, flowcharts, relation graphs, floor plans, and visual puzzles.
chartqa-evidence-conflict-table-v1
ChartQA-Conflict chart/table representation ablation
This derivative contains 229 audited ChartQA-Conflict items. Each row provides
two visual renderings of the same official ChartQA facts:
chart_image: the original ChartQA chart;
table_image: a plain table image rendered from the corresponding official
ChartQA CSV.
The shared question, chart-supported answer, evidence-bearing conflicting
report, report-supported answer, and Source A/B assignment are identical across
the two… See the full description on the dataset page: https://huggingface.co/datasets/vlm-modality-research/chartqa-evidence-conflict-table-v1.multimodal-modality-conflict-datasetmulti-modal_offensive_mememulti_modal_sampleconflicting_modalityheroicons_modalmulti_modal_sportsmath-rendered-vlm-v1chartqa-evidence-conflict-v1
ChartQA Evidence Conflict
This dataset contains 230 reviewed conflicts between a native ChartQA chart and
an evidence-bearing textual report. The original chart supports one answer,
while counterfactual facts in the report support a distinct answer to the same
question. Neither source is privileged in the evaluation prompt.
Each row contains the original chart, shared question, chart-supported answer,
report-supported answer, evidence-bearing report, unit and counterfactual… See the full description on the dataset page: https://huggingface.co/datasets/vlm-modality-research/chartqa-evidence-conflict-v1.modality-imbalens-circlesmulti_modal_sampleaqua-rat-rendered-vlm-v1multi_modalmodal-test-dataset
Dataset Card for "modal-test-dataset"
More Information needed
modality-conflict-dataset