datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
MMStar
MMStar (Are We on the Right Way for Evaluating Large Vision-Language Models?)
🌐 Homepage | 🤗 Dataset | 🤗 Paper | 📖 arXiv | GitHub
Dataset Details
As shown in the figure below, existing benchmarks lack consideration of the vision dependency of evaluation samples and potential data leakage from LLMs' and LVLMs' training data.
Therefore, we introduce MMStar: an elite vision-indispensible multi-modal benchmark, aiming to ensure each curated sample exhibits… See the full description on the dataset page: https://huggingface.co/datasets/Lin-Chen/MMStar.MMSU
[ICLR 2026] MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
Overview of MMSU
MMSU (Massive Multi-task Spoken Language Understanding and Reasoning Benchmark) is a comprehensive benchmark for evaluating fine-grained spoken language understanding and reasoning in multimodal models.
It systematically captures the variance of real-world linguistic phenomena in daily speech through 47 sub-tasks, including phonetics, prosody, rhetoric… See the full description on the dataset page: https://huggingface.co/datasets/ddwang2000/MMSU.MMSI-Bench
MMSI-Bench
This repo contains evaluation code for the paper "MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence"
🌐 Homepage | 🤗 Dataset | 📑 Paper | 💻 Code | 📖 arXiv
🔔News
🔥[2025-10-23]: We added the normalized human response time for each MMSI-Bench sample and its difficulty level to our dataset on Hugging Face.
🔥[2025-06-18]: MMSI-Bench has been supported in the LMMs-Eval repository.
✨[2025-06-11]: MMSI-Bench was used for evaluation in the… See the full description on the dataset page: https://huggingface.co/datasets/RunsenXu/MMSI-Bench.MMSearch-Plus
MMSearch-Plus✨: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents
Official repository for the paper "MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents".
🌟 For more details, please refer to the project page with examples: https://mmsearch-plus.github.io/.
[🌐 Webpage] [📖 Paper] [🤗 Huggingface Dataset] [🏆 Leaderboard]
💥 News
[2025.09.26] 🔥 We update the arXiv paperand release all MMSearch-Plus data samples in… See the full description on the dataset page: https://huggingface.co/datasets/Cie1/MMSearch-Plus.MM-SafetyBench-plus-plus
MM-SafetyBench++
Project Page | Paper | Code
MM-SafetyBench++ is a benchmark designed for evaluating contextual safety in Multi-Modal Large Language Models (MLLMs). It challenges models to distinguish subtle contextual differences between scenarios that may appear visually or textually similar but diverge significantly in safety intent.
Dataset Summary
For each unsafe image-text pair, the benchmark includes a corresponding safe counterpart created through minimal… See the full description on the dataset page: https://huggingface.co/datasets/EchoSafe-MLLM/MM-SafetyBench-plus-plus.MMSearch
MMSearch 🔥: Benchmarking the Potential of Large Models as Multi-modal Search Engines
Official repository for the paper "MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines".
🌟 For more details, please refer to the project page with dataset exploration and visualization tools: https://mmsearch.github.io/.
[🌐 Webpage] [📖 Paper] [🤗 Huggingface Dataset] [🏆 Leaderboard] [🔍 Visualization]
💥 News
[2024.09.25] 🌟 The evaluation code now… See the full description on the dataset page: https://huggingface.co/datasets/CaraJ/MMSearch.Omni_MMSI
Omni-MMSI: Toward Identity-attributed Social Interaction Understanding
Webpage |
arXiv |
YouTube |
Github
Xinpeng Li,
Bolin Lai,
Hardy Chen,
Shijian Deng,
Cihang Xie,
Yuyin Zhou,
James Matthew Rehg,
Yapeng Tian
Introduction
Omni-MMSI is an annotation dataset for identity-attributed social interaction understanding in multi-party social deduction game videos. It provides instruction-formatted metadata for clips, including labels, oracle cues, extracted cues… See the full description on the dataset page: https://huggingface.co/datasets/Xinpeng-Li/Omni_MMSI.Online_MMSI
Online-MMSI: Towards Online Multimodal Social Interaction Understanding
Webpage |
Paper |
YouTube |
Github
Xinpeng Li,
Shijian Deng,
Bolin Lai,
Weiguo Pian,
James Matthew Rehg,
Yapeng Tian
Online-MMSI is a metadata release for online multimodal social interaction understanding in multi-party social deduction game videos. It provides instruction pairs for identifying player references and predicting upcoming speaker turns and dialogue.
Tasks
MPP: mentioned… See the full description on the dataset page: https://huggingface.co/datasets/Xinpeng-Li/Online_MMSI.tau2-mms-teacher-traces
Tau2 Teacher Traces Dataset
Dataset Description
This dataset contains teacher reasoning traces for solving MMS (Multimedia Messaging Service) issues in the τ²-bench (Tau2-bench) framework. Each example includes a teacher model's thinking process and structured teaching guidance for resolving customer service tickets.
Dataset Summary
Domain: Telecom customer service
Task: MMS troubleshooting
Size: 49 examples
Format: JSONL
Dataset Structure… See the full description on the dataset page: https://huggingface.co/datasets/Arc-Intelligence/tau2-mms-teacher-traces.MMSI-Bench-test
MMSI-Bench
This repo contains evaluation code for the paper "[MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence]"
🌐 Homepage | 🤗 Dataset | 📑 Paper | 💻 Code | 📖 arXiv
🔔News
🔥[2025-05-30]: We released the ArXiv paper.
Load Dataset
from datasets import load_dataset
mmsi_bench = load_dataset("RunsenXu/MMSI-Bench")
print(dataset)
Evaluation
Please refer to the evaluation guidelines of VLMEvalKit
🏆… See the full description on the dataset page: https://huggingface.co/datasets/sihany/MMSI-Bench-test.mmscifact-dev
MMSciFact
Multimodal scientific fact-checking benchmark: model-generated
question-answer pairs over scientific PDFs, with every answer sentence
human-annotated for role (Observation/Interpretation), dependency
structure (depends_on), and grounding label
(supported/contradiction/NEI/not_a_claim) against the source document.
40 QA pairs across 6 papers, 440 annotated sentences.
Every included QA pair has at least one contradiction/nei sentence
(see… See the full description on the dataset page: https://huggingface.co/datasets/alecocc/mmscifact-dev.MMStar_tr
MMStar TR
This dataset is translation of Lin-Chen/MMStar dataset. The dataset was translated using DeepL. Each sample has been manually checked and fixed.
Some questions were altered to make them understandble and answerable. Some of the alterations in the questions:
Removing choices that are nearly identical
Changing a choice when two choices are correct
Some questions were written as if there were multiple images. They were changed similar to NCSOFT/K-MMStar
Some of the math… See the full description on the dataset page: https://huggingface.co/datasets/kesimeg/MMStar_tr.MMStar
MMStar (Are We on the Right Way for Evaluating Large Vision-Language Models?)
🌐 Homepage | 🤗 Dataset | 🤗 Paper | 📖 arXiv | GitHub
Dataset Details
As shown in the figure below, existing benchmarks lack consideration of the vision dependency of evaluation samples and potential data leakage from LLMs' and LVLMs' training data.
Therefore, we introduce MMStar: an elite vision-indispensible multi-modal benchmark, aiming to ensure each curated sample exhibits… See the full description on the dataset page: https://huggingface.co/datasets/xiaohei1/MMStar.MMSciFact
MMSciFact
Multimodal scientific fact-checking benchmark: model-generated
question-answer pairs over scientific PDFs, with every answer sentence
human-annotated for role, dependency structure, and grounding label
against the source document.
257 QA pairs across 45 papers, 2737 annotated sentences.
Files
mmscifact.jsonl — one row per QA pair.
pdfs.zip — one PDF per paper (pdfs/<paper_id>.pdf), referenced by
paper_pdf_path on every row.
More documentation (row… See the full description on the dataset page: https://huggingface.co/datasets/alecocc/MMSciFact.MMstar_Kazakh_Russian
Dataset Summary
These are the machine-translated Kazakh and Russian versions of the MMStar dataset.
MMStar is an elite vision-language benchmark consisting of high-quality, non-redundant multimodal samples designed to evaluate 6 core capabilities and 18 detailed dimensions of Large Multimodal Models (LMMs). Kazakh and Russian versions serve as a benchmark for evaluating how well models perform advanced visual recognition and multi-step reasoning while avoiding "data leakage"… See the full description on the dataset page: https://huggingface.co/datasets/issai/MMstar_Kazakh_Russian.MMSQL
MMSQL
This repository contains the DATASET:MMSQL in paper "Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types" [Arxiv] [IJCNN2025]. The repository is structured to ensure the reproducibility of the experiments and includes scripts, notebooks, test suits, and data outputs. You can get an overview of this project and the paper through the page.
Construction
We used 🐦QDA-SQL. to generate a training set with 4 question types:… See the full description on the dataset page: https://huggingface.co/datasets/GuoZiming/MMSQL.MMStar-PT
MMStar-PT
European Portuguese (pt-PT) machine translation of MMStar, a curated multimodal benchmark of vision-indispensable, balanced challenge samples.
Translated from the original English val split using gemini-3.1-pro.
Original Dataset: https://huggingface.co/datasets/Lin-Chen/MMStar
Note: This dataset is machine translated and may contain translation errors or artifacts.
This dataset is provided as part of the AMALIA project and is included in amalia-vl-eval, a… See the full description on the dataset page: https://huggingface.co/datasets/amalia-llm/MMStar-PT.MMStar_with_difficulty_level
MMStar with difficulty level tags
This dataset extends the 🤗 MMStar benchmark by introducing two additional tags: passrate_for_qwen2.5_vl_7b and difficulty_level_for_qwen2.5_vl_7b. Further details are available in our paper The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs.
🚀 Data Usage
from datasets import load_dataset
dataset = load_dataset("JierunChen/MMStar_with_difficulty_level")
print(dataset)… See the full description on the dataset page: https://huggingface.co/datasets/JierunChen/MMStar_with_difficulty_level.MMS-e
MMS-e: Benchmarking the Resilience of Large Multimodal Models to Visual Scrambling
Benchmark Examples
Patchwise Question Answering: Divide the images into 2x2, 4x4, and 8x8 patches, then shuffle all the patches, and measure the ability of LMMs to answer questions about these images.
Fixed Patch Question Answering: Divide the image into 4x4 patches, randomly fix some of the patches, and let LMMs answer questions based on the image.
Reconstruction task: Let LMMs… See the full description on the dataset page: https://huggingface.co/datasets/mms-e/MMS-e.