datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
MMStar
MMStar (Are We on the Right Way for Evaluating Large Vision-Language Models?)
🌐 Homepage | 🤗 Dataset | 🤗 Paper | 📖 arXiv | GitHub
Dataset Details
As shown in the figure below, existing benchmarks lack consideration of the vision dependency of evaluation samples and potential data leakage from LLMs' and LVLMs' training data.
Therefore, we introduce MMStar: an elite vision-indispensible multi-modal benchmark, aiming to ensure each curated sample exhibits… See the full description on the dataset page: https://huggingface.co/datasets/Lin-Chen/MMStar.MM-SpuBench
MM-SpuBench Datacard
Basic Information
Title: The Multimodal Spurious Benchmark (MM-SpuBench)
Description: MM-SpuBench is a comprehensive benchmark designed to evaluate the robustness of MLLMs to spurious biases. This benchmark systematically assesses how well these models distinguish between core and spurious features, providing a detailed framework for understanding and quantifying spurious biases.
Data Structure:
├── data/images
│ ├── 000000.jpg
│ ├── 000001.jpg
│… See the full description on the dataset page: https://huggingface.co/datasets/mmbench/MM-SpuBench.MMSI-Bench
MMSI-Bench
This repo contains evaluation code for the paper "MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence"
🌐 Homepage | 🤗 Dataset | 📑 Paper | 💻 Code | 📖 arXiv
🔔News
🔥[2025-10-23]: We added the normalized human response time for each MMSI-Bench sample and its difficulty level to our dataset on Hugging Face.
🔥[2025-06-18]: MMSI-Bench has been supported in the LMMs-Eval repository.
✨[2025-06-11]: MMSI-Bench was used for evaluation in the… See the full description on the dataset page: https://huggingface.co/datasets/RunsenXu/MMSI-Bench.MMSearch-Plus
MMSearch-Plus✨: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents
Official repository for the paper "MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents".
🌟 For more details, please refer to the project page with examples: https://mmsearch-plus.github.io/.
[🌐 Webpage] [📖 Paper] [🤗 Huggingface Dataset] [🏆 Leaderboard]
💥 News
[2025.09.26] 🔥 We update the arXiv paperand release all MMSearch-Plus data samples in… See the full description on the dataset page: https://huggingface.co/datasets/Cie1/MMSearch-Plus.MM-SafetyBench-plus-plus
MM-SafetyBench++
Project Page | Paper | Code
MM-SafetyBench++ is a benchmark designed for evaluating contextual safety in Multi-Modal Large Language Models (MLLMs). It challenges models to distinguish subtle contextual differences between scenarios that may appear visually or textually similar but diverge significantly in safety intent.
Dataset Summary
For each unsafe image-text pair, the benchmark includes a corresponding safe counterpart created through minimal… See the full description on the dataset page: https://huggingface.co/datasets/EchoSafe-MLLM/MM-SafetyBench-plus-plus.MMSearch
MMSearch 🔥: Benchmarking the Potential of Large Models as Multi-modal Search Engines
Official repository for the paper "MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines".
🌟 For more details, please refer to the project page with dataset exploration and visualization tools: https://mmsearch.github.io/.
[🌐 Webpage] [📖 Paper] [🤗 Huggingface Dataset] [🏆 Leaderboard] [🔍 Visualization]
💥 News
[2024.09.25] 🌟 The evaluation code now… See the full description on the dataset page: https://huggingface.co/datasets/CaraJ/MMSearch.MMS-e
MMS-e: Benchmarking the Resilience of Large Multimodal Models to Visual Scrambling
Benchmark Examples
Patchwise Question Answering: Divide the images into 2x2, 4x4, and 8x8 patches, then shuffle all the patches, and measure the ability of LMMs to answer questions about these images.
Reconstruction task: Let LMMs reconstruct the order of shuffled patches based on the image' s caption, and let LMMs reconstruct the shuffled caption based on the image.
Fixed Patch… See the full description on the dataset page: https://huggingface.co/datasets/jyjyjyjy/MMS-e.MMSI-Bench-test
MMSI-Bench
This repo contains evaluation code for the paper "[MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence]"
🌐 Homepage | 🤗 Dataset | 📑 Paper | 💻 Code | 📖 arXiv
🔔News
🔥[2025-05-30]: We released the ArXiv paper.
Load Dataset
from datasets import load_dataset
mmsi_bench = load_dataset("RunsenXu/MMSI-Bench")
print(dataset)
Evaluation
Please refer to the evaluation guidelines of VLMEvalKit
🏆… See the full description on the dataset page: https://huggingface.co/datasets/sihany/MMSI-Bench-test.MMStar_tr
MMStar TR
This dataset is translation of Lin-Chen/MMStar dataset. The dataset was translated using DeepL. Each sample has been manually checked and fixed.
Some questions were altered to make them understandble and answerable. Some of the alterations in the questions:
Removing choices that are nearly identical
Changing a choice when two choices are correct
Some questions were written as if there were multiple images. They were changed similar to NCSOFT/K-MMStar
Some of the math… See the full description on the dataset page: https://huggingface.co/datasets/kesimeg/MMStar_tr.MMStar
MMStar (Are We on the Right Way for Evaluating Large Vision-Language Models?)
🌐 Homepage | 🤗 Dataset | 🤗 Paper | 📖 arXiv | GitHub
Dataset Details
As shown in the figure below, existing benchmarks lack consideration of the vision dependency of evaluation samples and potential data leakage from LLMs' and LVLMs' training data.
Therefore, we introduce MMStar: an elite vision-indispensible multi-modal benchmark, aiming to ensure each curated sample exhibits… See the full description on the dataset page: https://huggingface.co/datasets/xiaohei1/MMStar.MMstar_Kazakh_Russian
Dataset Summary
These are the machine-translated Kazakh and Russian versions of the MMStar dataset.
MMStar is an elite vision-language benchmark consisting of high-quality, non-redundant multimodal samples designed to evaluate 6 core capabilities and 18 detailed dimensions of Large Multimodal Models (LMMs). Kazakh and Russian versions serve as a benchmark for evaluating how well models perform advanced visual recognition and multi-step reasoning while avoiding "data leakage"… See the full description on the dataset page: https://huggingface.co/datasets/issai/MMstar_Kazakh_Russian.MMStar-PT
MMStar-PT
European Portuguese (pt-PT) machine translation of MMStar, a curated multimodal benchmark of vision-indispensable, balanced challenge samples.
Translated from the original English val split using gemini-3.1-pro.
Original Dataset: https://huggingface.co/datasets/Lin-Chen/MMStar
Note: This dataset is machine translated and may contain translation errors or artifacts.
This dataset is provided as part of the AMALIA project and is included in amalia-vl-eval, a… See the full description on the dataset page: https://huggingface.co/datasets/amalia-llm/MMStar-PT.MMStar_with_difficulty_level
MMStar with difficulty level tags
This dataset extends the 🤗 MMStar benchmark by introducing two additional tags: passrate_for_qwen2.5_vl_7b and difficulty_level_for_qwen2.5_vl_7b. Further details are available in our paper The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs.
🚀 Data Usage
from datasets import load_dataset
dataset = load_dataset("JierunChen/MMStar_with_difficulty_level")
print(dataset)… See the full description on the dataset page: https://huggingface.co/datasets/JierunChen/MMStar_with_difficulty_level.MMS-e
MMS-e: Benchmarking the Resilience of Large Multimodal Models to Visual Scrambling
Benchmark Examples
Patchwise Question Answering: Divide the images into 2x2, 4x4, and 8x8 patches, then shuffle all the patches, and measure the ability of LMMs to answer questions about these images.
Fixed Patch Question Answering: Divide the image into 4x4 patches, randomly fix some of the patches, and let LMMs answer questions based on the image.
Reconstruction task: Let LMMs… See the full description on the dataset page: https://huggingface.co/datasets/mms-e/MMS-e.