lmms-lab-eval/MMVP

MMVP (Multimodal Visual Patterns) Benchmark

This is a corrected version of the MMVP benchmark, re-hosted by lmms-lab-eval for use with lmms-eval.

Why this copy?

The original MMVP/MMVP dataset was uploaded in imagefolder format, which only exposes the image column. The text annotations (Question, Options, Correct Answer, Index) from the accompanying Questions.csv were not loaded into the dataset, making it unusable for evaluation.

This version reconstructs the complete dataset by merging the images with Questions.csv from the original repository.

Ground Truth Corrections

Based on verification in lmms-eval issue #1018 and the original MMVP issue #30, we found that two pairs of samples had their answers swapped. The corrections are applied directly in this dataset:

Dataset Structure

• —300 samples organized as 150 pairs
• —Each pair has the same question but opposite correct answers
• —Tests 9 visual patterns: orientation, direction, color, counting, etc.

References

• —Paper: Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
• —Original Repository: https://github.com/tsb0601/MMVP
• —Original Dataset: https://huggingface.co/datasets/MMVP/MMVP
• —lmms-eval Task: https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main/lmms_eval/tasks/mmvp

Citation

@inproceedings{tong2024eyes,
  title={Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs},
  author={Tong, Shengbang and Liu, Zhuang and Zhai, Yuexiang and Ma, Yi and LeCun, Yann and Xie, Saining},
  booktitle={CVPR},
  year={2024}
}