neulab/VisualPuzzles

VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge

🏠 Homepage | 📊 VisualPuzzles | 💻 Github | 📄 Arxiv | 📕 PDF | 🖥️ Zeno Model Output

Overview

VisualPuzzles is a multimodal benchmark specifically designed to evaluate reasoning abilities in large models while deliberately minimizing reliance on domain-specific knowledge.

Key features:

• —1168 diverse puzzles
• —5 reasoning categories: Algorithmic, Analogical, Deductive, Inductive, Spatial
• —Difficulty labels: Easy, Medium, Hard
• —Less knowledge-intensive than existing benchmarks (e.g., MMMU)
• —More reasoning-complex than existing benchmarks (e.g., MMMU)

Key Findings

• —All models perform worse than humans; most can't surpass even 5th-percentile human performance.
• —Strong performance on knowledge-heavy benchmarks does not transfer well.
• —Larger models and structured "thinking modes" don't guarantee better results.
• —Scaling model size does not ensure stronger reasoning

Usage

To load this dataset via Hugging Face’s datasets library:

from datasets import load_dataset

dataset = load_dataset("neulab/VisualPuzzles")
data = dataset["train"]

sample = data[0]
print("ID:", sample["id"])
print("Category:", sample["category"])
print("Question:", sample["question"])
print("Options:", sample["options"])
print("Answer:", sample["answer"])

Citation

If you use or reference this dataset in your work, please cite:

@article{song2025visualpuzzles,
  title         = {VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge},
  author        = {Song, Yueqi and Ou, Tianyue and Kong, Yibo and Li, Zecheng and Neubig, Graham and Yue, Xiang},
  year          = {2025},
  journal       = {arXiv preprint arXiv:2504.10342},
  url           = {https://arxiv.org/abs/2504.10342}
}