CoolFace
Datasetpublic

grow-ai-like-a-child/mechanical-reasoning

Mechanical Reasoning Mechanical Reasoning is a multimodal benchmark designed to evaluate mechanical and physical reasoning in large vision-language models (VLMs). This dataset is introduced in the paper: Probing Mechanical Reasoning in Large Vision Language ModelsarXiv:2410.00318 It is part of the Grow AI Like a Child benchmark initiative, which explores structured, curriculum-aligned evaluation for embodied intelligence. 🧠 Dataset Overview This dataset… See the full description on the dataset page: https://huggingface.co/datasets/grow-ai-like-a-child/mechanical-reasoning.

sourceHugging Facecc-by-4.0updated 1y agoView on Hugging Face
2likes83downloads
Dataset Card

Mechanical Reasoning

Mechanical Reasoning is a multimodal benchmark designed to evaluate mechanical and physical reasoning in large vision-language models (VLMs). This dataset is introduced in the paper:

Probing Mechanical Reasoning in Large Vision Language Models arXiv:2410.00318

It is part of the Grow AI Like a Child benchmark initiative, which explores structured, curriculum-aligned evaluation for embodied intelligence.

🧠 Dataset Overview

This dataset focuses on mechanical reasoning grounded in either static images or short video clips. Each item presents a physical scenario and poses a question requiring spatial, causal, or commonsense understanding.

The dataset contains:

  • —148 samples
  • —Two modalities: image or video
  • —Two question formats: multiple-choice (MC) or true/false (TF)

🧩 Reasoning Types

The questions span a variety of physical reasoning challenges:

  • —Gear and pulley systems
  • —Fluid and buoyancy dynamics
  • —Torque and balance
  • —Conveyor motion
  • —Mechanical advantage
  • —Temporal prediction and causality

📁 Dataset Format

Each sample includes:

FieldDescription
indexUnique ID (e.g., d0001)
media_typeEither image or video
mediaLocal file name (e.g., d0001.png, d0131.mov)
question_typeEither MC or TF
sec_labelInteger from 1 to 6 (see section mapping below)
questionNatural language question with choices embedded (if MC)
correct_answerThe ground-truth answer (e.g., A, B, Yes, No)

🔢 sec_label Categories

LabelCategory
1System Ability
2Pulley Systems
3Gears & Rotations
4Leverage Principle
5Inertia and Motion
6Fluid Mechanics

📂 Folder Structure

data/
├── data.csv
├── images/
│   ├── *.png
│   └── metadata.jsonl
├── videos/
│   ├── *.mov / *.mp4
│   └── metadata.jsonl
  • —The metadata.jsonl files store structured sample entries.
  • —For video samples, file_name in metadata is mapped to .png format to avoid embedding issues on Dataset Viewer.

💡 Example

json
{
  "file_name": "d0001.png",
  "media_type": "image",
  "question_type": "MC",
  "sec_label": 6,
  "question": "In which direction is the leaking cylinder moving in the water in the picture? A. up; B. down",
  "correct_answer": "B"
}

📚 Citation

If you use this dataset, please cite:

bibtex
@misc{sun2025probingmechanicalreasoninglarge,
      title={Probing Mechanical Reasoning in Large Vision Language Models}, 
      author={Haoran Sun and Qingying Gao and Haiyun Lyu and Dezhi Luo and Yijiang Li and Hokin Deng},
      year={2025},
      eprint={2410.00318},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2410.00318}, 
}

🤝 Acknowledgments

This dataset is developed by the Grow AI Like a Child community.