Datasetpublic

AI4Math/MathVerse

Dataset Card for MathVerse Dataset Description Paper Information Dataset Examples Leaderboard Citation Dataset Description The capabilities of Multi-modal Large Language Models (MLLMs) in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams.… See the full description on the dataset page: https://huggingface.co/datasets/AI4Math/MathVerse.

sourceHugging Facemitupdated 1y agoView on Hugging Face

72likes3.2kdownloads

Dataset Card

Dataset Card for MathVerse

Dataset Description

The capabilities of Multi-modal Large Language Models (MLLMs) in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams.

To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality, contributing to 15K test samples in total. This approach allows MathVerse to comprehensively assess *whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning.*

<img src="https://raw.githubusercontent.com/ZrrSkywalker/MathVerse/main/figs/fig2.png" width="90%"> Six different versions of each problem in MathVerse transformed by expert annotators.

In addition, we propose a Chain-of-Thought (CoT) Evaluation strategy for a fine-grained assessment of the output answers. Rather than naively judging True or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and then score each step with detailed error analysis, which can reveal the intermediate CoT reasoning quality by MLLMs.

<img src="https://raw.githubusercontent.com/ZrrSkywalker/MathVerse/main/figs/fig3.png" width="90%"> The two phases of the CoT evaluation strategy.

Paper Information

—Code: https://github.com/ZrrSkywalker/MathVerse
—Project: https://mathverse-cuhk.github.io/
—Visualization: https://mathverse-cuhk.github.io/#visualization
—Leaderboard: https://mathverse-cuhk.github.io/#leaderboard
—Paper: https://arxiv.org/abs/2403.14624

Dataset Examples

🖱 Click to expand the examples for six problems versions within three subjects</summary>

<details> <summary>🔍 Plane Geometry</summary>

<details> <summary>🔍 Solid Geometry</summary>

<details> <summary>🔍 Functions</summary>

Leaderboard

Contributing to the Leaderboard

🚨 The Leaderboard is continuously being updated.

The evaluation instructions and tools will be released soon. For now, please send your results on the *testmini* set to this email: 1700012927@pku.edu.cn. Please refer to the following template to prepare your result json file.

—[outputtestminitemplate.json]()

License

This project is released under the MIT license.

Citation

If you find MathVerse useful for your research and applications, please kindly cite using this BibTeX:

latex

@inproceedings{zhang2024mathverse,
  title={MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?},
  author={Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, Hongsheng Li},
  booktitle={arXiv},
  year={2024}
}