AI4Math/MathVerse

1---2task_categories:3- multiple-choice4- question-answering5- visual-question-answering6language:7- en8size_categories:9- 1K<n<10K10configs:11- config_name: testmini12  data_files:13  - split: testmini14    path: testmini.parquet15- config_name: testmini_text_only16  data_files:17  - split: testmini_text_only18    path: testmini_text_only.parquet19dataset_info:20- config_name: testmini21  features:22  - name: sample_index23    dtype: string24  - name: problem_index25    dtype: string26  - name: problem_version27    dtype: string28  - name: question29    dtype: string30  - name: image31    dtype: image32  - name: answer33    dtype: string34  - name: question_type35    dtype: string36  - name: metadata37    struct:38    - name: split39      dtype: string40    - name: source41      dtype: string42    - name: subject43      dtype: string44    - name: subfield45      dtype: string46  - name: query_wo47    dtype: string48  - name: query_cot49    dtype: string50  - name: question_for_eval51    dtype: string52  splits:53  - name: testmini54    num_bytes: 16678996355    num_examples: 394056- config_name: testmini_text_only57  features:58  - name: sample_index59    dtype: string60  - name: problem_index61    dtype: string62  - name: problem_version63    dtype: string64  - name: question65    dtype: string66  - name: image67    dtype: string68  - name: answer69    dtype: string70  - name: question_type71    dtype: string72  - name: metadata73    struct:74    - name: split75      dtype: string76    - name: source77      dtype: string78    - name: subject79      dtype: string80    - name: subfield81      dtype: string82  - name: query_wo83    dtype: string84  - name: query_cot85    dtype: string86  - name: question_for_eval87    dtype: string88  splits:89  - name: testmini_text_only90    num_bytes: 25095991    num_examples: 78892license: mit93---94# Dataset Card for MathVerse95 96- [Dataset Description](https://huggingface.co/datasets/AI4Math/MathVerse/blob/main/README.md#dataset-description)97- [Paper Information](https://huggingface.co/datasets/AI4Math/MathVerse/blob/main/README.md#paper-information)98- [Dataset Examples](https://huggingface.co/datasets/AI4Math/MathVerse/blob/main/README.md#dataset-examples)99- [Leaderboard](https://huggingface.co/datasets/AI4Math/MathVerse/blob/main/README.md#leaderboard)100- [Citation](https://huggingface.co/datasets/AI4Math/MathVerse/blob/main/README.md#citation)101 102## Dataset Description103The capabilities of **Multi-modal Large Language Models (MLLMs)** in **visual math problem-solving** remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams.104 105<p align="center">106    <img src="https://raw.githubusercontent.com/ZrrSkywalker/MathVerse/main/figs/fig1.png" width="90%"> <br>107</p>108 109To this end, we introduce **MathVerse**, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into **six distinct versions**, each offering varying degrees of information content in multi-modality, contributing to **15K** test samples in total. This approach allows MathVerse to comprehensively assess ***whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning.*** 110 111<p align="center">112    <img src="https://raw.githubusercontent.com/ZrrSkywalker/MathVerse/main/figs/fig2.png" width="90%"> <br>113    Six different versions of each problem in <b>MathVerse</b> transformed by expert annotators.114</p>115 116In addition, we propose a **Chain-of-Thought (CoT) Evaluation strategy** for a fine-grained assessment of the output answers. Rather than naively judging True or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and then score each step with detailed error analysis, which can reveal the intermediate CoT reasoning quality by MLLMs.117 118<p align="center">119    <img src="https://raw.githubusercontent.com/ZrrSkywalker/MathVerse/main/figs/fig3.png" width="90%"> <br>120    The two phases of the CoT evaluation strategy.121</p>122 123## Paper Information124- Code: https://github.com/ZrrSkywalker/MathVerse125- Project: https://mathverse-cuhk.github.io/126- Visualization: https://mathverse-cuhk.github.io/#visualization127- Leaderboard: https://mathverse-cuhk.github.io/#leaderboard128- Paper: https://arxiv.org/abs/2403.14624129 130## Dataset Examples131🖱 Click to expand the examples for six problems versions within three subjects</summary>132 133<details>134<summary>🔍 Plane Geometry</summary>135 136<p align="center">137    <img src="https://raw.githubusercontent.com/ZrrSkywalker/MathVerse/main/figs/ver1.png" width="50%"> <br>138</p>139</details>140 141<details>142<summary>🔍 Solid Geometry</summary>143 144<p align="center">145    <img src="https://raw.githubusercontent.com/ZrrSkywalker/MathVerse/main/figs/ver2.png" width="50%"> <br>146</p>147</details>148 149<details>150<summary>🔍 Functions</summary>151 152<p align="center">153    <img src="https://raw.githubusercontent.com/ZrrSkywalker/MathVerse/main/figs/ver3.png" width="50%"> <br>154</p>155</details>156 157## Leaderboard158### Contributing to the Leaderboard159 160🚨 The [Leaderboard](https://mathverse-cuhk.github.io/#leaderboard) is continuously being updated. 161 162The evaluation instructions and tools will be released soon. For now, please send your results on the ***testmini*** set to this email: 1700012927@pku.edu.cn. Please refer to the following template to prepare your result json file.163 164- [output_testmini_template.json]()165 166## License167This project is released under the MIT license.168 169## Citation170 171If you find **MathVerse** useful for your research and applications, please kindly cite using this BibTeX:172 173```latex174@inproceedings{zhang2024mathverse,175  title={MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?},176  author={Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, Hongsheng Li},177  booktitle={arXiv},178  year={2024}179}180```