disi-unibo-nlp/MathGames

🧮 MathGames

MathGames is a novel benchmark of 2,183 high-quality mathematical problems—both text-only and multimodal—designed to evaluate Large Language Models (LLMs) on open-ended mathematical and logical reasoning tasks. It accompanies our EMNLP 2025 Main Track paper:

📄 Can Large Language Models Win the International Mathematical Games?

🌍 Overview

MathGames consists of carefully curated problems sourced from the **International Mathematical and Logical Games Championships**, an annual international competition promoting creative and logical problem solving.

• —Total problems: 2,183

• —🧠 1,389 textual problems
• —🖼️ 794 multimodal problems (text + image)
• —Format: Open-ended (no multiple-choice answers)
• —Age range: 8 years old to 25+
• —Skill taxonomy: Problems are labeled according to the underlying mathematical and logical skills required.
• —Validation: All problem-solution pairs were manually verified by expert annotators.

This benchmark aims to provide a structured and interpretable evaluation of the reasoning abilities of LLMs across both mathematical domains and cognitive skill levels.

All materials in MathGames are sourced from the official online archive of the PRISTEM Center, Bocconi University, which retains full rights to the original content.

We obtained explicit authorization from PRISTEM to use, translate, and distribute the problems and solutions in English, in full compliance with applicable copyright and licensing regulations. This dataset is provided exclusively for research and evaluation purposes, with the primary goal of advancing studies on mathematical and logical reasoning in LLMs. Use of the data for model training, commercial redistribution, or derivative works is not permitted without prior written authorization from PRISTEM, Bocconi University.

🧩 Structure of the Competition

• —The dataset mirrors the official structure of the International Mathematical and Logical Games Championships, which organizes participants by educational level into seven categories.
• —Each problem in MathGames is tagged with its original competition category and phase.
• —MathGames introduces also a coarse-grained taxonomy of mathematical and logical skills, capturing the primary reasoning abilities required to solve each problem:

🎓 Categories

These categories ensure coverage from elementary-level reasoning to advanced logical and mathematical problem-solving.

🧠 Skill Taxonomy

• —Arithmetic
• —Logic
• —Pattern Recognition
• —Geometry
• —Combinatorics
• —Algebra

This taxonomy enables skill-based evaluation of LLMs’ mathematical reasoning, highlighting strengths and weaknesses across distinct domains of cognitive ability.

🏆 Competition Phases

Each championship consists of four competitive phases, progressively filtering participants based on performance:

• —Quarterfinals
• —Semifinals
• —National Finals
• —International Finals

In addition, several training and preparatory events are included, such as:

• —Autumn Games
• —Rosi’s Games
• —Team Games

These events promote engagement and serve as practice opportunities for developing reasoning skills.

📂 Dataset Structure

Each entry in MathGames corresponds to a single problem–solution pair, optionally accompanied by an image. All problems are annotated with their competition metadata, modality, and subject category.

🧩 Load Data

You can easily load the MathGames benchmark using the 🤗 `datasets` library:

from datasets import load_dataset
# Load the multimodal or textual subset
ds_multimodal = load_dataset("disi-unibo-nlp/MathGames", split="multimodal")  # or split="textual"
# Inspect the first entry
print(ds_multimodal[0])

This is an example of a multimodal entry:

{
  'id': '1952',
  'year': '1996',
  'type': 'semifinal',
  'multimodal': 'yes',
  'category': 'C1 C2 L1 L2 GP',
  'subject': 'Logic',
  'answer': 'The sum of four cells is always 24. The grid is:\nRow 1: 1, 6, 3\nRow 2: 9, 8, 7\nRow 3: 2, 5, 4',
  'question': 'Fill the nine cells of the square above with the numbers from 1 to 9 (1 and 7 have already been placed) such that the sum of the numbers written in each 4-cell square (like those highlighted in the figure) is always the same (figure).',
  'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=132x75>
}

📑 Citation

If you use MathGames in your research, please cite:

@inproceedings{cocchieri-etal-2025-large,
    title = "Can Large Language Models Win the International Mathematical Games?",
    author = "Cocchieri, Alessio  and
      Ragazzi, Luca  and
      Tagliavini, Giuseppe  and
      Tordi, Lorenzo  and
      Carbonaro, Antonella  and
      Moro, Gianluca",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.488/",
    doi = "10.18653/v1/2025.emnlp-main.488",
    pages = "9656--9682",
    ISBN = "979-8-89176-332-6",
    abstract = "Recent advances in large language models (LLMs) have demonstrated strong mathematical reasoning abilities, even in visual contexts, with some models surpassing human performance on existing benchmarks. However, these benchmarks lack structured age categorization, clearly defined skill requirements, and{---}crucially{---}were not designed to assess human performance in international competitions. To address these limitations, we introduce MathGames, a new benchmark of 2,183 high-quality mathematical problems (both text-only and multimodal) in an open-ended format, sourced from an international mathematical games championships. Spanning seven age groups and a skill-based taxonomy, MathGames enables a structured evaluation of LLMs' mathematical and logical reasoning abilities. Our experiments reveal a substantial gap between state-of-the-art LLMs and human participants{---}even 11-year-olds consistently outperform some of the strongest models{---}highlighting the need for advancements. Further, our detailed error analysis offers valuable insights to guide future research. The data is publicly available at https://disi-unibo-nlp.github.io/math-games."
}

📬 Contact

For questions, feedback, or collaboration inquiries, please contact:

• —Alessio Cocchieri [[a.cocchieri@unibo.it](mailto:a.cocchieri@unibo.it)]
• —Luca Ragazzi [[l.ragazzi@unibo.it](mailto:l.ragazzi@unibo.it)]