cjerzak/MultimodalMathBenchmarks

MultimodalMathBenchmarks

This repository contains the datasets for the paper Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs (ACL Findings 2026).

It covers the public benchmark datasets and their modality assets (text, images, and audio) used to evaluate the arithmetic capabilities of multimodal LLMs.

Canonical Upload Manifest

Dataset Summary

The release contains three benchmark families:

1. 1.SharedMultimodalGrid.csv
2. 2.10,000 shared multiplication problems paired across text, image, and audio.
3. 3.Split counts: train=7026, val=1416, test=1558.
4. 4.Each row uses an mm_XXXXX ID that maps directly to:
5. 5.TextFiles/mm_XXXXX.txt
6. 6.Images/mm_XXXXX.png
7. 7.AudioFiles/mm_XXXXX.mp3

1. 1.HDSv2.csv
2. 2.1,000 heuristic-disagreement problems for fingerprinting and probe-style evaluation.
3. 3.Split counts: train=701, val=155, test=144.
4. 4.HDSv2Images/ contains exactly the 144 test-split item IDs from HDSv2.csv.

1. 1.Trapsv2.csv
2. 2.30 adversarial trap problems designed to target heuristic-specific failures.
3. 3.No split column; all rows are held-out trap items.
4. 4.Trapsv2Images/ contains one PNG for every row in Trapsv2.csv.

Dataset Structure

Shared Multimodal Grid

Canonical table: SharedMultimodalGrid.csv

Columns: id, a, b, a_times_b, template_a, template_b, digits_a, digits_b, nonzero_a, nonzero_b, digit_total, nonzero_total, complexity_c, stratum_id, split

Notes:

• —id is the stable benchmark ID, e.g. mm_00001.
• —a, b, and a_times_b are the multiplication operands and exact product.
• —digit_total, nonzero_total, and complexity_c support difficulty analyses.
• —split is the deterministic train/val/test assignment used by the pipeline.

HDSv2

Canonical table: HDSv2.csv

Columns: id, a, b, product, design_family, canonical_target_heuristic, canonical_target_margin, ot_cost, dd_cost, rc_cost, heuristic_definition_version, target_heuristic, ot_score, dd_score, rc_score, category, notes, digit_total, nonzero_total, complexity_c, split

Trapsv2

Canonical table: Trapsv2.csv

Columns: id, a, b, product, trap_type, design_family, canonical_target_heuristic, heuristic_definition_version, target_heuristic, expected_error_type, notes, digit_total, nonzero_total, complexity_c

Modality and Evaluation Notes

• —The paired multimodal benchmark includes text, image, and audio assets.
• —HDSv2Images/ is intentionally a test-only image release for probe evaluation.
• —Trapsv2Images/ covers the full trap set.
• —Studies on this dataset show that accuracy falls sharply as arithmetic load $C$ (the product of total and non-zero digit count) grows.

Citation

@inproceedings{balter2026multiplication,
  title={{Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs}},
  author={Balter, Samuel Gideon and Jerzak, Ethan and Jerzak, Connor Thomas},
  booktitle={Findings of the Association for Computational Linguistics: ACL 2026},
  year={2026}
}