danish-foundation-models/multilingual-gsm-symbolic

Multilingual GSM-Symbolic

Multilingual GSM-Symbolic is a benchmark for evaluating arithmetic reasoning in large language models across multiple languages. It extends Apple's GSM-Symbolic approach by providing symbolic templates that generate thousands of structurally equivalent but numerically distinct math problems. Templates and generation are handled by the `multilingual-gsm-symbolic` package.

The dataset lets you test whether a model genuinely understands a problem or merely pattern-matches on the specific numbers it saw during training. Comparing *_original and *_synthetic splits directly measures this gap.

<div style="text-align: center"> <img src="figures/example.png" width="90%" alt="Template illustration showing a symbolic template rendered into English and German examples"> </div> Want to add your own language to the dataset? It only requires validating 100 templates. Read more about how to contribute here.

Dataset Structure

Each language is a separate config (subset). All languages provide test_original and test_synthetic; only English currently also provides train_original and train_synthetic.

Available configs: ara, dan, deu, eng, eng_metric, est, fra, hin, isl, ita, jpn, mar, nld, nob, rus, ukr, urd, and zho.

Fields

Answer format

Answers follow the GSM8K convention — reasoning steps followed by a final numeric answer:

At 3 miles/hour, it will take 42/3=14 hours for the fog to cover the city.
#### 14

Loading the Dataset

With 🤗 datasets

from datasets import load_dataset

# English synthetic test split
eng = load_dataset("{anon}/multilingual-gsm-symbolic", name="eng", split="test_synthetic")

# German original test split
deu = load_dataset("{anon}/multilingual-gsm-symbolic", name="deu", split="test_original")

With inspect-ai

You can evaluate with inspect-ai simply using:

inspect eval hf/{anon}/multilingual-gsm-symbolic/synthetic_eng --model openai/gpt-5.4-nano --reasoning-effort low

This uses the eval.yaml in the repository. You can target any task id (e.g. original_deu, synthetic_isl) or omit the task name to run all languages.

Evaluation Results

As a sanity check for the results we evaluated the model with inspect-ai using openai/gpt-5.4-nano (reasoning effort: low) on Danish and English.

The gap between original and synthetic accuracy reflects performance degradation on novel number combinations — a proxy for how much a model relies on memorisation vs. genuine reasoning.

Performance degradation (original → synthetic)

<div style="text-align: center"> <img src="figures/distribution.png" width="80%" alt="Distribution of synthetic set accuracy"> </div>

Language gap (English vs. Danish, synthetic)

<div style="text-align: center"> <img src="figures/language_gap.png" width="60%" alt="Overlapping accuracy distributions for English and Danish"> </div>

Accuracy by problem difficulty (reasoning steps)

<div style="text-align: center"> <img src="figures/by_steps.png" width="80%" alt="Accuracy as a function of the number of reasoning steps"> </div>

Eval logs are available in the `logs/` folder on this repository. The complete GLM-5.3 saturation results (15 languages, original and synthetic) are in `logs/glm53-saturation/`.

Citation

If you use this dataset, please cite the original Multiligual GSM-Symbolic paper:

in review