Mihara-bot/olmo-igsm-arith

OLMo iGSM-Easy Arithmetic

This repository contains a frozen, evaluation-only release of the synthetic mod-7 arithmetic task called iGSM-Easy Arithmetic in the accompanying OLMo evaluation code. It contains 750 examples: 250 examples at each target depth 2, 3, and 4.

This is an i-GSM-style task variant, not a claim to be an official release of another dataset named iGSM. The olmo-igsm-arith name is used to make the implementation provenance explicit.

Dataset description

Each problem defines symbolic variables using conventional arithmetic operators +, -, *, and /. All calculations are performed modulo 7 and multi-operator expressions are evaluated strictly from left to right. Division uses modular inverses and never has a zero divisor.

The evaluation prompt is built from preamble, equations, and query. A model is scored by comparing the unnormalized continuation log-likelihood of the seven choices 0 through 6. The random baseline is 1/7, or about 14.29%.

The recommended prompt is:

{preamble}

{equations}
Question: {query}?
Answer:

The choices are:

 0,  1,  2,  3,  4,  5,  6

Dataset structure

The repository has one default configuration and one test split.

Load it with:

from datasets import load_dataset

dataset = load_dataset("Mihara-bot/olmo-igsm-arith", split="test")
depth_2 = dataset.filter(lambda row: row["target_depth"] == 2)

Fields

• —id: Stable example identifier.
• —mod: Arithmetic modulus; always 7.
• —ops_mode: Operator mode; always arith.
• —question: Combined question representation produced by the generator.
• —preamble: Operator definitions and evaluation convention.
• —equations: Shuffled variable definitions.
• —query: Variable whose value must be predicted.
• —answer: Integer gold answer in [0, 6]; also the choice index.
• —cot: Generator-produced derivation. It is provided for auditing and is not part of the recommended evaluation prompt.
• —operator_table: Structured operator definitions.
• —target_depth: Requested dependency depth, one of 2, 3, or 4.
• —achieved_depth: Verified dependency depth.
• —num_vars: Number of variables in the problem.
• —num_necessary: Variables required to answer the query.
• —num_distractors: Number of generated distractor variables; always 0 in this release.
• —var_layer: Mapping from variable name to dependency layer.

Generation and reproducibility

The frozen data was generated with:

python generator/construct.py \
  --ops arith \
  --n 250 \
  --depths 2 3 4 \
  --distractors 0 \
  --seed 2 \
  --outdir generated

The authoritative data file is data/test.jsonl, with SHA-256:

c8d77a9d3cc9cc6c5631b1b973231b674ea3d8845ec08fe5de406873e7efca9b

Run the included standard-library verifier before publishing or after downloading:

python verify_dataset.py

The verifier checks the checksum, schema, unique IDs, answer range, per-depth counts and answer histograms, and independently evaluates every symbolic problem with the bundled generator.

See manifest.json for the complete generation parameters, expected statistics, source commit, and file checksums. The frozen JSONL is the canonical artifact; regenerating it is an audit mechanism rather than a runtime step.

Intended use

This dataset is intended for zero-shot evaluation of language models on synthetic multi-step modular arithmetic. It is suitable for a multiple-choice/log-likelihood task in lm-evaluation-harness with raw accuracy as the primary metric.

It is not intended as a training corpus. The included cot field must not be inserted into the evaluation prompt unless a separate chain-of-thought setting is explicitly being studied.

Limitations

• —The dataset is small and entirely synthetic.
• —It tests modulo-7 symbolic arithmetic, not general mathematical ability.
• —It contains only one fixed seed and no train or validation split.
• —Public release makes future training contamination possible; record model training dates and decontamination policy when reporting results.
• —Tokenization can affect raw continuation likelihoods. For OLMo-compatible evaluation, use the exact lowercase prompt and space-prefixed digit choices described above.

Provenance

The data and generator were prepared from olmo-eval-suite commit 72e5db1548589efc04dc3f8628b0daa7a342a173. The generator is included in this repository so that every record can be independently verified.

License

The repository contents are released under the Apache License 2.0. See LICENSE.