arthu1/signalmaze-400

SignalMaze-400

SignalMaze-400 is a 400-item, source-grounded evaluation set for quantitative reasoning over real public statistics. It rewards reading supplied evidence and executing the requested operation; it is deliberately unsuitable as generic memorization practice.

Why it is useful

• —Every answer is derived from a live World Bank Open Data observation, with an item-level API URL, indicator code, country, and retrieval date.
• —Inputs are compact, self-contained time-series extracts. Models must distinguish signed change, percentage change, extrema, and comparison; world knowledge alone is insufficient.
• —The release uses 52 countries, 4 unrelated indicators, 7-year windows, and four different operations. IDs, contexts, and answers are unique.
• —Numeric grading uses an explicit plus-or-minus 0.01 tolerance; year answers require exact match.

Format and evaluation

data/test.jsonl contains question, context_markdown, answer, answer_type, tolerance, rationale, and structured provenance. Do not train on this set. Give models only the context and question, hiding answer and rationale. Report results by skill and difficulty, not only an aggregate.

Reproducibility and limits

The builder fetches the World Bank API at build time. Released records preserve source URLs and retrieval date for audit if upstream historical values change. This is a narrow structured-public-data benchmark, not a proxy for broad human intelligence. It reduces fact-recall leakage and template-only wins, but should be paired with held-out domains for serious model selection.

Attribution

Source values: World Bank Open Data, CC BY 4.0. Dataset packaging: Apache-2.0.