RLLab/eval-set

<!-- matharena-aime -->

AIME sources

The aime24 and aime25 transcriptions are from MathArena by Jasper Dekoninck et al. Their CC BY-NC-SA 4.0 license applies to these two configs. The other configs retain their respective upstream licenses.

Pinned sources:

• —MathArena/aime_2024_I
• —MathArena/aime_2024_II
• —MathArena/aime_2025

Previous AIME mirrors used by AetherEval:

• —aime24: HuggingFaceH4/aime_2024.
• —aime25: yentinglin/aime_2025.

The pre-migration eval-set configs matched those AetherEval copies in all 30 questions per year, row order and answers, ignoring whitespace and the appended math instruction. The original eval-set import provenance was not recorded, so these are identified as matching previous mirrors, not a verified import history.

Preprocessing: remove Asymptote blocks ([asy], <asy> and the LaTeX asy environment) and TikZ tikzpicture environments. Preserve all prose, mathematical formulas and tables outside those blocks. Collapse excess blank lines, trim outer whitespace, and append the step-by-step / boxed-answer instruction. Question order and gold formatting are unchanged.

Reference: Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs. <!-- /matharena-aime -->

<!-- eval-set-repairs -->

Previous data repairs

The original commits contain the complete row-level changes:

• —OlympiadBench (675 → 674 questions): previous content matched knoveleng/OlympiadBench, an older mirror; the original import provenance was not recorded. Rebuilt from Hothan/OlympiadBench, config OE_TO_maths_en_COMP. Updated 15 relay question transcriptions from the official release, removed one question removed upstream, and followed upstream's degree-unit formatting for one gold answer. Full commit.
• —MinervaMath (272 questions, unchanged): source knoveleng/Minerva-Math. The pre-repair RLLab/eval-set / minervamath rows match this source in all 272 questions, solution texts and row order, apart from the appended math instruction. Corrected the notation of 62 gold answers (58 scientific-notation and 4 symbolic rewrites) so math-verify preserves their intended values. Question text, order and mathematical answers are unchanged. Full commit.
• —Unparseable gold answers (5 rows): math-verify could not parse these golds, so every correct answer scored 0. Notation only; values unchanged. minervamath: -1./3 → -\frac{1}{3}, -3./2 → -\frac{3}{2}. olympiadbench: (-\infty, 0) \cup\{1\}. → (-\infty, 0) \cup\{1\}, t(0,4] → (0,4], m_{\max }=n^{2}-n-1 → n^{2}-n-1. The olympiadbench strings are verbatim from the official final_answer field. <!-- /eval-set-repairs -->

<!-- mcq-sources -->

Multiple-choice sources

• —`gpqa-d` (198 questions): official GPQA Diamond, from the simple-evals CSV (sha256 41d1213cd7a4998605a26c2798500652572007161b3a92817ba46b35befcd305). The previous config held 197 of these questions; its import provenance and option shuffle were not recorded. Options now follow the permutation of openai/simple-evals gpqa_eval.py (random.Random(0), rng.sample(range(4), 4) per question in CSV order), so gold letters differ from the previous revision. Question and option text is stripped of outer whitespace.
• —`mmlu-pro-subset` (10,681 questions): TIGER-Lab/MMLU-Pro test split without the math category (1,351 questions), in official order with official text. Gold letters are unchanged.

Both use the prompt The following are multiple choice questions (with answers)[ about <category>]. Think step by step ... \boxed{}, then Question:, a blank line, and Options: lettered from A. <!-- /mcq-sources -->