RLLab/eval-set

1---2dataset_info:3- config_name: aime244  features:5  - name: problem6    dtype: string7  - name: solution8    dtype: string9  splits:10  - name: train11    num_bytes: 1198312    num_examples: 3013  download_size: 921014  dataset_size: 1198315- config_name: aime2516  features:17  - name: problem18    dtype: string19  - name: solution20    dtype: string21  splits:22  - name: train23    num_bytes: 1380024    num_examples: 3025  download_size: 1035326  dataset_size: 1380027- config_name: amc2328  features:29  - name: problem30    dtype: string31  - name: solution32    dtype: float6433  splits:34  - name: train35    num_bytes: 1408836    num_examples: 4037  download_size: 965138  dataset_size: 1408839- config_name: gpqa-d40  features:41  - name: problem42    dtype: string43  - name: solution44    dtype: string45  splits:46  - name: train47    num_bytes: 16910248    num_examples: 19849  download_size: 7782650  dataset_size: 16910251- config_name: math50052  features:53  - name: problem54    dtype: string55  - name: solution56    dtype: string57  splits:58  - name: train59    num_bytes: 40309060    num_examples: 50061  download_size: 20435962  dataset_size: 40309063- config_name: minervamath64  features:65  - name: problem66    dtype: string67  - name: solution68    dtype: string69  splits:70  - name: train71    num_bytes: 23149872    num_examples: 27273  download_size: 10171674  dataset_size: 23149875- config_name: mmlu-pro-subset76  features:77  - name: problem78    dtype: string79  - name: solution80    dtype: string81  splits:82  - name: train83    num_bytes: 1003533784    num_examples: 1068185  download_size: 395867786  dataset_size: 1003533787- config_name: olympiadbench88  features:89  - name: problem90    dtype: string91  - name: solution92    dtype: string93  splits:94  - name: train95    num_bytes: 27498296    num_examples: 67497  download_size: 10981198  dataset_size: 27498299configs:100- config_name: aime24101  data_files:102  - split: train103    path: aime24/train-*104- config_name: aime25105  data_files:106  - split: train107    path: aime25/train-*108- config_name: amc23109  data_files:110  - split: train111    path: amc23/train-*112- config_name: gpqa-d113  data_files:114  - split: train115    path: gpqa-d/train-*116- config_name: math500117  data_files:118  - split: train119    path: math500/train-*120- config_name: minervamath121  data_files:122  - split: train123    path: minervamath/train-*124- config_name: mmlu-pro-subset125  data_files:126  - split: train127    path: mmlu-pro-subset/train-*128- config_name: olympiadbench129  data_files:130  - split: train131    path: olympiadbench/train-*132---133 134 135<!-- matharena-aime -->136## AIME sources137 138The `aime24` and `aime25` transcriptions are from [MathArena](https://matharena.ai/) by Jasper Dekoninck et al. Their [CC BY-NC-SA 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/) applies to these two configs. The other configs retain their respective upstream licenses.139 140Pinned sources:141 142- [MathArena/aime_2024_I](https://huggingface.co/datasets/MathArena/aime_2024_I/tree/ea5b061c3e8039dc9858defaafc407d04b995e9f)143- [MathArena/aime_2024_II](https://huggingface.co/datasets/MathArena/aime_2024_II/tree/29d5d31e9b46e215fc24d9b2a3047506823dd101)144- [MathArena/aime_2025](https://huggingface.co/datasets/MathArena/aime_2025/tree/c94da77eb22bbd6439e62a323bec18493a421302)145 146Previous AIME mirrors used by AetherEval:147 148- `aime24`: [HuggingFaceH4/aime_2024](https://huggingface.co/datasets/HuggingFaceH4/aime_2024).149- `aime25`: [yentinglin/aime_2025](https://huggingface.co/datasets/yentinglin/aime_2025).150 151The pre-migration `eval-set` configs matched those AetherEval copies in all 30 questions per year, row order and answers, ignoring whitespace and the appended math instruction. The original `eval-set` import provenance was not recorded, so these are identified as matching previous mirrors, not a verified import history.152 153Preprocessing: remove Asymptote blocks (`[asy]`, `<asy>` and the LaTeX `asy` environment) and TikZ `tikzpicture` environments. Preserve all prose, mathematical formulas and tables outside those blocks. Collapse excess blank lines, trim outer whitespace, and append the step-by-step / boxed-answer instruction. Question order and gold formatting are unchanged.154 155Reference: [Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs](https://arxiv.org/abs/2605.00674).156<!-- /matharena-aime -->157 158<!-- eval-set-repairs -->159## Previous data repairs160 161The original commits contain the complete row-level changes:162 163- **OlympiadBench (675 → 674 questions):** previous content matched [knoveleng/OlympiadBench](https://huggingface.co/datasets/knoveleng/OlympiadBench), an older mirror; the original import provenance was not recorded. Rebuilt from `Hothan/OlympiadBench`, config `OE_TO_maths_en_COMP`. Updated 15 relay question transcriptions from the official release, removed one question removed upstream, and followed upstream's degree-unit formatting for one gold answer. [Full commit](https://huggingface.co/datasets/RLLab/eval-set/commit/93bfada9b7e597586ab6cdf68d98ff88ae4a3f3a).164- **MinervaMath (272 questions, unchanged):** source [knoveleng/Minerva-Math](https://huggingface.co/datasets/knoveleng/Minerva-Math/tree/93d86e1f779d84c7393587c2cdbf6c7511591f95). The pre-repair `RLLab/eval-set` / `minervamath` rows match this source in all 272 questions, solution texts and row order, apart from the appended math instruction. Corrected the notation of 62 gold answers (58 scientific-notation and 4 symbolic rewrites) so `math-verify` preserves their intended values. Question text, order and mathematical answers are unchanged. [Full commit](https://huggingface.co/datasets/RLLab/eval-set/commit/c774e009046b9173716066989a392d0851ea9e4a).165- **Unparseable gold answers (5 rows):** `math-verify` could not parse these golds, so every correct answer scored 0. Notation only; values unchanged. `minervamath`: `-1./3` → `-\frac{1}{3}`, `-3./2` → `-\frac{3}{2}`. `olympiadbench`: `(-\infty, 0) \cup\{1\}.` → `(-\infty, 0) \cup\{1\}`, `t(0,4]` → `(0,4]`, `m_{\max }=n^{2}-n-1` → `n^{2}-n-1`. The olympiadbench strings are verbatim from the official `final_answer` field.166<!-- /eval-set-repairs -->167 168<!-- mcq-sources -->169## Multiple-choice sources170 171- **`gpqa-d` (198 questions):** official GPQA Diamond, from the [simple-evals CSV](https://openaipublic.blob.core.windows.net/simple-evals/gpqa_diamond.csv) (sha256 `41d1213cd7a4998605a26c2798500652572007161b3a92817ba46b35befcd305`). The previous config held 197 of these questions; its import provenance and option shuffle were not recorded. Options now follow the permutation of openai/simple-evals `gpqa_eval.py` (`random.Random(0)`, `rng.sample(range(4), 4)` per question in CSV order), so gold letters differ from the previous revision. Question and option text is stripped of outer whitespace.172- **`mmlu-pro-subset` (10,681 questions):** [TIGER-Lab/MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro/tree/b189ec765aa7ed75c8acfea42df31fdae71f97be) test split without the `math` category (1,351 questions), in official order with official text. Gold letters are unchanged.173 174Both use the prompt `The following are multiple choice questions (with answers)[ about <category>]. Think step by step ... \boxed{}`, then `Question:`, a blank line, and `Options:` lettered from A.175<!-- /mcq-sources -->176