Datasetpublic

FUSE-verifiers/IMO-Shortlist-Verifications

IMO Shortlist with Qwen3-30B-A3B-Thinking-2507 This dataset contains 123 questions from the International Mathematical Olympiad (IMO) Shortlist subset of the IMO AnswerBench benchmark with 50 candidate responses generated by Qwen3-30B-A3B-Thinking-2507 for each problem. Each response has been evaluated for correctness using a mixture of DeepSeek-R1-Distill-Llama-70B and Python code to parse different answer formats, and scored by multiple LLM judges according to a 0-5 rubric.… See the full description on the dataset page: https://huggingface.co/datasets/FUSE-verifiers/IMO-Shortlist-Verifications.

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face

0likes130downloads

Dataset Card

IMO Shortlist with Qwen3-30B-A3B-Thinking-2507

This dataset contains 123 questions from the International Mathematical Olympiad (IMO) Shortlist subset of the IMO AnswerBench benchmark with 50 candidate responses generated by Qwen3-30B-A3B-Thinking-2507 for each problem. Each response has been evaluated for correctness using a mixture of DeepSeek-R1-Distill-Llama-70B and Python code to parse different answer formats, and scored by multiple LLM judges according to a 0-5 rubric. The benchmark consists of modified versions of past problems in the IMO Shortlist, where the modification is done by experts to help avoid memorization.

Dataset Structure

—Split: Single split named "data"
—Number of rows: 123 HLE questions
—Generations per query: 50

Key Fields

Field	Type	Description
`answer`	`str`	Correct ground-truth answer
`category`	`str`	IMO category (Algebra, Combinatorics, Geometry, and Number Theory)
`extracted_answers`	`List[str]`	Final answers extracted from completions
`id`	`str`	ID of question taken from IMO AnswerBench
`instruction`	`str`	Prompt given to Qwen3-30B-A3B-Thinking-2507
`is_correct`	`List[bool]`	Whether each extracted answer matches the correct groundtruth answer (50 per problem)
`question`	`str`	Content of question taken from IMO AnswerBench
`responses`	`List[str]`	Model-generated answers (50 per problem)
`source`	`str`	Question source, including year the question was on IMO Shortlist
`subcategory`	`str`	Subcategory of problem (problem solving technique, topic etc.)
`*_scores`	`List[float]`	Scalar scores on a 0-5 rubric from verifier models (50 per problem)
`*_justification`	`List[str]`	Justifications from verifier models (50 per problem)

Verifier Models

—DeepSeek-R1-Distill-Qwen-32B
—Kimi-Linear-48B-A3B-Instruct
—Llama-3.3-70B-Instruct
—Ministral-3-14B-Reasoning-2512
—Ministral-3-8B-Instruct-2512
—NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
—Qwen3-30B-A3B-Thinking-2507
—gemma-3-27b-it
—gpt-oss-20b

Source

Original IMO Shortlist problems from superhuman/imobench. Description of data provided at imobench.github.io.