JWei05/Polaris-hard-w-solutions-24209

Polaris-Hard-w-Solutions

24,209 hard competition-math problems (the hardest difficulty bands of the Polaris dataset) paired with two verified solutions each: a full original solution and a concise summarized solution. Every retained problem has a machine-verifiable final answer, every solution's boxed answer grades correct against the reference (sympy-based grading), and the summarized solutions have additionally been put through a reasoning-rigor pass (see step 5 below).

This is mainly hard competition-math questions, filtered to those with easily verifiable answers. It is intended for generating hints from the summarized solutions and for generative value-function work, where a clean, length-controlled target solution is wanted alongside the original (longer) derivation.

How the dataset was generated

Starting from Polaris problems in difficulty bands 0/8, 1/8, 2/8, 3/8 (Polaris labels 0/8 as the hardest), the pipeline was:

1. 1.Solve (answer-free, then graded). The reference answer was never shown during solving, so a kept solution is a legitimate independent solve that was subsequently graded correct.
2. 2.Qwen3.5-35B-A3B was evaluated on every problem (temperature 1.0, top-p 0.95, top-k 20). For correct traces, the presented (post-thinking) solution was kept.
3. 3.For problems Qwen3.5-35B-A3B got wrong, gpt-5.5 (via the Codex CLI) attempted a fresh solve, escalating reasoning effort low → high, keeping the first attempt that graded correct.

1. 1.Length-normalize to a clean target (`summarized_solution`). Each correct original solution was processed by Qwen3.5-9B into a 200–500 token target:
2. 2.solutions < 80 tokens were elaborated into a full worked solution (action = elaborated);
3. 3.solutions > 600 tokens were summarized (action = summarized);
4. 4.solutions in 80–600 tokens were kept verbatim (action = kept).
5. 5.The rewrite was re-graded; rewriting repeated until the boxed answer graded correct.

1. 1.Tighten the long tail. Any rewritten solution still > 1000 tokens was re-summarized with gpt-5.5 (Codex, low effort), keeping the result only if it still graded correct and was shorter. A handful of pathological cases were re-solved fresh from the problem + answer.

1. 1.Filter to machine-verifiable answers. Problems whose reference answer is not a concrete checkable value were dropped, because for those the grading would be meaningless. Removed categories:
2. 2.multiple-choice-only answers (\textbf{(A)}, (C)\6) and bare choice letters;
3. 3.free-form prose answers;
4. 4.proof-type problems, where the "answer" merely restates the goal rather than being a value to verify — e.g. geometry predicates (OH\perp MN, "prove M, N, P, Q are concyclic"), relations and identities to prove (KM=LN, abc=0, l_1^2 > \sqrt 3 S > l_2^2), and yes/no existence questions.

Every retained answer is a concrete value the grader handles, and the final set is 100% grade-correct on the summarized solutions.

1. 1.Reasoning-rigor pass. Grading only checks the final answer, so a solution can reach the right answer through faulty reasoning. To raise reasoning quality, every summarized solution was triaged:
2. 2.all logic-grid / "Einstein" puzzles (~4,460) and every non-puzzle that a gpt-5.5 rigor check judged not rigorous were regenerated from scratch with gpt-5.5 at high reasoning effort into a clean, fully-justified ≤1000-token solution, re-graded against the answer (rigor_status = regenerated / puzzle_regenerated);
3. 3.solutions the check judged rigorous were kept unchanged (rigor_status = rigorous_kept).

The grade-and-retry step doubled as a reference-answer validator: a handful of problems where gpt-5.5 could never reproduce the stored answer turned out to have wrong or ill-posed reference answers — these were corrected (reference_fixed_resolved) or dropped, and a few solutions with a correct answer but a false intermediate lemma were re-derived (false_logic_repassed).

An independent adversarial audit (re-solve + reasoning critique on a stratified sample, every flag re-verified) put the post-pass set at ~96% rigorous-or-minor-gaps reasoning and <1% "right answer via wrong reasoning" — a ~5–6× reduction in defective reasoning versus before the pass.

Columns

All token lengths use the shared Qwen3.5 tokenizer (Qwen/Qwen3.5-9B).

Statistics

• —24,209 problems. By difficulty: 0/8: 10,887 · 1/8: 4,799 · 2/8: 4,315 · 3/8: 4,208.
• —Original-solution source: Qwen3.5-35B-A3B ~21,700 · gpt-5.5 ~2,500.
• —Summarized-solution source (after the rigor pass): gpt-5.5 (high-effort, rigorized) 12,709 · Qwen3.5-9B 9,220 · other kept (verbatim / earlier gpt-5.5 passes) 2,280.
• —Rigor status: regenerated 8,240 · puzzle_regenerated 4,458 · rigorous_kept 11,499 · reference_fixed_resolved 8 · false_logic_repassed 3 · skipped_kept 1.
• —summarized_solution length (Qwen3.5 tokens): median 600, mean 598, p95 933.
• —original_solution length (Qwen3.5 tokens): median ~1,021, mean ~1,280.

Verification

Every summarized_solution ends with its answer in \boxed{}, and that answer grades correct against answer via sympy-based equivalence checking. Solutions were produced answer-free (the model was not shown the reference answer when solving) and only graded afterward, so they are genuine derivations rather than justifications of a supplied answer.

Source & license

Problems are derived from the Polaris dataset; please cite Polaris if you use this data. Solutions were generated by Qwen3.5 models and gpt-5.5 as described above.