joanvelja/polaris-53k-repaired

POLARIS-53K, label-repaired

49,289 of the 53,291 rows in `POLARIS-Project/Polaris-Dataset-53K`, with 4,580 stored answers corrected and 4,002 rows removed as unrepairable.

Measurements on the source set put its bad-label rate at roughly 15.9% [14.3, 17.6] (two independent detectors agreeing on a 2,000-row sample). Mislabelled rows are not uniformly distributed: they concentrate in the problems models fail, which is exactly where a difficulty-calibration pipeline looks.

What was done

Every row flagged as suspect was solved blind by two independent strong models — they saw the problem and neither the stored answer nor each other's answer. A row is:

• —repaired when both arms independently produce the same answer and it differs from the stored one (label_repaired: true)
• —kept unchanged when both arms reproduce the stored answer (the flag was wrong)
• —dropped when the arms disagree, when both judge the problem statement broken, or when either arm declines to answer

Answer equivalence uses the same math_verify + LLM-judge chain the downstream trainer grades with, so a repaired label is not one the grader would then reject.

Validation

• —30 synthetic controls with known true answers, injected as corrupted labels: repairs were 24/24 correct, 95% CI [86.2%, 100%], with 0 corrupted labels retained.
• —257 known-different answer pairs, contributed by 12 independent reviewers (anagrams, tuple permutations, numeric underflow, symbol case, ratio/time parsing, extremum heads, operator arguments): 0 false positives.
• —Shipped rows carry no abstention, non-answer, or nonexistence strings as labels, and no character-corruption artefacts.

Known limitations

• —39% of the negative corpus passes by deferring rather than discriminating. For those shapes the guarantee rests on the LLM judge, whose accuracy on deferred pairs specifically has not been measured. This is the main open risk.
• —The blind protocol measures specificity (it does not invent defects: 0/30 on controls) but there is no positive control for sensitivity, so the fraction of genuinely-broken problems it misses is unknown.
• —label_repaired: true marks a machine-corrected answer. It has not been checked by a human.

Fields

Source schema, plus label_repaired (bool) on corrected rows.

Audit v2 swap (2026-09-06)

Problem statements and reference answers were swapped in place from the POLARIS well-posedness audit (figure-dependent rows rendered into text, yes/no questions converted to their extremal quantity, multiple-choice option blocks stripped, damaged or unguessable-answer rows dropped). Row ids are unchanged. Measurement columns were computed on the pre-swap text and are stale for rows tagged problem_changed / answer_changed.

Drop reasons (train split): duplicate_problem 1146, park:unrecoverable 402, park:letter_answer 239, review_drop 153, figure_unrecoverable 62, figure_not_found 48, mc_letter_answer 43, mc_refers_to_options 18, review_regate_mismatch 14.

tags (list): audit origin (keep, repaired, reviewed, figure_reconstructed), review pile and verdict (review_<pile>, review_<verdict>), figure_<outcome>, arbitrated, glm_unsolved / unverified (no glm-5.3-flash solve matched the reference), mc_options_stripped, problem_changed, answer_changed.