khoilamalphaai/chess-coach-grand-eval

Chess Coach — Grand Eval (comprehensive leaderboard)

One fresh, apples-to-apples comparison of every model in the chess move-review coaching project — our tuned specialists, the untuned baselines, and the full frontier lineup — on the same held-out validation slice (120 positions × 3 tiers = 360 scenarios), scored with two independent layers:

1. 1.Deterministic moat metrics (free, python-chess over pre-computed Stockfish/Maia facts): tier-fit, distinct-moves-per-level, move-soundness, raw faithfulness (verify-pass on draft 1), tier-coherence, and shipped-gate soundness.
2. 2.Blinded cross-family frontier council (GPT-5.5 + Claude Opus 4.8 + Gemini 3.1 Pro via the TrueFoundry gateway), grading each anonymised response 0–10 on move and instructiveness, with 95 % cluster-bootstrap CIs. Council: 225 items × 3 judges = 675 gradings.

Every gateway (TFY) model was regenerated fresh on these exact positions; our Modal/MLX tuned models are deterministic given their adapter (reused where noted — see the "How each row was generated" table below).

Files

One fresh, apples-to-apples comparison of all 20 models — our tuned specialists, the untuned baselines, and the full frontier lineup — on the SAME held-out VAL slice, scored with BOTH layers:

• —Deterministic moat metrics (free; python-chess over pre-computed Stockfish/Maia facts) over all 120 positions × 3 tiers = 360 scenarios: tier-fit, distinct-moves-per-level, move-soundness, raw faithfulness (verify-pass on draft 1), tier-coherence, shipped-gate soundness.
• —Blinded cross-family frontier council (GPT-5.5 + Claude Opus 4.8 + Gemini 3.1 Pro via TrueFoundry), 0-10 move + instructiveness with 95% CIs, over 75 of the 120 positions (675 gradings) — sized to the TFY budget.

Every TFY gateway model was regenerated FRESH on these exact positions (never reusing the old frontier gens); our Modal/MLX tuned models are deterministic given their adapter (reused where noted). ours_v5 is the finish-v5 controller's fresh Modal Volume gen.

Frontier reachability: the 14-model lineup = 3 frontier APIs + 11 open candidates; 12 reachable (dsr1 via bedrock-oss-group/deepseek-r1), 2 blocked: llama4-maverick (400, Meta Llama access denied) and kimi-k2-thinking (403, not authorized).

Metric framing (2026-07-09 honest reframe)

Numbers below are the as-computed grand-eval values (canonical/frozen); the framing terms are aligned to the honest reframe:

• —"tier-fit" = "tier-policy exact match" — exact agreement with the preregistered select_tier_move rule, a PROJECT RULE, not validated pedagogy. Lead with the all-scenario number (v4 0.767 vs best frontier 0.553), not a head-to-head win rate.
• —The "moat" head-to-head is a project-rule metric, not a general win rate. The per-tuned W/L/T table below is SELECTION-CONDITIONED (only the positions where OURS already gives a distinct, sound, correctly-graded move AND diverges from the frontier). For v4, the honest unbiased head-to-head over all 92 diverging positions is 56-24-12 (56-24-40 over all 120), recomputed from the committed raw/greedy gens and asserted by scripts/reproduce_v4.py (it supersedes an earlier eval-audit figure that did not reproduce — same 56 wins and 12 ties, but four audit-only losses over four extra diverging positions that did not replay); the 51-5-6 over 62 shown below is the conditioned subset.
• —distinct-moves denominator: the distinct↑ column is distinct / (positions where the model named both tier moves) as computed at grand-eval time (v4 = 73/93 = 0.785). The honest all-opportunities denominator (every canonical beginner!=advanced position, a no-answer counting as a miss) gives v4 73/100 = 0.730; see data/benchmark_honest/report_v4.json.

Leaderboard — ranked by tier-appropriate move selection (the trained behavior)

Sort key: ranked by tier-appropriate move selection — the deterministic tier-fit↑ metric (the behavior we trained and the graded axis), ties broken by distinct-moves-per-level↑ then move-soundness↑. Every deterministic axis (tier-fit / distinct / move-sound / coherence and the moat) uses the canonical STRICT any-legal move extractor (coach_gate.pick_recommendation, accept = any legal move; no in-pool backfill) — so an output that names no clearly-legal move is a miss everywhere and the leaderboard method matches the moat method exactly. The per-tuned head-to-head W/L/T vs the best frontier is in the moat table below. Instructiveness (the blinded cross-family council) is shown as a secondary axis in the instr 0-10 / move 0-10 / rank↓ columns — OURS-v4 is intentionally weaker on council prose and that is reported here honestly and unchanged. Only the row order reflects tier-fit; every model's measured numbers are the deterministic + council values.

gen: FRESH = regenerated this run; reuse = deterministic adapter/MLX gen reused. gated: `yes` = full shipped verify-and-regenerate pipeline (4B trio); `raw` = ungated draft (raw-draft gate axes shown). raw-faith = verify-pass on draft 1 (1 − fabrication). tier-fit / distinct / move-sound / raw-faith / coherence are deterministic (free); instr / move 0-10 + rank are the blinded council.

The moat — each tuned model vs the best frontier (tier-fit then soundness)

On positions where OURS gives distinct, sound, correctly-graded per-tier moves AND diverges from the best-frontier move, who wins the platform's move-quality moat (the assemble.derive_wins definition)? Instructiveness (where the frontier leads) is reported separately above.

Shipped-gate soundness (tuned models through the SAME verify+fallback gate)

Once gated, tuned soundness/format hit a shared ~100% floor (zero verifier-detectable mechanical violations by construction) — a fairness floor, not a differentiator; the differentiators are tier-fit / distinct-moves / instructiveness.

Deterministic gate axes (raw draft for ungated rows; telemetry for gated 4B)

How each row was generated