khoilamalphaai/chess-coach-move-review

Chess coach move-review SFT dataset

Supervised fine-tuning data for one specific, trained behavior: given a chess position and the student's rating tier (Beginner, Intermediate, or Advanced), select the tier-appropriate instructive move and tag it with a short principle, for example "Nf3, develop toward the center."

That single move choice is the trained objective, and it is deterministically checkable. The plain-English explanation rendered beside the move is a secondary display layer. It is still present in the SFT loss, but it is not separately optimized and it is not the evaluation claim. The truthfulness of that prose is enforced at serving time by a separate non-LLM verifier, not by the fine-tune.

This framing matches the project thesis documents (BRAINLIFT.md and README.md in the source repo): the fine-tune reproduces a tier-selection policy that a prompt on the same base does not, while the canonical move itself is defined by engine and human-move grounding, not by the model weights.

Shipped version: v4 (default)

v4/train.jsonl is the training set behind the shipped coach, the chess-coach-32b-v4-qlora adapter and its sibling small-model coaches. The v4 config is the default:

from datasets import load_dataset

ds = load_dataset("khoilamalphaai/chess-coach-move-review")  # resolves to v4
print(ds)                            # DatasetDict with train / validation
print(ds["train"][0]["messages"])    # a system / user / assistant chat triple

Earlier iterations remain available and are selected by name:

ds_v3 = load_dataset("khoilamalphaai/chess-coach-move-review", "v3")

Splits and sizes

Validation splits are game-disjoint holdouts, and the shipped model's evaluation slice is checked to have zero board-key overlap with its training data.

Row schema

Every row is a JSON object with a single key, messages: an OpenAI-style chat list of exactly three turns (system, user, assistant). There is no separate metadata column; all grounding lives inside the text of the turns.

Tiers and the leveling signal

Three tiers are used throughout: Beginner (1000-1200), Intermediate (1300-1600), and Advanced (1700-2000). The dataset is built as contrastive multi-tier sets: the same position is taught at more than one tier so the correct label varies by level. A beginner is guided to a sound, human-findable move; an advanced player is guided toward the sharper engine-best line. This is what directly supervises the model to change its move by tier, the one behavior a prompt on the base does not reliably provide.

How it was built

The dataset is fully synthetic, distilled from a frontier teacher on top of real engine grounding.

Move labels are engine-verified: the recommended move is always drawn from and checked against the Stockfish sound pool, so there are no unsound move labels. The prose check is high-precision but low-recall: it drives verifier-detectable board-fact violations to zero, which is not the same as certified truthfulness (relational, threat, and evaluation claims it does not cover can still be imperfect).

What the data buys (measured)

The behavior is graded deterministically as tier-policy exact match, meaning agreement with the select_tier_move rule computed from the engine and Maia, with no LLM judge in the loop (this metric is labeled tier-fit in some benchmark tables). On the held-out validation slice, the v4-trained 32B adapter reaches:

• —tier-policy match 0.767, versus 0.347 for the untuned Qwen3-32B base and 0.553 for the best frontier reference (Gemini 3.1 Pro),
• —distinct-moves-per-level 0.730 (73 of 100 positions where a beginner and an advanced label should differ),
• —raw move-soundness 0.942.

Over all positions where v4 diverges from the best frontier, the unbiased head-to-head is 56-24-12. The frequently cited 51-5-6 is a selection-conditioned subset (the 62 positions where v4 already gives a distinct, sound, correctly-graded move), so it overstates a general win rate. These numbers measure agreement with the project's own move rule, not validated teaching quality. Because the evaluation is periodically regenerated, refer to the live artifacts for the current field:

Version lineage

The shipped set was reached through a documented iteration line, kept here so results stay reproducible:

• —v1 to v2: the original data intervention. v2 is the genuinely small, on-spec result; the 1.7B tune it produced reproduces the tier policy where a prompt on the same base cannot.
• —v3: a 32B all-rounder that balanced move selection against prose quality.
• —v4 (shipped): a 32B set tuned to maximize tier-policy match. It leads the field on move selection and is deliberately weaker on prose, which is consistent with prose being an optional layer rather than the trained objective.

A deeper-verified v6 rebuild exists as the current data frontier. It is not published here as a config, and the shipped v4 data is left untouched. The v6 labels are re-derived with Stockfish 17 using a two-depth root search (depths 14 and 20 with agreement bands), Syzygy tablebases for endgames of seven pieces or fewer, and Maia-2 as a human-likelihood constraint. It builds complete triads by construction (every board carries all three tiers, with no collapsed levels), fixes the advanced tier to the verified engine-best move, and re-derives about 45% of the training labels (3,307 of 7,269 comparable rows). It feeds the downstream DPO and engine-distillation retrains.

Limitations and honest caveats

• —Advanced tier is close to engine-best. By construction the advanced label is the sharp engine move, so advanced rows largely mirror the engine. The distinctive leveling signal is strongest at the Beginner and Intermediate tiers.
• —The policy is learnable, not deployment-necessary. select_tier_move already computes the canonical move at about 1.0 from the same grounding, so a model trained on this data approximates a policy the grounded product already produces. The fine-tune becomes load-bearing only in a grounding-free, fully-local setting that was not built or measured.
• —Grounding is required at inference. A model trained here reproduces the move policy reliably only when given the same Stockfish and Maia grounding in the prompt. Without the per-tier Maia signal, the three tiers collapse to a single move.
• —The explanation gate is not certified truth. Move labels are engine-sound, but the prose verifier is high-precision and low-recall. Truthfulness at serving time depends on the runtime verifier, not on further fine-tuning.
• —tier-policy match is fidelity to a heuristic. It measures agreement with select_tier_move, a project rule, not certified best teaching. Whether these moves help students learn faster is unvalidated.
• —Teacher distillation. Move labels are hard-filtered against the engine, but the "instructive" judgment in the prose is distilled from an LLM teacher (GPT-5.5), not ground truth.

Links and license

Positions derive from the public CC0 Lichess Open Database; the coaching text is distilled from GPT-5.5. Released for research and education under CC-BY-NC-4.0. Respect the source models' terms.