violetxi/chess-20m-tactical-eval-traces

Exact 20M Chess Tactical Evaluation Traces

This dataset makes the complete checkpoint evaluation at violetxi/chess-20m-tactical-eval-traces inspectable. It contains the exact benchmark prompt, batched input IDs and attention mask, released target, all 16 decoded model generations, extracted LAN/UCI moves, legality/correctness flags, and per-puzzle pass@k values.

The original evaluation JSON retained every extracted UCI prediction but not the raw decoded strings. The generated_text field here was deterministically reconstructed with the identical checkpoints, evaluator, seed, batching, and device class. All 497,280 reconstructed UCI predictions matched the saved evaluation traces exactly. Text after the first extracted move cannot be independently byte-compared with the original because that raw text was not retained.

Source dataset

• —Repository: pavelslab-nyu/chess_puzzle_benchmark
• —Pinned revision: 56ca4a68faf0eac0a2963d0ca4382a079ef35458
• —Variant: non-think
• —Raw rows: 1,484
• —Retained rows: 1,480; four prompts longer than 512 tokenizer tokens are excluded
• —B1-B4 / puzzle rating <2400: 1,160 puzzles
• —B5 / puzzle rating >=2400: 320 puzzles

Here, rating means Lichess puzzle difficulty, not either source player's Elo.

Exact input and target

The raw prompt is a PGN game history ending immediately before the move to predict. prompt_token_ids is produced by the 81-token compositional LAN tokenizer after removing its trailing EOS.

For the causal model, batches are sorted by prompt length and left-padded with masked BOS IDs. Generation is autoregressive.

For HRM, the input is the prompt followed by 20 <unk> target slots. A single bidirectional fixed-segment forward pass produces logits for those slots, and each slot is sampled independently. model_input_token_ids, model_attention_mask, and hrm_target_positions store the exact tensors used for each row, including batch padding.

No labels are supplied during generation. The evaluation target is the first UCI move in the official reward_model field.

Exact scoring

Each checkpoint produces 16 samples at temperature 1.0 with 20 new-token slots and seed 20260808. The evaluator extracts the first complete LAN move, converts it to UCI using the supplied FEN, checks legality, and requires exact equality with target_uci.

For a puzzle with c correct samples out of n=16:

pass@k = 1 - C(n-c, k) / C(n, k).

Aggregate metrics average the per-puzzle values. The published macro metric first averages within B1-B5 and then gives each bracket equal weight.

Included checkpoints

• —Causal rows: 14,800
• —HRM rows: 16,280
• —One row corresponds to one checkpoint-puzzle pair.

Important interpretation note

The HRM traces use one-shot independent sampling of the marginal distribution at each masked token position. A normal LAN move spans multiple tokens, so this can combine individually plausible piece/source/destination tokens into an illegal move. These traces faithfully expose the evaluation that was run; they should not be interpreted as a legal-move-constrained HRM evaluation.

Reproducibility files

• —evaluation_spec.json: immutable settings and artifact hashes
• —benchmark_manifest.json: pinned benchmark file hashes and filter counts
• —evaluator/eval_tactical_sweep.py: evaluator source used for generation/scoring
• —metrics/: aggregate and per-checkpoint result files
• —figures/checkpoint_pass_at_k.png: checkpoint curve

Models

• —violetxi/chess-causal-20m-elo-lt2400
• —violetxi/chess-hrm-20m-elo-lt2400