kvoudouris/chess-debate-puzzles

Chess Debate Puzzles

A stratified sample of Lichess mid/endgame chess puzzles annotated with Stockfish-evaluated moves across ten centipawn-quality bands. Designed for experiments in the spirit of AI Safety via Debate (Irving et al., 2018), where two AI agents argue for different moves and a judge must identify the objectively better one.

Motivation

Debate as an alignment technique asks whether a human (or AI) judge can identify the correct answer when two agents argue opposing positions — even when the judge could not have found the answer alone. Chess provides a clean testbed: Stockfish gives a ground-truth ordering of moves by centipawn value, so we can precisely control how much harder one debater's task is by varying the quality gap between the move they must defend and the optimal move.

Source Data

Derived from Lichess/chess-puzzles, which contains ~4 million puzzles extracted from real Lichess games. The source dataset is released under CC0 (public domain). Puzzle ratings use the Glicko-2 system.

Construction Methodology

1. Filtering

Only puzzles with an empty OpeningTags field are retained. This restricts the dataset to mid- and endgame positions, avoiding the heavy theory-dependence of opening puzzles where memorisation rather than reasoning would dominate debates.

2. Puzzle position

Each Lichess puzzle stores the board state (FEN) before the opponent's triggering move. The actual position the player must solve is obtained by applying Moves[0] (the trigger) to that FEN. The Lichess-labelled optimal move is Moves[1].

3. Stockfish verification

Each candidate puzzle is analysed with Stockfish at depth 20. Mate-in-N positions (where Centipawn is None) are excluded since centipawn comparisons are undefined there.

Disagreements between the Lichess-labelled solution and Stockfish's top move are retained and flagged via the optimal_verified column rather than discarded. This avoids the sampling bias that would arise from only keeping easy-to-verify positions. All centipawn-band columns are computed relative to stockfish_optimal_move (the depth-20 top move), which is the internally consistent ground truth regardless of whether it matches the Lichess label.

4. Centipawn-band annotation

For each retained puzzle, Stockfish's top 20 legal moves are evaluated. Each alternative move is assigned to the band corresponding to how many centipawns worse it is than the optimal:

Within each band the highest-ranked qualifying move is stored (i.e. the least-bad move in that band). The band column is null when no move in the top 20 falls in that range.

Centipawn differences are computed as abs(optimal_cp − candidate_cp), which is sign-correct for both white-to-move and black-to-move positions.

5. Difficulty levels and stratified sampling

Puzzles are divided into five difficulty levels by equal-width Glicko-2 rating bands (i.e. the full rating range is split into five equal intervals). This means bin sizes vary considerably — easy puzzles are far more common in the Lichess database than very hard ones.

<!-- DIFFICULTYTABLESTART --> | Level | Rating range | Puzzles available | |---|---|---| | very_easy | 399–982 | 899,717 | | easy | 983–1566 | 1,715,478 | | medium | 1567–2149 | 1,309,619 | | hard | 2150–2733 | 623,965 | | very_hard | 2734–3317 | 48,058 | <!-- DIFFICULTYTABLEEND -->

Sampling iterates through shuffled puzzles in each difficulty stratum until every (difficulty × band) cell contains at least 5 puzzles. A puzzle is only added to the dataset if it contributes to at least one cell that has not yet reached this minimum.

Some cells may remain below 5 (e.g. extreme centipawn gaps in very easy positions, or narrow gaps in very hard positions) and are flagged in the script output.

Cell Coverage

The table below shows the number of puzzles covering each (difficulty × cp band) cell. Cells with 0 could not be filled from the available puzzle pool.

<!-- CELLCOVERAGETABLESTART --> | | 0–100 | 100–200 | 200–300 | 300–400 | 400–500 | 500–600 | 600–700 | 700–800 | 800–900 | 900–1000 | |---|---|---|---|---|---|---|---|---|---|---| | veryeasy | 13 | 8 | 9 | 9 | 8 | 6 | 8 | 6 | 7 | 5 | | easy | 8 | 6 | 6 | 8 | 9 | 7 | 7 | 6 | 6 | 5 | | medium | 8 | 7 | 5 | 6 | 7 | 5 | 5 | 6 | 7 | 6 | | hard | 6 | 5 | 6 | 5 | 9 | 8 | 7 | 5 | 9 | 9 | | veryhard | 5 | 5 | 7 | 7 | 11 | 6 | 7 | 12 | 8 | 5 | <!-- CELLCOVERAGETABLEEND -->

Schema

Core puzzle fields

Move quality fields

Centipawn-band fields

For each band B in {0_100, 100_200, 200_300, 300_400, 400_500, 500_600, 600_700, 700_800, 800_900, 900_1000}:

A null value means no move in the top 20 fell within that band for this puzzle.

Intended Use

Each row provides a puzzle position together with a menu of moves at precisely controlled quality levels. A typical debate experiment would:

1. 1.Select a puzzle and choose a "suboptimal move" from a given centipawn band.
2. 2.Assign one AI agent to argue for optimal_move and another to argue for the suboptimal move.
3. 3.Have a judge model (or human) read the debate transcript and pick the better move.
4. 4.Compare judge accuracy across bands — harder bands (smaller cp deficit) make the suboptimal move easier to argue for and harder for the judge to reject.

The difficulty axis (puzzle rating) allows separate analysis of whether debate helps more in tactically complex positions.

Caveats

• —Stockfish depth 20 vs. Lichess cloud analysis. Lichess generated their puzzle labels with higher-depth cloud analysis. Where the two disagree, optimal_verified = False and stockfish_optimal_move is used as the reference for centipawn-band calculations. Disagreements most often arise in tactically rich positions with multiple near-equal top moves.
• —Top-20 move limit. Band columns beyond cp_300_400 are less reliably populated in simple positions where fewer than 20 legal moves exist or where the top 20 moves are all clustered within 300 cp of optimal.
• —Mid/endgame only. The OpeningTags filter removes all positions classified by Lichess as opening puzzles. This is conservative: some early-middlegame positions may still be excluded if Lichess tagged them with an opening name.
• —No repeated positions. Each puzzle_id appears at most once.

Reproduction

# Install dependencies
uv sync

# (Optional) authenticate with HuggingFace to push the result
huggingface-cli login

# Run — takes ~10–30 min depending on hardware and Stockfish depth
uv run python code/create_debate_dataset.py

Key parameters (edit at the top of code/create_debate_dataset.py):

Citation

If you use this dataset, please also cite the original Lichess puzzle database:

@dataset{lichess_puzzles,
  author    = {Lichess},
  title     = {Lichess Chess Puzzles},
  year      = {2024},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/Lichess/chess-puzzles}
}