ethanjtang/GAMBIT-lichess-puzzle-positions

GAMBIT: <ins>G</ins>ener<ins>a</ins>lization or <ins>M</ins>emorization? <ins>B</ins>r<ins>i</ins>ttleness <ins>T</ins>esting for Chess-Trained Language Models

 <br>  <br>  <br>

Overview

Training/validation split of puzzles used in GAMBIT: Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models

Parsed in text format from the Lichess Puzzle Database

• —sampling_log.txt — Displays statistics about samples for each theme (74 unique themes)
• —training-puzzle-positions.txt — Large set of puzzle positions as position + best move pairs
• —All positions in the training set are disjoint from all validation puzzles!
• —validation_puzzles_THEME — N=1000 sample of puzzles tagged with THEME
• —72625 total unique validation puzzles
• —Note that there is some overlap in puzzles between validation sets because puzzles are assigned multiple themes!

Sample

Training Positions

From: training-puzzle-positions.txt

<|position-start|>
FEN: r2qr1k1/b1p2ppp/p5n1/P1p1p3/4P1n1/B2P2Pb/3NBP1P/RN1QR1K1 w - - 0 17
Best move (UCI): e2g4
Best move (SAN): Bxg4
<|position-end|>

Each training position is separated into paragraphs via a newline character.

Validation Puzzles

From: validationpuzzlesmateIn1.txt

<|puzzle-start|>
<|position-start|>
FEN: r6r/1ppbk3/1b1p4/pP1PpnN1/P1P5/3P2p1/4N1PP/R2Q1R1K b - - 0 24
Best move (UCI): h8h2
Best move (SAN): Rxh2#
<|position-end|>
<|puzzle-end|>

From: validationpuzzlesmateIn3.txt

<|puzzle-start|>
<|position-start|>
FEN: 2r2rk1/3Q1ppp/pq2p3/1p2N3/1P3P2/P1P5/5nPP/RNB2RK1 b - - 4 18
Best move (UCI): f2h3
Best move (SAN): Nh3+
<|position-end|>
<|position-start|>
FEN: 2r2rk1/3Q1ppp/pq2p3/1p2N3/1P3P2/P1P4n/6PP/RNB2R1K b - - 6 19
Best move (UCI): b6g1
Best move (SAN): Qg1+
<|position-end|>
<|position-start|>
FEN: 2r2rk1/3Q1ppp/p3p3/1p2N3/1P3P2/P1P4n/6PP/RNB3RK b - - 0 20
Best move (UCI): h3f2
Best move (SAN): Nf2#
<|position-end|>
<|puzzle-end|>

Citation

@misc{tang2026generalizationmemorizationbrittlenesstesting,
      title={Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models}, 
      author={Ethan Tang},
      year={2026},
      eprint={2605.17565},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2605.17565}, 
}