ethanjtang/GAMBIT-stockfish18-selfplay

GAMBIT: <ins>G</ins>ener<ins>a</ins>lization or <ins>M</ins>emorization? <ins>B</ins>r<ins>i</ins>ttleness <ins>T</ins>esting for Chess-Trained Language Models

 <br>  <br>  <br>

Overview

1050 self-play games between base Stockfish 18 and Stockfish Skill Level 0-20 variants.

Each skill level variant plays 25 White/25 Black games against a base instance of Stockfish.

All games use depth=15 and no opening books were used.

• —sf18_selfplay_games.pgn — PGN file containing all labeled 1050 SF18 self-play games
• —sf18_selfplay_match-summary.txt — match summary between SF18 Base vs. SF18 Level 0-20
• —sf18_selfplay_positions.parquet — All unique position + best move pairs from all games played as (fen, uci, san, eval_cp) tuples.
• —sf18_selfplay_unique-position-bestmove-pairs.txt — Filtered unique position + best move pairs from only "reasonable" players (engines who won/drew)
• —A position + best move pair is only recorded if the engine which played it won/drew the game.
• —This filters for bad moves played by weak SF18 variants.

Citation

@misc{tang2026generalizationmemorizationbrittlenesstesting,
      title={Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models}, 
      author={Ethan Tang},
      year={2026},
      eprint={2605.17565},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2605.17565}, 
}