CoolFace
Modelpublic

kaupane/ChessFormer-SL

sourceHugging Facemitupdated 1y agoView on Hugging Face
1likes55downloads
Model Card

ChessFormer-SL

ChessFormer-SL is a transformer-based chess model trained via supervised learning on Stockfish evaluations. This model explores training chess engines without Monte Carlo Tree Search (MCTS), using only neural networks.

Model Description

  • —Model type: Transformer for chess position evaluation and move prediction
  • —Language(s): Chess (FEN notation)
  • —License: MIT
  • —Parameters: 100.7M

Architecture

ChessFormer uses a custom transformer architecture optimized for chess:

  • —Blocks: 20 transformer layers
  • —Hidden size: 640
  • —Attention heads: 8
  • —Intermediate size: 1728
  • —Features: RMSNorm, SwiGLU activation, custom FEN tokenizer

Input Format

The model processes FEN strings and repetition counts, tokenizing them into 75-token sequences representing:

  • —64 board square tokens (pieces + positional embeddings)
  • —9 metadata tokens (turn, castling, en passant, clocks, repetitions)
  • —2 special tokens (action, value)

Output Format

  • —Policy head: Logits over 1,969 structurally valid chess moves
  • —Value head: Position evaluation from current player's perspective

Training Details

Training Data

  • —Dataset: kaupane/lichess-2023-01-stockfish-annotated (depth18 split)
  • —Size: 56M positions with Stockfish evaluations
  • —Validation: depth27 split

Training Procedure

  • —Method: Supervised learning on Stockfish move recommendations and evaluations
  • —Objective: Cross-entropy loss (moves) + MSE loss (values) + invalid move penalty
  • —Hardware: RTX 4060Ti 16GB
  • —Duration: ~2 weeks
  • —Checkpoints: 20 total, this model is the final checkpoint

Training Metrics

  • —Action Loss: 1.6985
  • —Value Loss: 0.0407
  • —Invalid Loss: 0.0303

Performance

Capabilities

  • —✅ Reasonable opening and endgame play
  • —✅ Fast inference without search
  • —✅ Better than next-token prediction chess models
  • —✅ Can defeat Stockfish occasionally with search enhancement

Limitations

  • —❌ Frequent tactical blunders in midgame
  • —❌ Estimated ELO ~1500 (informal assessment)
  • —❌ Struggles with complex tactical combinations
  • —❌ Tends to give away pieces ("free captures")

Usage

Installation

bash
pip install torch transformers huggingface_hub chess
# Download model.py from this repository

Basic Usage

python
import torch
from model import ChessFormerModel

# Load model
model = ChessFormerModel.from_pretrained("kaupane/ChessFormer-SL")
model.eval()

# Analyze position
fens = ["rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 0 1"]
repetitions = torch.tensor([1])

with torch.no_grad():
    move_logits, position_value = model(fens, repetitions)
    
# Get best move (requires additional processing for legal moves)
print(f"Position value: {position_value.item():.3f}")

With Chess Engine Interface

python
from engine import Engine, ChessformerConfig
import chess

# Create engine
config = ChessformerConfig(
    chessformer=model,
    temperature=0.5,
    depth=2  # Enable search enhancement
)
engine = Engine(type="chessformer", chessformer_config=config)

# Play move
board = chess.Board()
move_uci, value = engine.move(board)
print(f"Suggested move: {move_uci}, Value: {value:.3f}")

Limitations and Bias

Technical Limitations

  • —Tactical weakness: Prone to hanging pieces and missing simple tactics
  • —Computational inefficiency: FEN tokenization creates training bottlenecks, preprocess the entire dataset before training should be benefical

Potential Biases

  • —Trained exclusively on Stockfish evaluations, may inherit engine biases
  • —May not generalize to unconventional openings or endgames

Known Issues

  • —Piece embeddings have consistently lower norms than positional embeddings
  • —Model sometimes assigns probability (though unlikely, ~3%) to invalid moves despite training penalty
  • —Performance degrades without search enhancement

Ethical Considerations

This model is intended for:

  • —✅ Educational purposes and chess learning
  • —✅ Research into neural chess architectures
  • —✅ Developing chess training tools

Not recommended for:

  • —❌ Competitive chess tournaments
  • —❌ Production chess engines without extensive testing
  • —❌ Applications requiring reliable tactical calculation

Additional Information