abatilo/sudokubench

Dataset Card for SudokuBench

Dataset Details

This dataset contains a list of sudoku puzzles and their solutions, all at varying levels of difficulty.

The difficulties are based on the number of squares (also sometimes referred to as cells) that are provided at the start of the puzzle.

The puzzles are guaranteed to have a single unique solution without any overlap.

Within a difficulty config, you will find 10,000 puzzles at every number of available cells at the start of the board.

This means that within the kids category, you will find 10,000 sudoku boards that have at least 63 filled squares. You will find 10,000 boards with 64 filled squares, etc all the way up to having 10,000 boards with 80 filled squares.

I hope that this granularity provides for a clear understanding of where models start to have problems.

• —Curated by: Aaron Batilo

Uses

Direct Use

The intended use for SudokuBench is to be able to evaluate language models on their ability to handle long context reasoning tasks.

Dataset Structure

All puzzles have the following parquet format:

• —starting_cells: How many cells are already filled (integer).
• —puzzle: The puzzle string (compact format).
• —puzzle_pretty: The puzzle string in a human-readable pretty format.
• —solution: The solution string (compact format).
• —solution_pretty: The solution string in a human-readable pretty format.

Configs

The dataset is organized into multiple parquet files grouped by difficulty thresholds, each represented as a separate config:

Lastly, there's eval, which contains the first 200 puzzles of every single difficulty from 80 already filled squares to 17 already filled squares. This is sampling of a smaller number of puzzles is much more manageable for running holistic evals, compared to running 10,000 attempts on every single difficulty level.