amd/TTT-Bench

1---2language:3- en4license: apache-2.05task_categories:6- question-answering7library_name:8- datasets9configs:10- config_name: oTTT11  data_files:12  - split: test13    path: ttt_puzzles_test.json14- config_name: dTTT15  data_files:16  - split: test17    path: dttt_puzzles_test.json18- config_name: cTTT19  data_files:20  - split: test21    path: cube_puzzles_test.json22- config_name: sTTT23  data_files:24  - split: test25    path: stones_puzzles_test.json26---27 28# TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games29 30📃 [Paper](https://arxiv.org/abs/2506.10209) | 🤗 [Dataset](https://huggingface.co/datasets/amd/TTT-Bench) | 🌐 [Website](https://prakamya-mishra.github.io/TTTBench/)31 32We introduce **TTT-Bench**, a new benchmark specifically created to evaluate the reasoning capability of LRMs through a suite of **simple and novel two-player Tic-Tac-Toe-style games**.33Although trivial for humans, these games require basic strategic reasoning, including predicting an opponent's intentions and understanding spatial configurations.34TTT-Bench is a simple and diverse benchmark consisting of four types of two-player games, namely **oTTT** (ordinary TTT), **dTTT** (double TTT), **cTTT** (TTT in cube), and **sTTT** (TTT with squares) (see Figure below for examples), and the questions in the benchmark test the reasoning capability of LRMs by posing the question of predicting the best move by a player. 35The objectives of these games are not only simple but are also introduced here for the first time (except oTTT), with no documented prior literature available online, making our benchmark uncontaminated and highly reliable.36 37<div align="center">38<img src="TTTBench_hf.png" style="object-fit: contain;"/>39<em><b>Figure 1:</b> Visualization of four game types in TTT-Bench (oTTT, dTTT, cTTT, and sTTT). For each game, the illustration consists of the board position annotations (above), the current game state with Alice's (white) and Bob's (black), and the correct next move (represented by a check mark).</em>40</div>41 42Briefly, for each task in TTT-Bench, two players play a strategic game on a configuration dependent on the task to reach a winning state that satisfies a task-dependent winning constraint, and the player who reaches the winning state first wins the game. All the problems in this benchmark are formulated as a text-based board game question that starts with the description of the game, as well as the current game state with all the moves played by both players. The final TTT-Bench task questions are formulated to predict the next best move position by the current player, like: `Where should Bob play next?`.43 44<div align="center">45<img src="TTTBench_test_set_dist_hf.png" style="object-fit: contain;" width="50%"/>46<em><b>Figure 2:</b> Distribution of TTT-Bench question solution types. <b>Above:</b> Distribution of questions and their game verdicts (Win, Blocked, or Fork) based on how the game concludes after the corresponding solution move is played. <b>Below:</b> Distribution of questions and their solution types (single answer or multiple next-best moves).</em>47</div>48 49The final TTT-Benchtest set consists of ~100 samples from each game. As shown in Figure above, not only is this dataset diverse across different game verdicts, but it also consists of questions with both single and multiple next-best move solutions. Overall, the final TTT-Bench benchmark consists of 412 questions from four simple two-player TTT games.50 51# Dataset Details52 53The benchmark dataset consists of four splits corresponding to the four types of games, where in each split the dataset columns represents:54 55- `problem`: Prompt question for the game.56- `solution`: Ground truth next best move solution(s).57- `is_unique_solution`: Boolean indicating whether there exists a single or multiple next best move solution(s).58- `verdict`: Game verdict for the ground truth solution.59- `currect_board_visualization`: Game board ASCII visualization with played moves.60- `board_visualization`: Game board ASCII visualization with position labels.61- `problem_number`: Problem ID62 63 64# License65 66For fostering future research efforts in evaluating the reasoning capability of LRMs, we release this benchmark under the `Apache 2.0 License`.67 68Copyright(c) 2025 Advanced Micro Devices,Inc. All rights reserved.69 70Licensed under the Apache License, Version 2.0 (the "License");71you may not use this file except in compliance with the License.72You may obtain a copy of the License at73 74    http://www.apache.org/licenses/LICENSE-2.075Unless required by applicable law or agreed to in writing, software76distributed under the License is distributed on an "AS IS" BASIS,77WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.78See the License for the specific language governing permissions and79limitations under the License.80 81# Citation82```83@misc{mishra2025tttbenchbenchmarkevaluatingreasoning,84      title={TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games}, 85      author={Prakamya Mishra and Jiang Liu and Jialian Wu and Xiaodong Yu and Zicheng Liu and Emad Barsoum},86      year={2025},87      eprint={2506.10209},88      archivePrefix={arXiv},89      primaryClass={cs.CL},90      url={https://arxiv.org/abs/2506.10209}, 91}92```