jub-aer/Absurd-World

<div align="center">

<img src="absurd.png" alt="Absurd Soccer Logo" width="200"/>

<h1>Absurd World: <em> A Simple Yet Powerful Method to Absurdify the Real-world for Probing LLM Reasoning Capabilities </em> </h1>

When the rules of reality change, even in trivial ways, can AI adapt as seamlessly as humans do? <br>

  

Try it now:


</div>

Absurd Soccer Dataset

Dataset Description

Absurd Soccer is a reasoning benchmark dataset designed to evaluate how Large Language Models adapt to modified world models. The dataset tests whether LLMs can maintain reasoning capabilities when familiar rules are systematically altered in "absurd" but internally consistent ways.

Dataset Summary

This dataset contains game commentaries and questions about soccer matches with modified rulesets. Each instance requires models to:

1. 1.Understand the modified ruleset
2. 2.Track game state through commentary
3. 3.Determine the winning team based on absurd rules

Why This Matters: When rules of reality change (even trivially), can AI adapt as seamlessly as humans do?

Supported Tasks

• —Reasoning Evaluation: Assess model ability to adapt to modified world models
• —Rule Following: Test adherence to explicitly stated but counter-intuitive rules
• —Question Answering: Determine game outcomes based on commentary

Task Variants

The dataset includes 4 different task types:

Languages

• —English (en)

Dataset Structure

Data Instances

Each instance contains:

• —game #: Unique identifier for the game
• —task: Task type (DO, DO-FS, DO-CoT)
• —ruleset: One of 7 rule variants (REAL, MISSING, LEAST, ICE CREAM, CAR, SWITCH, MISS & SWITCH)
• —prompt: Game commentary and question
• —answer: Correct answer (Team A or Team B or Both Teams)
• —Additional metadata fields

Data Fields

Data Splits

Rulesets

The dataset includes 7 systematically modified rulesets:

Dataset Creation

Curation Rationale

Created to investigate whether LLMs can reason effectively when world models deviate from training data. Soccer was chosen because:

• —Universal familiarity across cultures
• —Simple, deterministic domain
• —Allows systematic rule variations
• —Humans solve these tasks effortlessly (small tasks)

Source Data

Initial Data Collection

Games were procedurally generated with:

• —5 matches per game
• —2 teams alternating shots
• —Deterministic outcomes based on rules
• —Varied shot success rates

Who are the source language producers?

Synthetically generated using rule-based systems. Commentary patterns designed to be clear and unambiguous.

Annotations

Annotation process

Answers are deterministically computed based on:

1. 1.Shot outcomes (hit/miss)
2. 2.Score accumulation rules
3. 3.Win condition evaluation

No human annotation required - all labels are algorithmically verified.

Personal and Sensitive Information

None. Dataset contains only synthetic game commentaries with fictional teams.

Considerations for Using the Data

Social Impact of Dataset

Positive Impacts:

• —Advances understanding of LLM reasoning limitations
• —Provides benchmark for cognitive flexibility
• —Low-stakes domain (soccer games) avoids harmful applications

Potential Concerns:

• —Models that fail these tasks may be deployed in critical reasoning scenarios
• —Highlights brittleness of LLM world models

Citation

@misc{albright2026absurdworldsimplepowerful,
      title={Absurd World: A Simple Yet Powerful Method to Absurdify the Real-world for Probing LLM Reasoning Capabilities}, 
      author={Ryan Albright and Golam Md Muktadir and Zarif Ikram and S M Jubaer and Mehrab Hossain and Dianbo Liu},
      year={2026},
      eprint={2605.09678},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2605.09678}, 
}