Datasetpublic

limberc/this-that-spatial-bench

spatial-decisions 7,305 multiple-choice decision questions over 6,525 distinct simulated states, in 15 families and two environments. Every answer is computed from the simulator, not annotated by a person and not taken from a model. That is the point of the set: on a question whose answer is derived from the rules of the environment, a disagreement is a mistake, and there is nothing to argue about. The set was built to replace a much narrower public artefact: a recording of 68… See the full description on the dataset page: https://huggingface.co/datasets/limberc/this-that-spatial-bench.

sourceHugging Facemitupdated 5d agoView on Hugging Face

0likes92downloads

Dataset Card

spatial-decisions

7,305 multiple-choice decision questions over 6,525 distinct simulated states, in 15 families and two environments. Every answer is computed from the simulator, not annotated by a person and not taken from a model. That is the point of the set: on a question whose answer is derived from the rules of the environment, a disagreement is a mistake, and there is nothing to argue about.

The set was built to replace a much narrower public artefact: a recording of 68 questions of a single shape -- "can the agent step one cell north?" -- over 17 maze windows. That recording is useful and tiny. This one keeps the computed-truth property and adds question types, environments, renderings, option counts and answer types.

What is in it

field	meaning
`id`	stable identifier, `family-index`
`family`	which generator produced the item
`environment`	`maze` or `snake`
`state`	the rendered state shown to the system
`question`	the question text
`options`	the answer options, in canonical order
`answer_index`	index into `options`
`answer`	the option text itself
`answer_distribution`	the true distribution over `options`. One-hot for the deterministic families; for `stochastic` it is the probability the environment's transition rules define
`deterministic`	`false` only for `stochastic`
`seen_in_training`	whether a question of this shape appears in the authors' training mixture
`fingerprint`	sha256 over the rendered state, question and sorted options

Families

family	env	n	options	majority label	question
`move_legal`	maze	500	2	0.50	Can the agent step one cell in a named direction?
`move_two_step`	maze	500	2	0.50	Are both steps of a two-move plan legal?
`open_count`	maze	400	5	0.25	How many of the four orthogonal neighbours are traversable?
`cell_kind`	maze	498	3	0.33	Is the agent on a dead end, a corridor, or a junction?
`only_way`	maze	415	6	0.20	Which single direction is open, if exactly one is?
`blocked_pair`	maze	500	2	0.50	Are two named directions both blocked? (a conjunction)
`onward`	maze	500	2	0.50	After a named move, is a further named move legal from the new cell?
`plan_survives`	maze	500	4	0.25	Does a three-step plan complete, and if not at which step does it fail?
`goal_bearing`	maze	496	8	0.12	In which of eight compass directions does the goal lie?
`reachable_within`	maze	500	2	0.50	Is the goal reachable within k moves?
`distance_band`	maze	500	5	0.20	Which band does the shortest-path distance fall in?
`first_move`	maze	500	4	0.25	Which single first move starts a shortest path?
`stochastic`	maze	500	2	0.50	Under a noisy actuator of stated reliability, will the executed move be safe?
`snake_safe`	snake	500	2	0.50	Does the snake survive the next move in a named direction?
`snake_food`	snake	496	8	0.12	In which of eight compass directions does the food lie?

Properties worth knowing before you use it

Every family is exactly label-balanced by construction. Items are bucketed by answer and the same number is drawn from each bucket, so the majority-label column above equals one over the option count in every row. A family whose rare answer is genuinely rare yields fewer questions rather than being topped up from the common one. A constant answer scores chance, everywhere.

One family has a real probability, not a label. In stochastic the actuator executes the intended move with probability rho and otherwise picks uniformly among the others, so the probability that the executed move is safe follows from the transition rules. answer_index is a single sampled outcome; answer_distribution is the analytic probability. Scoring accuracy against the sample has a ceiling below 1 by construction; scoring distance to answer_distribution does not, and is the more informative measurement.

2,000 of 7,305 items are in families the authors' own models were trained on, marked by seen_in_training. Results should be reported for the two halves separately. A specialist model beating a general one on a question shape it was trained for is a real product fact and a weak scientific one; they are different claims and averaging them hides both.

Contamination. Every item was checked by fingerprint against the authors' training mixtures and any verbatim match was discarded. The check is over the rendered state, question and option set.

Limitations

—Synthetic. It measures spatial and procedural decision-making in two toy environments, and nothing about natural language, world knowledge or open-ended reasoning.
—The maze and snake simulators are small. A system that has seen a lot of grid puzzles may find the deterministic families easy; the stochastic family is the one that stays hard, because it asks for a number rather than a fact.
—The English is templated, with two or three phrasings per family. It is not a test of robustness to natural phrasing.

Provenance and licence

The maze and snake simulators, and the noisy-actuator construction, derive from the open NanoJev project under the MIT licence; they are used as published. The generators, the balancing, the contamination check and this dataset are released under the MIT licence as well.