WhySoCodius/in-context-grid-reasoning

In-Context Grid Reasoning (ICGR)

A small, fully synthetic benchmark for demonstration-conditioned rule induction: each task shows 2–4 (input grid → output grid) support pairs that share one hidden transformation, and the model must apply the same transformation to a held-out query input.

It targets the same behaviour probed by recent in-context / latent-reasoning work on ARC-AGI (e.g. BDH-CQ: In-Context Learning with Recurrent Latent Reasoning, arXiv:2608.09888), but is deliberately tiny, transparent, and license-clean so it can be used freely for quick probes and ablations.

Why this exists

ARC-AGI itself is excellent but small and easy to overfit to via public solvers. ICGR is procedurally generated from a single script (`generate.py`), so you can:

• —regenerate it deterministically (--seed),
• —scale it up (--n),
• —know exactly which rule family each task belongs to (for per-concept scoring),
• —trust its provenance — no scraped text, images, or third-party datasets, so there is no upstream copyright. Released under CC-BY-4.0.

Format

One JSON object per line.

Grids are serialised as rows of space-separated integers, rows joined by ;. Example: "1 2 0;0 1 2" is the 2×3 grid [[1,2,0],[0,1,2]].

from datasets import load_dataset

ds = load_dataset("WhySoCodius/in-context-grid-reasoning", split="test")
ex = ds[0]
print(ex["rule_description"])
for pair in ex["support"]:
    print(pair["input"], "->", pair["output"])
print("Q:", ex["query_input"], "=>", ex["query_output"])

Rule families

flip_h, flip_v, transpose, rotate90, add_mod (add constant mod colour count), color_swap, shift_rows (cyclic), tile_h (self-concat), border (paint outer ring), max_pool2 (2×2 max). ~35% of tasks compose two of the size-preserving rules; rule records which and in what order.

Splits

Split is a random shuffle at a fixed seed; the same rule family appears in both. It is a convenience split, not an adversarial generalisation split — if you need held-out rules, filter by rule.

Suggested metric

Exact string match on query_output after normalising whitespace. Report overall accuracy plus a breakdown by rule and by rule_kind.

Limitations

• —Rules are simple and enumerable; a symbolic solver can reach 100%. The point is to measure whether a learning system infers the rule from demonstrations alone, not to be unsolvable.
• —Grids are small (3×5 max, 6×6 for pooling) and dense.
• —English rule descriptions are templated.

Reproduce / extend

python generate.py --n 5000 --seed 123
python test_generate.py   # self-check

Citation

@misc{icgr2026,
  title  = {In-Context Grid Reasoning (ICGR): a synthetic benchmark for
            demonstration-conditioned rule induction},
  author = {WhySoCodius},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/datasets/WhySoCodius/in-context-grid-reasoning}}
}

License: CC-BY-4.0. Attribution appreciated; no other restrictions.