HankYang428/GENIUS

<div align="center">

🧠 GENIUS

Generative Fluid Intelligence Evaluation Suite

   

[Paper] &nbsp; [Code] &nbsp; [Blog] &nbsp; [Dataset]

</div>

[image]

Leaderboard

GENIUS evaluates every generated image on three complementary axes, each scored 0 (fail), 1 (partial), or 2 (perfect):

• —Rule Compliance (RC): follows the newly defined rule, grounded by expert-written evaluation hints.
• —Visual Consistency (VC): preserves required identities, objects, and contextual visual attributes.
• —Aesthetic Quality (AQ): remains visually coherent and avoids obvious generation artifacts.

The Overall score uses the paper's weighted metric ratio RC:VC:AQ = 6:3.5:0.5. The primary Gemini score averages three independent judge runs per sample. Qwen2.5-VL-72B is an independent robustness judge: it assigns lower absolute scores overall, while preserving the main relative performance trends.

GENIUS leaderboard judged by Gemini-3-Pro:

[image]

What is GENIUS?

Most image-generation benchmarks primarily test crystallized intelligence: whether a model can retrieve and reproduce concepts learned during pre-training. GENIUS instead evaluates Generative Fluid Intelligence (GFI)—the ability to solve visual-generation problems whose rules are defined entirely by the immediate multimodal context.

GENIUS contains 510 expert-curated test cases with interleaved text and multi-image inputs. Removing either modality makes an instance unsolvable. The benchmark spans three core capabilities:

Dataset composition

Each record contains:

Image placeholders use the form <image:NAME> and resolve to CONFIG/images/NAME.png. For example, <image:space_0> in the implicit_pattern config refers to implicit_pattern/images/space_0.png.

Viewer representation. The five viewer/*.jsonl files are generated from the canonical */test_data.json annotations. They preserve all annotation text while normalizing the occasionally mixed string/list representation of vc_hint into a nullable string, giving the Dataset Viewer a stable schema. Rebuild them with python scripts/build_viewer_data.py after changing the source annotations.

Quick start

Load any of the five task configs directly with 🤗 Datasets:

from datasets import load_dataset

dataset = load_dataset(
    "HankYang428/GENIUS",
    "implicit_pattern",  # or symbolic_constraint, visual_constraint,
                         # prior_conflicting, multi_semantic
    split="test",
)

print(dataset[0])

Download the full repository, including all reference images:

git lfs install
git clone https://huggingface.co/datasets/HankYang428/GENIUS

Alternative mirrors: Google Drive · Baidu Netdisk (password: iek1)

For model-output layout and evaluation commands, see the official code repository.

Intended use and limitations

GENIUS is designed for evaluating multimodal image-generation and image-editing systems on novel, context-defined problems. It is a compact evaluation suite, not a training corpus. Scores measure performance on the benchmark's curated rules and judge protocol; they should not be interpreted as a complete measure of general intelligence or image-generation quality. LMM-as-a-judge evaluation can retain model-specific biases, which is why the paper reports results from two independent judge families and uses expert-written gold hints.

License

GENIUS is released under the Creative Commons Attribution-NonCommercial 4.0 International License and is intended for non-commercial research use.

Citation

If GENIUS is useful in your research, please cite:

@misc{an2026geniusgenerativefluidintelligence,
  title         = {GENIUS: Generative Fluid Intelligence Evaluation Suite},
  author        = {Ruichuan An and Sihan Yang and Ziyu Guo and Wei Dai and Zijun Shen and Haodong Li and Renrui Zhang and Xinyu Wei and Guopeng Li and Wenshan Wu and Wentao Zhang},
  year          = {2026},
  eprint        = {2602.11144},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2602.11144}
}

Contact

Questions and feedback are welcome via GitHub Issues or arctanxarc@gmail.com.