CoolFace
Datasetpublic

dh7/imagebench

ImageBench — 50 Text-to-Image Models Judged by VLMs on 192 Prompts Reproducibility dataset for imagebench.ai: the 192-prompt V1.2 benchmark, per-(model, prompt) VLM verdicts, and per-model aggregate scores for 50 text-to-image models. Live leaderboard + every generated image: https://imagebench.ai Methodology: https://imagebench.ai/methodology-v1 Reproducibility repo: https://github.com/dh7/image-bench-ai What's in this dataset File Rows Description… See the full description on the dataset page: https://huggingface.co/datasets/dh7/imagebench.

sourceHugging Facecc-by-4.0updated 27d agoView on Hugging Face
0likes84downloads
Dataset Card

ImageBench — 50 Text-to-Image Models Judged by VLMs on 192 Prompts

Reproducibility dataset for imagebench.ai: the 192-prompt V1.2 benchmark, per-(model, prompt) VLM verdicts, and per-model aggregate scores for 50 text-to-image models.

Live leaderboard + every generated image: <https://imagebench.ai> Methodology: <https://imagebench.ai/methodology-v1> Reproducibility repo: <https://github.com/dh7/image-bench-ai>

What's in this dataset

FileRowsDescription
prompts.csv64 tests × 3 variantsThe V1.2 benchmark: prompt variants, category/subcategory/difficulty, VLM judge questions, evaluation criteria, VLM-routing assignment.
verdicts.csv9,600 (50 models × 192 prompts)Per-image pass/fail verdict, VLM judge reasoning, HPS aesthetic score, VLM model used.
scores.csv50Per-model aggregate: Overall (0-100), pass rate, aesthetic Estimated Preference Score, per-category pass rates.

Column reference

verdicts.csv

  • —model_slug — canonical model identifier (matches URLs on imagebench.ai)
  • —model_name — human-readable name
  • —category / subcategory / difficulty — benchmark taxonomy
  • —prompt_id — matches image filename on imagebench.ai/gallery
  • —prompt — the actual prompt sent to the model
  • —verdict — PASS or FAIL from the VLM judge
  • —vlm — which VLM produced the verdict (Qwen 3.5 122B / Gemini 3.1 Pro / etc.)
  • —judge_response — the VLM's textual reasoning for the verdict
  • —hps_mu — HPSv3 aesthetic score (higher = judged more aesthetically pleasing by human-preference model)
  • —hps_pref — pairwise HPS preference metadata

scores.csv

  • —model_slug, model_name
  • —overall — 0-100 blended score (capability + aesthetic preference)
  • —pass_rate — % of prompts passing the VLM verdict
  • —eps — Estimated Preference Score (aesthetic, based on HPSv3)
  • —pass_<category> — pass rate per category (text rendering, spatial reasoning, human realism, truthfulness, studio quality, graphical design)

How this was produced

  1. 1.Prompts: 64 tests × 3 variants, hand-curated across 6 capability categories with binary judge questions per test.
  2. 2.Generation: each model produces 192 images (one per prompt) via its native API — no cherry-picking, no re-runs.
  3. 3.Judging: VLM-routed — Qwen 3.5 122B handles most categories, with per-category routing to specialists (Gemini 3.1 Pro for hands, etc.) based on calibration.
  4. 4.Aesthetic score: HPSv3 (Human Preference Score v3) applied to every generated image.
  5. 5.Overall: 0.5 × normalized pass rate + 0.5 × normalized EPS.

Full methodology + calibration study: <https://imagebench.ai/methodology-v1>

Known limitations

  • —Text-to-image only — no editing, inpainting, video, long-form coherence.
  • —Prompt bias — the 192-prompt set is weighted toward tests that separate models. "Every model passes" mundane prompts are under-represented, so pass rates are lower than typical real-world use.
  • —VLM judge bias — VLMs have their own systematic errors. We validate on a hands-specific calibration blog: <https://imagebench.ai/blog/hands-benchmark-qwen35-122b>
  • —Category weighting — Overall gives each category equal weight, but "text rendering" is inherently harder to pass than "professional studio quality". Interpret cross-category comparisons carefully.

Citation

If you use this dataset in research or writing, please cite:

ImageBench: 50 Text-to-Image Models Judged by VLMs on 192 Prompts
https://imagebench.ai

License

CC-BY 4.0 — free to use with attribution.

Feedback

Issues, corrections, or methodology critique: open an issue at <https://github.com/dh7/image-bench-ai> or leave a comment on this dataset.