Perle-ai/T2I-benchmark

T2I Benchmark: Frontier Text-to-Image Models on Image Description Prompts

Accompanying dataset for Benchmarking Frontier Text-to-Image Models on the Image Description Prompts (Perle AI).

Four frontier text-to-image systems are compared on the 48 hardest prompts in the DataSeeds.AI Sample Dataset (DSD). Every prompt is a verbatim, human-written description of a real photograph — no prompt engineering. Every generated image is graded against a per-prompt weighted rubric by an independent-judge pipeline in which the model that writes the rubric never scores it.

This is a strict matched-N comparison: all four systems are scored on the identical 48 prompts.

Benchmark construction

1. 1.Complexity scoring. All ~7,000 valid rows of the DSD training split were scored 1–10 for generation difficulty by an automated classifier, weighing factors such as exact object counting, legible text rendering, spatial binding, conditional logic, and attribute binding.
2. 2.Hardest-48 selection. The highest-complexity prompts were kept; ties at the ceiling score were broken by a seeded random shuffle before a stable sort. The generation prompt is the verbatim concatenation of the DSD image_description and scene_description fields.
3. 3.Generation. Each prompt was submitted identically to all four systems via their public APIs — single sample, default settings, no negative prompts, no seed, no best-of-k.
4. 4.Rubric writing. For each prompt, GPT-5.4-Pro wrote a weighted rubric from the prompt text alone: positive criteria (what the prompt requests) and negative criteria (failure modes to avoid), each with an integer weight.
5. 5.Independent scoring. Gemini 3.1 Pro Preview scored each generated image against that rubric with vision input. Each criterion gets a met/triggered verdict, a failure code, and a written explanation. final_score normalizes earned positive weight against incurred negative weight onto 0–100.
6. 6.Matched-N aggregation. Only prompts where all four systems both generated and graded successfully are included (N = 48). Prompts declined by a provider's content-moderation system were excluded rather than retried or circumvented.

Failure-code taxonomy

MISCOUNT · TEXT_GARBLING · SPATIAL_RELATION_ERROR · ATTRIBUTE_BINDING_ERROR · MISSING_STEP · CONSTRAINT_BREACH · FORMAT_VIOLATION · INCOMPLETE_TRAJECTORY · GEOMETRY_ARTIFACT · HALLUCINATED_OBSERVATION · PHYSICS_VIOLATION

Positive-axis codes mark a request the image failed to satisfy; negative-axis codes mark an active defect the image introduced.

Results

Identical 48 prompts, all four systems. Net score is the mean unnormalized weighted balance (earned − incurred); unmet criteria and active errors are mean counts per prompt.

Top failure codes on the matched-48 set

Legible text rendering is the single most common active defect for every system, and exact counting is the most common missed request for the two leaders.

Configs

prompts — 48 rows

evaluations — 192 rows (48 prompts × 4 models)

One fully graded rubric per model per prompt: 2,658 individual criterion verdicts in total.

Usage

from datasets import load_dataset

prompts = load_dataset("Perle-ai/T2I-benchmark", "prompts", split="test")
evals   = load_dataset("Perle-ai/T2I-benchmark", "evaluations", split="test")

# Reproduce the leaderboard
import collections
scores = collections.defaultdict(list)
for e in evals:
    scores[e["model_display_name"]].append(e["final_score"])
for name, s in sorted(scores.items(), key=lambda kv: -sum(kv[1]) / len(kv[1])):
    print(f"{name:40s} N={len(s)}  {sum(s) / len(s):.1f}")

# Failure-code frequencies for one model
codes = collections.Counter(
    c["failure_code"]
    for e in evals if e["model_key"] == "hunyuan3"
    for c in e["negative_criteria"] if c["triggered"] and c["failure_code"]
)
print(codes.most_common())

Reference photographs for every prompt are available in the source DSD dataset by image_id.

Evaluated systems

Gemini 3 Pro Image (Nano Banana Pro) · Black Forest Labs FLUX.2 · Ideogram 3.0 · Hunyuan 3.0 (HunyuanImage-3.0)

Limitations

One generation per prompt per model with no fixed seed and no resampling, so scores reflect default out-of-the-box quality rather than best-of-k. Grades come from a single automated judge pair (GPT-5.4-Pro → Gemini 3.1 Pro Preview); human-SME agreement has not yet been measured. Prompt selection is deliberately biased toward maximum difficulty, so absolute scores are not comparable to general-purpose prompt sets, and N = 48 bounds the resolution of small between-model gaps.

License and provenance

Prompts are derived verbatim from the DataSeeds.AI Sample Dataset (DSD), released under Apache-2.0. Rubrics, criterion verdicts, scores, and metadata in this repository are released under Apache-2.0.

Citation

@misc{perle2026t2ibenchmark,
  title        = {Benchmarking Frontier Text-to-Image Models on the Image Description Prompts},
  author       = {Perle AI},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/Perle-ai/T2I-benchmark}}
}

Please also cite the source dataset:

@misc{dataseeds2025,
  title  = {Peer-Ranked Precision: Creating a Foundational Dataset for Fine-Tuning Vision Models from GuruShots' Annotated Imagery},
  author = {Abdoli, Sajjad and others},
  year   = {2025},
  eprint = {2506.05673},
  archivePrefix = {arXiv}
}