Datasetpublic

lmarena-ai/Arena-T2I-Hard

Arena-T2I-Hard A 310-prompt stress benchmark for evaluating faithfulness (prompt-following) of text-to-image models, drawn from real, hard arena user requests — long, multi-entity prompts with attributes, spatial relations, counts, and stylistic constraints. Each prompt ships pre-decomposed into a dependency-aware DAG of yes/no questions; when scoring an image, failing a parent question zeroes out its descendants. The benchmark stays discriminative where DPG-Bench and DSG… See the full description on the dataset page: https://huggingface.co/datasets/lmarena-ai/Arena-T2I-Hard.

sourceHugging Facemitupdated 3mo agoView on Hugging Face

1likes87downloads

Dataset Card

Arena-T2I-Hard

A 310-prompt stress benchmark for evaluating faithfulness (prompt-following) of text-to-image models, drawn from real, hard arena user requests — long, multi-entity prompts with attributes, spatial relations, counts, and stylistic constraints. Each prompt ships pre-decomposed into a dependency-aware DAG of yes/no questions; when scoring an image, failing a parent question zeroes out its descendants. The benchmark stays discriminative where DPG-Bench and DSG saturate.

—🌐 Project page: https://banyuanhao.github.io/Arena-T2I-Hard-Page/
—💻 Evaluation code: https://github.com/banyuanhao/Arena-T2I-Hard
—📄 Paper: https://arxiv.org/abs/2606.31711

Files

—benchmark.json — a JSON array of 310 records.

Record schema

jsonc

{
  "prompt_id": 0,                 // 0..309
  "prompt": "Draw me an image ...",
  "questions": [
    {
      "id": 0,
      "question": "Is there a warrior in the image?",
      "depends_on": [],           // parent question ids (empty = root)
      "type": "faithfulness"      // "faithfulness" or "aesthetics"
    },
    { "id": 1, "question": "Is the warrior holding arrows?", "depends_on": [0], "type": "faithfulness" }
    // ...
  ]
}

Across the 310 prompts there are ~13.9k questions (~9.6k faithfulness, ~4.4k aesthetics). The paper's faithfulness metric uses faithfulness questions only.

Usage

python

from datasets import load_dataset

ds = load_dataset("lmarena-ai/Arena-T2I-Hard", split="test")
print(ds[0]["prompt"])
print(ds[0]["questions"][0])   # {'id': 0, 'question': '...', 'depends_on': [], 'type': 'faithfulness'}

Or grab the raw JSON file directly:

python

import json
from huggingface_hub import hf_hub_download

path = hf_hub_download("lmarena-ai/Arena-T2I-Hard", "benchmark.json", repo_type="dataset")
data = json.load(open(path))

Scoring

To score a text-to-image model on this benchmark (generate images for the 310 prompts, then run the dependency-aware checklist judge), see the evaluation code at https://github.com/banyuanhao/Arena-T2I-Hard.

License

MIT.

Citation

bibtex

@misc{ban2026arenat2ihard,
  title         = {Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist},
  author        = {Ban, Yuanhao and Xie, Tong and An, Sohyun and Hong, Yunqi and Frick, Evan and Hsu, I-Hung and Chiang, Wei-Lin and Stoica, Ion and Hsieh, Cho-Jui},
  year          = {2026},
  eprint        = {2606.31711},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2606.31711}
}