JustSecret/Imag-Eval

IMAG-EVAL

<p align="center"> <b>IMAG-EVAL:</b> A Language-Grounded Framework for Interpretable Text-to-Image Instruction Following Evaluation </p>

  

🚀 Overview

IMAG-EVAL is a controlled benchmark designed to evaluate instruction-following capabilities in Text-to-Image (T2I) generation models.

Unlike prior benchmarks that primarily rely on prompt length or isolated skill evaluation, IMAG-EVAL explicitly disentangles linguistic complexity from compositional complexity by independently varying:

• —the number of grounded object instances,
• —the number of compositional constraints,
• —and the combination of evaluated skills.

The benchmark provides interpretable diagnostic evaluations and enables fine-grained analysis of failure modes in modern Text-to-Image systems.

📊 Benchmark Statistics

Evaluated Skills

• —🔢 Counting: Generating the exact number of requested object instances.
• —🎨 Color Attribution: Correctly assigning the specified color to each relevant instance.
• —📏 Size Relations: Preserving the required relative size constraints between instances (e.g., larger than, smaller than).
• —📍 Spatial Relations: Correctly grounding spatial relationships between instances (e.g., A is to the right of B).
• —😊 Emotion Attribution: Accurately rendering the specified facial expressions and emotions.
• —🔤 Text Rendering: Faithfully reproducing the target text, including prompts containing spelling variations or typographical errors.
• —🧩 Cohesiveness: Generating anatomically plausible, structurally consistent, and visually coherent instances and scenes.

📂 Repository Structure

prompts/

Contains the benchmark prompt definitions used during evaluation.

Each JSON file includes:

• —prompt metadata,
• —skill combinations,
• —difficulty levels,
• —scene specifications,
• —synthetic prompts,
• —evaluation constraints.

skill_codes.csv

Mapping between benchmark identifiers and skill combinations.

Example:

These codes are used throughout the benchmark generation and evaluation pipeline.

texts.csv

Reference textual annotations used for Text Rendering evaluation.

Each row contains:

• —prompt identifier,
• —expected textual content,
• —corresponding benchmark reference.

This file is used to compute Word Error Rate (WER) during evaluation.

🎯 Supported Tasks

IMAG-EVAL is intended for:

• —Text-to-Image Generation
• —Instruction Following
• —Compositional Reasoning
• —Multimodal Grounding
• —Text Rendering Evaluation
• —Benchmarking and Diagnostic Analysis

📈 Evaluation Protocol

Models are evaluated on seven dimensions:

⚠️ Limitations

• —The benchmark relies on COCO object categories.
• —Emotion attribution is currently restricted to human instances.
• —Prompts are written in English.

🤗 Related Resources

• —📄 Paper [EMNLP 2026]: *IMAG-EVAL: A Language-Grounded Framework for Interpretable Text-to-Image Instruction Following Evaluation*
• —💻 Code: https://github.com/Justsecret123/Imag-Eval

📚 Citation (pending EMNLP 2026 proceedings)

@misc{serouis2026imageval,
  title={IMAG-EVAL: A Language-Grounded Framework for Interpretable Text-to-Image Instruction Following Evaluation},
  author={Serouis, Ibrahim Mohamed and Jaramillo Duque, David},
  year={2026},
  note={Accepted at the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026)}
}

📜 License

Please refer to the repository license for terms of use.