QuantumWhisper42/anonymous_dataset

KnowVis: A Dual-View Benchmark for Diagnosing World-Knowledge Grounding in Text-to-Image Models

📖 Overview

Text-to-image (T2I) models have made substantial progress in visual realism, aesthetic quality, and instruction following. However, real-world prompts often go beyond explicit visual descriptions and require implicit facts, structured knowledge, and domain-specific commonsense. Existing evaluations mainly focus on explicit prompt-to-image semantic alignment or specific knowledge subdomains, lacking a unified and diagnostic framework for evaluating world-knowledge grounding in T2I models. We introduce KnowVis, a dual-view diagnostic benchmark for world-knowledge grounding in T2I generation. KnowVis is built on a hierarchical knowledge taxonomy with case-level annotations, and contains two complementary views: a Skill-Tree Benchmark for fine-grained atomic knowledge evaluation and a Generalized Benchmark for open-ended knowledge composition and expression. We further design a shared MLLM-based VQA judge protocol for reliable evaluation and error attribution. The protocol uses Required VQAs to assess atomic knowledge correctness in the Skill-Tree Benchmark, and combines Required, Optional, and dynamically discovered knowledge expressions with human-judge collaborative evaluation in the Generalized Benchmark. Experiments show that KnowVis reveals notable gaps in world-knowledge grounding across existing T2I models, providing a unified basis for evaluating and improving knowledge-intensive image generation.

📂 File Structure

.
├── assets/                   # Skill-Tree test and val input images (492 cases; some cases produce multi-image outputs)
├── data
│   ├── all_en.jsonl          # All prompts (English)
│   ├── all_zh.jsonl          # All prompts (Chinese)
│   ├── common_en.jsonl       # Common T2I prompts — English (100 cases)
│   ├── common_zh.jsonl       # Common T2I prompts — Chinese (100 cases)
│   ├── generalized_en.jsonl  # Generalized Benchmark — English (550 cases)
│   ├── generalized_zh.jsonl  # Generalized Benchmark — Chinese (550 cases)
│   ├── skill_tree_en.jsonl   # Skill-Tree Benchmark — English (1,622 cases)
│   ├── skill_tree_zh.jsonl   # Skill-Tree Benchmark — Chinese (1,622 cases)
│   ├── test_gt_en.jsonl      # Skill-Tree test split w/ human annotations — English (240 cases)
│   ├── test_gt_zh.jsonl      # Skill-Tree test split w/ human annotations — Chinese (240 cases)
│   ├── val_gt_en.jsonl       # Skill-Tree val split w/ human annotations — English (240 cases)
│   └── val_gt_zh.jsonl       # Skill-Tree val split w/ human annotations — Chinese (240 cases)
├── README-zh.md              # README (Chinese)
├── README.md                 # README (English)
└── taxonomy_tree.json        # Full taxonomy tree structure (English and Chinese)

Note on language variants: Each _en / _zh pair contains the same cases — the Chinese files are translations of the English originals. All splits and annotations are identical across both languages.

📊 Dataset Statistics

🗂️ Data Format

All files are in JSON Lines format — one JSON object per line.

Common (type: "common")

General-purpose prompts with no structured knowledge annotation.

{
  "type": "common",
  "case_id": "06cd7d68-daca-5855-b2a1-60edcc9bd3e3",
  "prompt": "Two Chinese fashionable women are walking on a commercial street, each holding balloons shaped like the OK gesture."
}

Generalized (type: "generalized")

Prompts requiring knowledge composition. Each case includes a structured VQA checklist and a detailed chain-of-thought description for evaluation and reference.

{
  "type": "generalized",
  "case_id": "57ccd75e-012f-534c-890e-977736ec9601",
  "prompt": "A construction worker prying up a stone with a wooden board, lever principle, fulcrum, lever arm, and heavy object clearly labeled.",
  "category": ["Natural Science", "Physics", "Mechanics", "Torque Equilibrium"],
  "vqa_checklist": [
    {
      "question": "There is a person whose appearance presents as a construction worker in the image.",
      "knowledge_hint": "...",
      "component": "ENTITY",
      "eval_type": "VISUAL",
      "judge_type": "REQUIRED",
      "has_text": false,
      "weight": 3
    },
    ...
  ],
  "cot_prompt": "A construction site scene. In the center of the image, there is a massive stone ..."
}

`vqa_checklist` item fields:

Skill-Tree (type: "skill_tree")

Fine-grained prompts targeting atomic knowledge units, sharing the same structure as Generalized cases.

{
  "type": "skill_tree",
  "case_id": "ac02e004-1fcd-5206-87c5-8644c24a394f",
  "prompt": "Show the real experimental phenomenon of a magnesium ribbon held by crucible tongs burning in the air.",
  "category": ["Natural Science", "Chemistry", "Chemical Reaction Phenomena", "Reactions of Metals with Oxygen"],
  "labels": {
    "education_level": "Junior High School",
    "knowledge_operation_type": "Explicit Knowledge Recall"
  },
  "vqa_checklist": [ ... ],
  "cot_prompt": "..."
}

Skill-Tree cases carry an additional labels field with structured metadata:

🚀 Usage

We strongly recommend using the official codebase on GitHub for loading, evaluating, and reproducing results with this dataset. The repository includes helper utilities, the MLLM-based VQA judge, evaluation scripts, and the full pipeline:

👉 [GitHub Repository](https://github.com/QuantumWhisper42/anonymous_project)

import json

def load_jsonl(path):
    with open(path, "r", encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]

# Load individual subsets
common    = load_jsonl("common_en.jsonl")
generalized = load_jsonl("generalized_en.jsonl")
skill_tree  = load_jsonl("skill_tree_en.jsonl")

# Load the full benchmark
all_cases = load_jsonl("all_en.jsonl")

# Load human-annotated evaluation splits
test_gt = load_jsonl("test_gt_en.jsonl")
val_gt  = load_jsonl("val_gt_en.jsonl")

For loading, prompt generation, VQA judging, and full evaluation support, please refer to the GitHub repository above.

📜 License

This dataset is released under the Creative Commons Attribution 4.0 International License (CC BY 4.0).

Under the Creative Commons definition of a *Collection*, this license applies to the dataset as a whole — its structure, annotations, prompts, and metadata — and does not extend to the individual informational items it references. Facts and knowledge referenced within the dataset are not subject to copyright.

This dataset includes links to resources from Wikimedia Commons. No such resources are distributed with this dataset — only URLs are collected. The title, author, source URL, and license of each referenced resource are recorded in the visual_grounding field of the corresponding entry.

🤝 Contributing

We welcome feedback and contributions. Feel free to open an issue to report problems or suggest improvements.