RyanWW/Super-CLEVR

Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning

[CVPR 2023 Highlight (top 2.5%)]

Paper: Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning

Authors: Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, Alan Yuille

Dataset Description

Super-CLEVR is a synthetic dataset designed to systematically study the domain robustness of visual reasoning models across four key factors:

• —Visual complexity — varying levels of scene and object complexity
• —Question redundancy — controlling redundant information in questions
• —Concept distribution — shifts in the distribution of visual concepts
• —Concept compositionality — novel compositions of known concepts

Dataset

Super-CLEVR contains 30k images of vehicles (from UDA-Part) randomly placed in the scenes, with 10 question-answer pairs for each image. The vehicles have part annotations and so the objects in the images can have distinct part attributes.

Here [[link]](https://www.cs.jhu.edu/~zhuowan/zhuowan/SuperCLEVR/objpartlist/all_objects.html) is the list of objects and parts in Super-CLEVR scenes.

The first 20k images and paired are used for training, the next 5k for validation and the last 5k for testing.

The dataset is available on Hugging Face:

Dataset Viewer

The interactive Dataset Viewer is served from flattened Parquet files under viewer/ (train.parquet, validation.parquet, test.parquet), exposed as the questions config with three splits (200k / 50k / 50k questions, derived from image index per the train/val/test partition above). These contain the question-level fields (question, answer, image_filename, image_index, template_filename, question_family_index, question_index); the full functional program annotations and scene graphs remain in the canonical superCLEVR_questions_30k*.json and superCLEVR_scenes.json files.

from datasets import load_dataset

ds = load_dataset("RyanWW/Super-CLEVR", "questions", split="validation")
print(ds[0]["question"], "->", ds[0]["answer"])

Usage

from huggingface_hub import hf_hub_download

# Download a specific file
path = hf_hub_download(
    repo_id="RyanWW/Super-CLEVR",
    filename="superCLEVR_questions_30k.json",
    repo_type="dataset",
)

Citation

@inproceedings{li2023super,
  title={Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning},
  author={Li, Zhuowan and Wang, Xingrui and Stengel-Eskin, Elias and Kortylewski, Adam and Ma, Wufei and Van Durme, Benjamin and Yuille, Alan L},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={14963--14973},
  year={2023}
}

Links

• —Code: github.com/Lizw14/Super-CLEVR
• —Paper: arxiv.org/abs/2212.00259

License

This dataset is released under the MIT License.