konpat/visual-jenga-datasets

Visual Jenga Datasets

This directory contains the original datasets for Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting. Visual Jenga is a novel scene understanding task that involves progressively removing objects from a single image one at a time while keeping the rest of the scene stable. This process reveals object dependencies and provides a new way to evaluate grounded scene understanding by systematically exploring which objects can be removed while preserving scene coherence in both physical and geometric sense.

Datasets Overview

1. ClutteredParse

• —Directory: clutteredparse/
• —Samples: 40 scenes
• —Structure: Each subdirectory contains:
• —img.* - Original scene image (various formats: jpg, jpeg, webp)
• —A.png - Mask showing objects to remove (before)
• —B.png - Mask showing objects after removal (after)
• —Visualization: dataset_clutteredparse.html

2. COCO

• —Directory: coco/
• —Samples: 200 scenes
• —Structure: Each subdirectory (numbered 000-199) contains:
• —img.* - Original scene image (jpeg format)
• —A.png - Mask showing objects to remove (before)
• —B.png - Mask showing objects after removal (after)
• —Visualization: dataset_coco.html

3. Full Scene Decom

• —Directory: full_scene_decom/
• —Samples: 56 images
• —Structure: Flat directory with sequentially numbered images (00-55) in various formats (jpg, png, jpeg, webp, avif)
• —Purpose: Scene decomposition visualization showing progressive scene breakdown
• —Visualization: dataset_full_scene_decom.html

4. NYU

• —Directory: nyu/
• —Samples: 668 scenes
• —Structure: Each subdirectory contains:
• —img.* - Original scene image (jpg format)
• —A.png - Mask showing objects to remove (before)
• —B.png - Mask showing objects after removal (after)
• —scene_graph.jpg or scene_graph.png - Scene graph visualization
• —Visualization: dataset_nyu.html

File Structure

originals/
├── clutteredparse/          # ClutteredParse dataset
│   ├── 01/
│   │   ├── img.jpg
│   │   ├── A.png
│   │   └── B.png
│   ├── 02/
│   └── ...
├── coco/                     # COCO dataset
│   ├── 000/
│   │   ├── img.jpeg
│   │   ├── A.png
│   │   └── B.png
│   ├── 001/
│   └── ...
├── full_scene_decom/         # Full Scene Decom dataset
│   ├── 00.jpg
│   ├── 01.png
│   └── ...
├── nyu/                      # NYU dataset
│   ├── 10_0/
│   │   ├── img.jpg
│   │   ├── A.png
│   │   ├── B.png
│   │   └── scene_graph.jpg
│   └── ...
├── dataset_clutteredparse.html
├── dataset_coco.html
├── dataset_full_scene_decom.html
├── dataset_nyu.html
└── README.md

Visualizations

Interactive HTML visualizations are available for each dataset:

• —[ClutteredParse Visualization](dataset_clutteredparse.html) - Browse all 40 scenes with before/after masks
• —[COCO Visualization](dataset_coco.html) - Browse all 200 scenes with before/after masks
• —[Full Scene Decom Visualization](dataset_full_scene_decom.html) - Browse all 56 scene decomposition images
• —[NYU Visualization](dataset_nyu.html) - Browse all 668 scenes with before/after masks and scene graphs

Dataset Statistics

License

This dataset combines images and annotations from multiple sources with different licensing terms. Users must comply with the original licenses of each source:

Source Licenses

1. 1.COCO Dataset
2. 2.Licensed under Creative Commons Attribution 4.0 (CC BY 4.0)
3. 3.Reference: COCO Dataset

1. 1.NYU Depth V2 Dataset
2. 2.Reference: NYU Depth Dataset V2
3. 3.Citation: Please cite: "Indoor Segmentation and Support Inference from RGBD Images" (ECCV 2012) by Silberman et al.

1. 1.ClutteredParse & Full Scene Decom
2. 2.Images sourced from the internet with no guaranteed license
3. 3.These images may be subject to copyright protection

Disclaimer: This dataset is provided "as-is" without warranty. Users are solely responsible for ensuring their use complies with all applicable licenses and copyright laws.