j-min/layoutbench

1---2license: mit3task_categories:4- text-to-image5language:6- en7configs:8- config_name: default9  data_files:10  - split: number_few11    path:12    - number/images/*_0-2_*.png13  - split: number_many14    path:15    - number/images/*_11-13_*.png16    - number/images/*_14-16_*.png17  - split: position_boundary18    path:19    - position/images/*_position_boundary_*.png20  - split: position_center21    path:22    - position/images/*_position_center_*.png23  - split: shape_horizontal24    path:25    - shape/images/*_H2W1_*.png26    - shape/images/*_H3W1_*.png27  - split: shape_vertical28    path:29    - shape/images/*_H1W2_*.png30    - shape/images/*_H1W3_*.png31  - split: size_tiny32    path:33    - size/images/*size_020_*.png34  - split: size_large35    path:36    - size/images/*size_090_*.png37    - size/images/*size_110_*.png38    - size/images/*size_130_*.png39    - size/images/*size_150_*.png40pretty_name: LayoutBench41---42 43# LayoutBench44 45Release of LayoutBench dataset from [Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation (CVPR 2024 Workshop)](https://layoutbench.github.io/)46 47See also [LayoutBench-COCO](https://huggingface.co/datasets/j-min/layoutbench-coco) for zero-shot evaluation on OOD layouts with real objects.48 49[[Project Page](https://layoutbench.github.io/)]50[[Paper](https://arxiv.org/abs/2304.06671)]51 52Authors: 53[Jaemin Cho](https://j-min.io),54[Linjie Li](https://www.microsoft.com/en-us/research/people/linjli/),55[Zhengyuan Yang](https://zyang-ur.github.io/),56[Zhe Gan](https://zhegan27.github.io/),57[Lijuan Wang](https://www.microsoft.com/en-us/research/people/lijuanw/),58[Mohit Bansal](https://www.cs.unc.edu/~mbansal/)59 60## Summary61 62LayoutBench is a diagnostic benchmark that examines layout-guided image generation models on arbitrary, unseen layouts. LayoutBench consists of 8K images with 1K images per task:63- `number_few`64- `number_many`65- `position_center`66- `position_boundary`67- `size_tiny`68- `size_large`69- `shape_horizontal`70- `shape_vertical`.71  72We assume that the layout-to-image generation models were trained on [CLEVR](https://cs.stanford.edu/people/jcjohns/clevr/) dataset (in-distribution). Then we evaluate the models on LayoutBench (out-of-distribution).73Below we compare CLEVR and LayoutBench examples.74 75![CLEVR vs LayoutBench](./assets/CLEVR_vs_LayoutBench.png)76 77## How was it created?78 79To disentangle spatial control from other aspects in image generation, such as generating diverse objects, LayoutBench keeps the object configurations of [CLEVR](https://cs.stanford.edu/people/jcjohns/clevr/) whose objects have 3 shapes, 2 materials, and 8 colors (48 combinations in total), and changes the spatial layouts.80Images in LayoutBench are collected in two steps:81- (1) sample scenes for each skill, where a scene82is defined by the objects and their positions83- (2) render images from the scenes with [Blender](https://www.blender.org/) simulator (2.93.13) and obtain bounding box layouts.84 85 86## Skill Details87 88We measure 4 spatial control skills (number, position, size, shape), where each skill consists of 2 OOD layout splits, i.e., in total 8 tasks = 4 skills x 2 splits. 89In total, we collect 8K images for LayoutBench evaluation, with 1K images per task.90 91### Skill 1: Number.92 93This skill involves generating images with a specified number of objects. In contrast to the ID CLEVR images with 3∼10 objects, we evaluate models on two OOD splits:94- (1) few: images with 0∼2 objects95- (2) many: images with 11∼16 objects.96 97###  Skill 2: Position.98 99This skill involves generating images with objects placed at specific positions. Different from ID CLEVR images featuring evenly distributed object position without much occlusion between objects, we design two OOD splits:100- (1) center: objects are placed at the center, thus leading to more occlusions101- (2) boundary: objects are only placed on boundaries (top/bottom/left/right).102 103###  Skill 3: Size.104 105This skill involves generating images with objects of a specified size. We construct two OOD splits:106- (1) tiny: objects with scale 2107- (2) large: objects with scale {9, 11, 13, 15}.108 109In comparison, the objects in CLEVR images have only two scales {3.5, 7}. We use 3∼5 objects for this skill, as we find that using more than this number of large objects can often obstruct the object visibilities.110 111###  Skill 4: Shape.112 113This skill involves generating images with objects of a specified aspect ratio. As the objects in CLEVR images mostly have square aspect ratios, we evaluate models with two OOD splits:114- (1) horizontal: objects in which one of the horizontal (x/y) axes are 2 or 3 times longer than the other axis, leading to object bounding boxes with an aspect ratio (width:height) of 2:1 or 3:1115- (2) vertical: objects whose vertical (z) axis are 2 or 3 times longer than horizontal (x/y) axes, resulting in object bounding boxes with an aspect ratio of 1:2 or 1:3. We use 3∼5 objects for this skill, as we find that using more than this number of objects can often obstruct the object visibilities.116 117# Use of LayoutBench118 119## 1) Train your model on CLEVR dataset120 121## 2) Evaluate your model on LayoutBench main splits (4 skills x 2 splits = 8 tasks)122 123![Eval overview](./assets/task_overview.png)124 125We test the OOD layout skills of layout-guided image generation models trained on CLEVR (ID) dataset. First, we generate images with LayoutBench (OOD) layouts. Then, we detect the objects from the generated images, and calculate the layout accuracy in average precision (AP), with an object detector. Please see [https://github.com/j-min/LayoutBench](https://github.com/j-min/LayoutBench) for evaluation guideline with pretrained DETR.126 127## 3) (optional) Fine-grained evaluation128 129As described in Sec 5.3 in the paper, we also provide fine-grained evaluation splits for each skill. Specifically, we divide the 4 skills into more fine-grained splits to cover both in-distribution (ID; CLEVR configurations) and out-of-distribution (OOD; LayoutBench configurations) examples. We sample 200 images for each split and report layout accuracy.130 131# Dataset File Structure132 133For each skill, we provide the following files:134 135- `scene files`: created for image rendering with Blender simulator. Each scene file includes the object configurations and their positions.136- `images`: rendered images from the scenes.137- `scene files in COCO format`: scene files converted into [COCO format](https://cocodataset.org/#format-data) for evaluation.138 139The dataset file structure is as follows:140 141```bash142number/143    # layout metadata for main splits (1K each)144    scenes_number_few.json145    scenes_number_many.json146 147    # (optional - for fine-grained evalutation - see Sec 5.3 in the paper for more details)148    # 200 scenes for each sub-split149    # (0-2 / 11-13 / 14-16 are parts of few/many, and 3-5 / 6-8 / 9-10 were additionally generated as CLEVR dataset has 3-10 objects, so there are 2 splits x 1000 images + 200 x 3 extra sub-splits = 2600 images in total)150    scenes_number_0-2_200.json151    scenes_number_3-5_200.json152    ....153    scenes_number_14-16_200.json154 155    scenes.json # the file that includes the whole scenes156 157    # actual images158    images/ 159        LayoutBench_val_number_0-2_000000.png160        ...161        LayoutBench_val_number_14-16_002599.png162 163    # scene files converted into COCO format for evaluation164    coco/165        # for main splits166        scenes_number_few_coco.json167        scenes_number_many_coco.json168 169        # for fine-grained analysis170        scenes_number_0-2_200_coco.json171        scenes_number_14-16_200_coco.json172 173# same structure for other skills174 175position/176 177shape/178 179size/180```181 182 183 184## Citation185 186```bibtex187@inproceedings{Cho2024LayoutBench,188  author    = {Jaemin Cho and Linjie Li and Zhengyuan Yang and Zhe Gan and Lijuan Wang and Mohit Bansal},189  title     = {Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation},190  booktitle = {The First Workshop on the Evaluation of Generative Foundation Models},191  year      = {2024},192}193```