revision-t2i/revision-generator

REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models (ECCV 2024)

<img src="./misc/revision_genenator.png" width="80%" height="80%"/>

This is the official dataset of the REVISION framework with all the corresponding assets (i.e. objects, backgrounds, and floors).

⚒️ Requirements

REVISION requires blenderproc. Simply install it with pip:

pip install blenderproc

👁️ Single Test Run

<img src="./misc/spatial_rel.png" width="80%" height="80%"/>

To generate a two-object reference image deterministically on your own, you may invoke one of the 4 blenderproc scripts in util/. E.g., to generate a scene of 'an apple to the left of a banana' in an indoor background, you may use

blenderproc run util/blender_left_right_floor.py apple banana background/photo_studio_loft_hall_2k.hdr output/debug/ 0 0

The command above is equivalent for generating 'a banana to the right of an apple' in an indoor background.

🏃 Batched Test Run

We also provide `revision_gen_sample_t2i_comp.sh or revision_gen_sample_mscoco.sh` to synthesize a sample batch of REVISION reference images in hdf5 format. You may then visualize the reference images with:

blenderproc vis hdf5 <path_to_ref_images>/<image_name>.hdf5

🖼️ Sample rendered outputs

For convenience, we also have provided rendered outputs in PNG format for all two-object-pairs in MSCOCO or those specified in T2I-CompBench. These images are also the ones used in the RevQA Benchmark. Please find out more under the folder sample_output/ .

🤝🏼 Citation

@misc{chatterjee2024revisionrenderingtoolsenable,
      title={REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models}, 
      author={Agneet Chatterjee and Yiran Luo and Tejas Gokhale and Yezhou Yang and Chitta Baral},
      year={2024},
      eprint={2408.02231},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.02231}, 
}

💖 Acknowledgement

The floor textures and the object models are sourced and modified from sketchfab.com. The textured background assets are sourced from polyhaven.com. All assets are shared in accordance with CC-BY-4.0 License.