user66666/IS_Bench_dataset

Dataset Card for IS-Bench

This dataset is for paper IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks

⭐ You can check our Paper, Github, Project Page for more information.

Usage

• —Online Evaluation: Download scenes.tar.gz and load these scene files in Omnigibson simulator.

• —Offline Evaluation: Download scene_images.tar.gz and use the scene images in it as input directly.

Please see IS-Bench code for more details.

Dataset Details

Our dataset statistics are listed in the following:

<img src="https://github.com/AI45Lab/IS-Bench/blob/main/assets/statistics.png?raw=true"/>

Here are examples in our dataset:

<img src="https://github.com/AI45Lab/IS-Bench/blob/main/assets/example.png?raw=true"/>

The evaluation results on leading VLMs. (SR: Success Rate, SSR: Safe and Success Rate, Srec: Safety Recall, Ll: implicit safety reminder, L2: safety CoT reminder configurations.)

<img src="https://github.com/AI45Lab/IS-Bench/blob/main/assets/results2.png?raw=true"/>

Citation

@article{lu2025bench,
  title={IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks},
  author={Lu, Xiaoya and Chen, Zeren and Hu, Xuhao and Zhou, Yijin and Zhang, Weichen and Liu, Dongrui and Sheng, Lu and Shao, Jing},
  journal={arXiv preprint arXiv:2506.16402},
  year={2025}
}