user66666/IS_Bench_dataset

--- license: apache-2.0 --- # Dataset Card for IS-Bench This dataset is for paper IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks ⭐ You can check our [Paper](https://www.arxiv.org/abs/2506.16402), [Github](https://github.com/AI45Lab/IS-Bench), [Project Page](https://ursulalujun.github.io/isbench.github.io/) for more information. ## Usage - Online Evaluation: Download scenes.tar.gz and load these scene files in Omnigibson simulator. - Offline Evaluation: Download scene_images.tar.gz and use the scene images in it as input directly. Please see [IS-Bench code](https://github.com/AI45Lab/IS-Bench) for more details. ## Dataset Details Our dataset statistics are listed in the following: <img src="https://github.com/AI45Lab/IS-Bench/blob/main/assets/statistics.png?raw=true"/> Here are examples in our dataset: <img src="https://github.com/AI45Lab/IS-Bench/blob/main/assets/example.png?raw=true"/> The evaluation results on leading VLMs. (SR: Success Rate, SSR: Safe and Success Rate, Srec: Safety Recall, Ll: implicit safety reminder, L2: safety CoT reminder configurations.) <img src="https://github.com/AI45Lab/IS-Bench/blob/main/assets/results2.png?raw=true"/> ## Citation ```bibtex @article{lu2025bench, title={IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks}, author={Lu, Xiaoya and Chen, Zeren and Hu, Xuhao and Zhou, Yijin and Zhang, Weichen and Liu, Dongrui and Sheng, Lu and Shao, Jing}, journal={arXiv preprint arXiv:2506.16402}, year={2025} } ```