MM-SafetyBench-plus-plus

# MM-SafetyBench++ [**Project Page**](https://EchoSafe-mllm.github.io) | [**Paper**](https://huggingface.co/papers/2603.15800) | [**Code**](https://github.com/EchoSafe-MLLM/EchoSafe) MM-SafetyBench++ is a benchmark designed for evaluating **contextual safety** in Multi-Modal Large Language Models (MLLMs). It challenges models to distinguish subtle contextual differences between scenarios that may appear visually or textually similar but diverge significantly in safety intent. ## Dataset Summary For each unsafe image-text pair, the benchmark includes a corresponding safe counterpart created through minimal modifications that flip the user intent while preserving the underlying contextual meaning. This enables a controlled evaluation of whether models can adapt their safety behaviors based on deep contextual understanding rather than just relying on keyword triggers or common visual patterns. The dataset consists of 2,844 test samples across several harm categories, including: - Illegal Activity - Hate Speech - (and others) This benchmark was introduced in the paper [Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory](https://huggingface.co/papers/2603.15800). ## Citation If you find this dataset useful, please consider citing the following work: ```bibtex @article{zhang2026evolving, title={Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory}, author={Zhang, Ce and He, Jinxi and He, Junyi and Sycara, Katia and Xie, Yaqi}, journal={arXiv preprint arXiv:2603.15800}, year={2026} } ```