MM-Hallu/MAD-Bench

MAD-Bench是一个用于评估多模态大型语言模型（MLLM）对包含错误信息的欺骗性提示的脆弱性的基准数据集。包含957个测试样本，覆盖5个类别，使用GPT-4o作为评判标准进行评估。数据集包含图像（来自COCO、Flickr和网络来源）、包含错误信息的欺骗性提示以及所属的欺骗类别。五个类别分别是：不存在的物体（748个样本，提示关于图像中不存在的物体）、场景理解（109个样本，欺骗性的场景描述）、物体计数（29个样本，错误的物体数量）、文本识别（50个样本，误导性的文本相关问题）和物体属性（21个样本，错误的物体属性）。由于URL损坏，43张图像无法下载。评估指标为欺骗抵抗率，模型生成对欺骗性提示的响应，GPT-4o判断其正确性。数据来源为arXiv 2024年的MAD-Bench论文。

Benchmark for evaluating MLLM vulnerability to deceptive prompts containing incorrect information. 957 test samples across 5 categories, evaluated using GPT-4o as judge. The dataset includes images (from COCO, Flickr, and web sources), deceptive prompts containing incorrect information, and the category of deception. The five categories are: non-existent_object (748 samples, prompts about objects not in the image), scene_understanding (109 samples, deceptive scene descriptions), count_of_object (29 samples, incorrect object counts), text_recognition (50 samples, misleading text-related questions), and object_attribute (21 samples, wrong object attributes). 43 images could not be downloaded due to broken URLs. Evaluation metric is deception resistance rate, where the model generates a response to the deceptive prompt and GPT-4o judges its correctness. Original data from MAD-Bench (arXiv 2024).