MM-Hallu/BEAF

BEAF是一个用于评估视觉语言模型中对象幻觉的基准数据集，通过前后图像对比的方式构建。数据集包含26,064个问答对，覆盖2,223张图像（500张原始COCO图像和1,723张经过处理的图像）。每个数据样本包含图像、问题、真实答案等字段。问题采用POPE风格的二元问答形式，询问图像中是否存在特定对象。数据集还标注了图像是否为原始COCO图像、问题是否涉及被移除的对象等信息。问题分为四类：原始图像中存在的对象、原始图像中将被移除的对象、处理后图像中被移除的对象以及处理后图像中仍然存在的对象。评估指标包括准确率、精确率、召回率和F1分数。数据来源于ECCV 2024的BEAF项目。

BEAF is a benchmark for evaluating object hallucination in vision-language models using before-after image manipulation pairs. 26,064 QA pairs over 2,223 images (500 original COCO images + 1,723 manipulated images) with POPE-style yes/no questions. Each data sample includes fields such as image, question, ground truth answer, and annotations indicating whether the image is original COCO or whether the question refers to a removed object. Questions are categorized into four types: present objects in original images, to-be-removed objects in original images, removed objects in manipulated images, and remaining objects in manipulated images. Evaluation metrics include Accuracy, Precision, Recall, and F1. The data originates from the BEAF project presented at ECCV 2024.