MM-Hallu/CIEM

CIEM（对比性图像评估指标）基准测试数据集基于COCO val2017，包含4,952对事实性问题和对比性问题，用于测试模型是否能区分图像中是否存在特定对象。每对问题包括一个关于图像中存在的对象的事实性问题（答案为“是”）和一个关于图像中不存在的对象的对比性问题（答案为“否”）。数据集还包含图像、图像ID、问题对ID、事实性问题及其答案、对比性问题及其答案、以及图像中存在的对象列表。评估指标包括准确率、对比性准确率下降和F1分数，问题解析方式为二分类（是/否）。

The CIEM (Contrastive Image Evaluation Metric) benchmark is based on COCO val2017 and contains 4,952 paired factual vs contrastive yes/no questions per image, testing whether models can distinguish present vs absent objects. Each pair includes a factual question about an object present in the image (answer: Yes) and a contrastive question about an object absent from the image (answer: No). The dataset also includes the image, image ID, pair ID, factual question and answer, contrastive question and answer, and a list of all objects present in the image. Evaluation metrics include Accuracy, Contrastive Accuracy Drop, and F1, with parsing as yes/no binary.