ToughStone/ContextualBench

ContextualBench数据集包含72个上下文实例，每个实例包含7个上下文一致的图像和7个上下文违反的图像。该数据集设计了4个视觉推理任务，并收集了人类注释。图像生成过程涉及从外部数据库收集上下文实例，并使用文本到图像模型生成图像，确保图像符合给定约束、无视觉错觉且无歧义或潜在冒犯性。任务设计包括视觉问答、图像描述和图像识别，以及针对上下文违反图像的图像解释任务。图像注释方面，对于图像描述、图像识别和图像解释，收集了来自不同人的5个注释；对于视觉问答，基于图像描述生成问答对。

ContextualBench consists of 6 categories × 12 instances = 72 context instances, with each context instance containing 7 context-consistent images and 7 context-violating images. We designed 4 visual reasoning tasks and collected human annotations. The image generation process involves collecting context instances from an external database and formatting them as key-value pairs, then editing some pairs to generate context-violating images. Each image is generated after meeting specific conditions. Task design includes visual question answering, image captioning, image identification, and image explanation. For image captioning, image identification, and image explanation, we collected annotations from 5 different individuals. For visual question answering, we generated Q-A pairs based on image captions.