Anon-compass/COMPASS

COMPASS是一个用于诊断视觉语言模型（VLMs）组合失败的受控评估基准。它通过从Visual Genome场景图构建带有显式对象、属性和关系结构的标题，实现两种针对性分析：组合集成差距（量化联合推理成本）和技能负载（测量每个技能如何随原始类型数量增加而退化）。数据集包含分层标题结构（如仅对象、对象+属性、对象+关系、对象+属性+关系），总共有138万个组合地面真值标题，以及用于评估的87K对（组合集成）和274K对（技能负载）。硬负样本通过语义上合理的原始替换构建，支持组合集成和技能负载分析。数据基于5K个Visual Genome图像-场景图对构建，使用GPT-4o mini生成标题和硬负样本，并通过质量控制确保语言不可区分性。评估基于图像到文本检索，关键发现包括联合推理成本部分解释退化、自我负载主导、交叉负载提供上下文益处，且模式与架构无关。

COMPASS is a controlled evaluation benchmark for diagnosing compositional failure in vision-language models (VLMs). It constructs captions from Visual Genome scene graphs with explicit object, attribute, and relation structure, enabling two targeted analyses: the compositional integration gap (cost of joint reasoning) and skill load (how each skill degrades as primitive counts increase). The dataset includes hierarchical caption structures (e.g., objects only, objects+attributes, objects+relations, objects+attributes+relations), with a total of 1.38M composed ground-truth captions, 87K pairs for compositional integration evaluation, and 274K pairs for skill load evaluation. Hard negatives are constructed via semantically plausible primitive replacements, supporting both compositional integration and skill load analyses. Data is built from 5K Visual Genome image-scene graph pairs, with captions and hard negatives generated using GPT-4o mini and quality-controlled for linguistic indistinguishability. Evaluation uses image-to-text retrieval, with key findings showing that joint reasoning has a partial cost, self-load dominates, cross-load provides grounding benefits, and patterns are architecture-agnostic.