ai4bharat/Focus

Focus是一个元评估基准数据集，旨在评估视觉语言模型(VLMs)在各种图像到文本(I2T)和文本到图像(T2I)任务中的鲁棒性。该数据集包含两种主要类型的扰动：I2T(图像到文本)和T2I(文本到图像)。I2T部分包括视觉基础、语义解释、视觉推理、长文本生成和分数不变性等子集；T2I部分包括视觉保真度、场景连贯性、物理合理性和文本渲染等子集。

Focus is a meta-evaluation benchmark dataset designed to evaluate the robustness of Vision-Language Models (VLMs) across various image-to-text (I2T) and text-to-image (T2I) tasks. This dataset includes two primary categories of perturbations, specifically corresponding to the I2T and T2I task types. The I2T section comprises subsets such as visual grounding, semantic interpretation, visual reasoning, long-text generation, and score invariance; the T2I section covers subsets including visual fidelity, scene coherence, physical plausibility, and text rendering.