MMMU/MMMU_Pro

MMMU-Pro是一个增强的多模态基准数据集，旨在严格评估先进AI模型在多个模态下的真实理解能力。它建立在原始MMMU基准的基础上，通过引入几个关键改进，使其更具挑战性和现实性，确保模型在整合和理解视觉和文本信息方面的真实能力得到评估。数据集包含多种需要模型解释和整合视觉和文本信息的问题，反映了现实世界中用户经常与嵌入式内容互动的场景。数据集分为标准子集和视觉子集，标准子集增加了候选答案的数量，而视觉子集则要求模型在没有单独文本输入的情况下整合视觉和文本信息。

MMMU-Pro is an enhanced multimodal benchmark designed to rigorously assess the true understanding capabilities of advanced AI models across multiple modalities. It builds upon the original MMMU benchmark by introducing several key improvements that make it more challenging and realistic, ensuring that models are evaluated on their genuine ability to integrate and comprehend both visual and textual information. The dataset includes a diverse set of questions that require models to interpret and integrate visual and textual information, reflecting real-world scenarios where users often interact with embedded content. The dataset is organized into two subsets: the Standard subset, which increases the number of candidate answers, and the Vision subset, which requires integration of visual and textual information without separate text input.