mrble/MARBLE

MARBLE是一个旨在评估多模态语言模型在处理复杂多模态问题时的推理能力的挑战性基准。它包含两个高度挑战性的任务M-Portal和M-Cube，这些任务需要制定和理解多步骤计划，利用空间、视觉和物理约束。M-Portal是基于Portal 2关卡设计的多步骤空间规划谜题，而M-Cube则是受到Happy Cube拼图谜题启发的3D立方体组装任务。此外，M-Cube还包含一个简单的感知任务cube_perception。

MARBLE is a challenging benchmark designed to assess the reasoning ability of multimodal language models (MLLMs) when handling complex multimodal problems. It consists of two highly challenging tasks, M-Portal and M-Cube, which require the crafting and understanding of multistep plans leveraging spatial, visual, and physical constraints. M-Portal is a multi-step spatial-planning puzzle based on levels from Portal 2, while M-Cube is a 3D cube assembly task inspired by Happy Cube puzzles. Additionally, M-Cube includes a simple perception task called cube_perception.