PhyX

该数据集名为PhyX，是一个大规模的基准测试，旨在评估模型在视觉场景中对物理知识推理的能力。它包含了3,000个精心策划的多模态问题，覆盖了6种推理类型，分布在6个核心物理领域的25个子领域中。该数据集还包括了一个名为testmini的代表性质子集，以便进行简化的评估。这些问题涵盖了热力学、电磁学、力学、现代物理、光学以及波与声学等多个物理领域。数据集的规模为3,000个问题和18,000个实例（其中testmini子集包含1,000个问题和6,000个实例）。其任务是进行物理推理评估。This dataset, named PhyX, is a large-scale benchmark designed to evaluate models' capacity for physical knowledge reasoning in visual scenes. It comprises 3,000 carefully curated multimodal questions, covering 6 types of reasoning and spanning 25 subfields across 6 core physics domains. The dataset also includes a representative subset named testmini to facilitate simplified evaluation. These questions span multiple physics disciplines including thermodynamics, electromagnetism, mechanics, modern physics, optics, as well as waves and acoustics. The full dataset contains 3,000 questions and 18,000 instances in total, with the testmini subset holding 1,000 questions and 6,000 instances respectively. The core task of this benchmark is physical reasoning assessment.

搜集汇总

背景与挑战

背景概述

PhyX是一个大规模多模态基准数据集，包含3,000个人工标注的物理问题，覆盖热力学、电磁学等6个核心物理领域和25个子领域，旨在评估模型在视觉场景中的物理推理能力。该数据集强调现实性和去文本冗余，评估显示即使先进模型（如GPT-4o）准确率也较低（32.5%），与人类专家（最高78.9%）存在显著差距，突显物理推理的挑战性。