USC-PSI-Lab/PhysBench

---
language:
- en
license: apache-2.0
task_categories:
- video-text-to-text
pretty_name: PhysBench
modalities:
- text
- image
- video
---

<div align="center">
<h1> <img src="assets/physbench.png" width="50" /> PhysBench </h1>
</div>
<h5 align="center">
<a href="https://physbench.github.io/">🌐 Homepage</a> | <a href="https://huggingface.co/datasets/USC-GVL/PhysBench">🤗 Dataset</a> | <a href="https://huggingface.co/papers/2501.16411">📑 Paper</a> | <a href="https://github.com/USC-GVL/PhysBench/tree/main/eval">💻 Code</a> | <a href="https://eval.ai/web/challenges/challenge-page/2461/overview">🔺 EvalAI</a>
</h5>

This repo contains evaluation code for the paper "[PhysBench: Benchmarking and Enhancing VLMs for Physical World Understanding](https://huggingface.co/papers/2501.16411)"
If you like our project, please give us a star ⭐ on GitHub for latest update.


## Introduction
**Understanding the physical world** is a fundamental challenge in embodied AI, critical for enabling agents to perform complex tasks and operate safely in real-world environments. While Vision-Language Models (VLMs) have shown great promise in reasoning and task planning for embodied agents, their ability to comprehend physical phenomena remains extremely limited.
To close this gap, we introduce **PhysBench**, a comprehensive benchmark designed to evaluate VLMs' physical world understanding capability across a diverse set of tasks.
**PhysBench** is categorized into four major domains: physical object properties, physical object relationships, physical scene understanding, and physics-based dynamics, further divided into 19 subclasses and 8 distinct capability dimensions.
Our extensive experiments, conducted on 39 representative VLMs, reveal that while these models excel in common-sense reasoning, they struggle with understanding the physical world---likely due to the absence of physical knowledge in their training data and the lack of embedded physical priors.
To tackle the shortfall, we introduce **PhysAgent**, a novel framework that combines the generalization strengths of VLMs with the specialized expertise of vision models, significantly enhancing VLMs' physical understanding across a variety of tasks, including an 18.4% improvement on GPT-4o.
Furthermore, our results demonstrate that enhancing VLMs’ physical world understanding capabilities can significantly help the deployment of embodied agents, pushing the boundaries of machine intelligence in comprehending and interacting with the physical world. We believe that PhysBench and PhysAgent offer valuable insights and contribute to bridging the gap between VLMs and physical world understanding.

## Dataset Creation

## Dataset Summary

The complete **PhysBench-test** dataset consists of 10,002 entries, which are more challenging and diverse, as the test set, and 200 entries as the validation set for parameter choosing.

<img src="assets/stat.png" width="900" />

## Load Dataset

ou can access the evaluation 💻scripts [here](https://github.com/USC-GVL/PhysBench/tree/main/eval), and we have packaged all 39 Vision-Language Models (VLMs) used in the paper to facilitate the reproducibility of our results.

```shell
cd <your_path_for_dataset>
huggingface-cli download USC-GVL/PhysBench --local-dir . --local-dir-use-symlinks False --repo-type dataset
yes | unzip image.zip -d image
yes | unzip video.zip -d video
```

## Item Format

All the questions are contained in [test.json](https://huggingface.co/datasets/USC-GVL/PhysBench/blob/main/test.json). For each item's key, our explanations are as follows (though we strongly recommend using our [packaged scripts]() for testing purposes).

| key | description |
| ----------- | ------------------------------------------------------------ |
| scene | Describes the context of the data, which may be left blank. Primarily, it records the HDR used in the simulation data. |
| object | Refers to the primary object in the scene. For instance, `glass_x` designates a specific instance x that may appear in multiple scenes, whereas `glass` refers to a general instance. |
| source | `real` indicates data collected by our camera, `simulation` refers to data generated by the simulation platform, and `web` denotes data sourced from websites. |
| file_name | Refers to the sequential input of visual content, including` <video>` and `<image>`. It should be noted that we have ensured that the sequence can be replaced in order from the beginning to the end. |
| question | Specifies the question along with four corresponding answers. |
| description | For video dialogues (other types may differ), it is structured as a list: [<video_description>, <detailed_description>]. <video_description> is human-annotated, while <detailed_description> is annotated by VILA-1.5. |

An example is like that:
```json
[
{
"scene": "black background",
"object": ["glass", "rubber bullet"],
"source": "web",
"file_name": ["iNINChj51Aqn.mp4", "iNINChj51Aqj.png", "iNINChj51Aqk.png", "iNINChj51Aql.png", "iNINChj51Aqm.png"],
"question": "Following the content of the <video>, which option's corresponding picture will happen first?\n A. <image>\nB. <image>\nC. <image>\nD. <image>\n",
"answer": "A",
"task_type": "phenomena",
"sub_type": "collision",
"ability_type": "prediction",
"description": null
}
]
```

## 🏆 Mini-Leaderboard

This is a subset of the leaderboard for the PhysBench test set. For the complete leaderboard, please refer to the [**🌐 Homepage**](https://physbench.github.io/).

You can submit your model’s predictions for the **test set** on **[EvalAI](https://eval.ai/web/challenges/challenge-page/2287/overview)**.

| **#** | **Model** | **ALL** | **Property** | **Relationships** | **Scene** | **Dynamics** |
| ----- | --------------------- | --------- | ---------- | ----------- | --------------- | ------------- |
| - | **Human Performance** | **95.87** | 97.10 | 95.67 | 94.91 | 95.68 |
| 1 | **InternVL2.5-38B 🥇** | **51.94** | 58.77 | 67.51 | 39.04 | 45.00 |
| 2 | **InternVL2.5-78B 🥈** | **51.16** | 60.32 | 62.13 | 37.32 | 46.11 |
| 3 | **GPT-4o 🥉** | **49.49** | 56.91 | 64.80 | 30.15 | 46.99 |
| 4 | Gemini-1.5-pro | **49.11** | 57.26 | 63.61 | 36.52 | 41.56 |
| 5 | InternVL2.5-26B | **48.56** | 59.08 | 58.33 | 36.61 | 41.79 |
| 6 | NVILA-15B | **46.91** | 59.16 | 42.34 | 38.78 | 45.72 |
| 7 | InternVL2-76B | **46.77** | 57.65 | 52.43 | 38.07 | 40.12 |
| 8 | Gemini-1.5-flash | **46.07** | 57.41 | 52.24 | 34.32 | 40.93 |
| 9 | InternVL2-40B | **45.66** | 55.79 | 50.05 | 35.86 | 41.33 |
| 10 | NVILA-Lite-15B | **44.93** | 55.44 | 40.15 | 38.11 | 44.38 |
| 11 | InternVL2.5-8B | **43.88** | 55.87 | 48.67 | 29.35 | 41.20 |
| 12 | NVILA-8B | **43.82** | 55.79 | 40.29 | 33.95 | 43.43 |
| 13 | InternVL2-26B | **43.50** | 51.92 | 45.20 | 37.94 | 39.34 |
| 14 | GPT-4o-mini | **43.15** | 53.54 | 44.24 | 30.59 | 42.90 |
| 15 | mPLUG-Owl3-7B | **42.83** | 49.25 | 45.62 | 35.90 | 40.61 |
| 16 | NVILA-Lite-8B | **42.55** | 53.81 | 39.25 | 34.62 | 41.17 |
| 17 | InternVL2.5-4B | **42.44** | 51.03 | 44.77 | 31.34 | 41.79 |
| 18 | GPT-4V | **41.26** | 49.59 | 45.77 | 26.34 | 42.15 |
| 19 | LLaVA-interleave | **41.00** | 47.23 | 44.62 | 35.64 | 37.21 |
| 20 | LLaVA-interleave-dpo | **40.83** | 47.97 | 42.67 | 33.73 | 38.78 |
| 21 | InternVL2-8B | **40.00** | 49.05 | 43.58 | 27.05 | 39.47 |
| 22 | Phi-3.5V | **39.75** | 45.72 | 40.15 | 33.02 | 39.40 |
| 23 | InternVL2-4B | **39.71** | 47.12 | 39.96 | 30.94 | 39.76 |
| 24 | InternVL2.5-2B | **39.22** | 49.63 | 38.15 | 29.44 | 38.39 |
| 25 | Phi-3V | **38.42** | 43.67 | 37.92 | 34.93 | 36.92 |
| 26 | Mantis-siglip-llama3 | **37.64** | 42.47 | 32.78 | 36.83 | 37.51 |
| 27 | LLaVA-NV-dpo | **37.43** | 38.83 | 44.31 | 33.86 | 37.21 |
| 28 | Mantis-Idefics2 | **37.39** | 41.97 | 41.44 | 29.53 | 36.56 |
| 29 | VILA-1.5-13B | **37.15** | 40.53 | 40.15 | 31.96 | 36.07 |
| 30 | Mantis-clip-llama3 | **36.92** | 40.61 | 35.11 | 32.45 | 38.36 |
| 31 | Mantis-LLaVA | **36.69** | 44.48 | 30.45 | 36.25 | 34.73 |
| 32 | InternVL2-2B | **36.57** | 44.17 | 35.06 | 30.54 | 35.64 |
| 33 | InternVL2.5-1B | **36.15** | 44.25 | 33.30 | 26.87 | 38.13 |
| 34 | LLaVA-NV | **35.42** | 38.33 | 30.83 | 34.00 | 37.17 |
| 35 | mPLUG-Owl3-2B | **34.87** | 40.92 | 35.11 | 26.69 | 35.64 |
| 36 | VILA-1.5-3B | **34.11** | 32.40 | 33.02 | 34.84 | 35.78 |
| 37 | VILA-1.5-3B-s2 | **33.07** | 33.14 | 30.26 | 35.72 | 33.00 |
| 38 | VILA-1.5-8B | **32.85** | 33.41 | 29.88 | 30.85 | 35.91 |
| 39 | InternVL2-1B | **32.35** | 37.05 | 33.06 | 22.84 | 34.92 |
| 40 | mPLUG-Owl3-1B | **31.68** | 38.02 | 31.54 | 21.87 | 33.00 |

## Disclaimers

Some of the data in PhysBench has been annotated based on existing datasets, as noted in the appendix of the paper. For the forensics detection task, we manually collected images that are publicly available through online searches. We have made every effort to comply with applicable copyright laws and ensure proper attribution of the images used in this paper. However, if you are the copyright holder of any image included in our work and believe its use conflicts with your licensing agreements, please [contact](#contact) us directly. We are committed to promptly addressing any legitimate concerns.

## Contact
- Wei Chow: [e-mail]

Other links:

[PhysBench-media](https://huggingface.co/datasets/WeiChow/PhysBench-media)

[PhysBench-train](https://huggingface.co/datasets/WeiChow/PhysBench-train)

[PhysBench-assets](https://huggingface.co/datasets/WeiChow/PhysBench-assets)

## Citation

**BibTeX:**
```bibtex
@article{chow2025physbench,
title={PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding},
author={Chow, Wei and Mao, Jiageng and Li, Boyi and Seita, Daniel and Guizilini, Vitor and Wang, Yue},
journal={arXiv preprint arXiv:2501.16411},
year={2025}
}
```

---
语言:
- 英语
许可证: Apache 2.0
任务类别:
- 视频-文本到文本
展示名: PhysBench
模态:
- 文本
- 图像
- 视频
---

<div align="center">
<h1> <img src="assets/physbench.png" width="50" /> PhysBench </h1>
</div>
<h5 align="center">
<a href="https://physbench.github.io/">🌐 官网</a> | <a href="https://huggingface.co/datasets/USC-GVL/PhysBench">🤗 数据集页面</a> | <a href="https://huggingface.co/papers/2501.16411">📑 论文</a> | <a href="https://github.com/USC-GVL/PhysBench/tree/main/eval">💻 代码</a> | <a href="https://eval.ai/web/challenges/challenge-page/2461/overview">🔺 EvalAI平台</a>
</h5>

本仓库包含论文《PhysBench：面向物理世界理解的视觉语言模型评测与增强》对应的评测代码。若您喜爱本项目，欢迎前往GitHub为我们点亮Star以获取最新更新。


## 引言
**理解物理世界**是具身人工智能（embodied AI）领域的核心挑战之一，对于智能体完成复杂任务并在真实环境中安全运行至关重要。尽管视觉语言模型（Vision-Language Models, VLMs）在具身智能体的推理与任务规划方面展现出巨大潜力，但其理解物理现象的能力仍极为有限。
为填补这一空白，我们推出**PhysBench**——一款旨在从多样化任务维度评测视觉语言模型物理世界理解能力的综合性基准测试集。
PhysBench共分为四大核心领域：物理对象属性、物理对象关系、物理场景理解以及基于物理的动力学，进一步细分为19个子类别与8个独立的能力维度。
我们在39款代表性视觉语言模型上开展的大规模实验表明，尽管这些模型在常识推理方面表现优异，但在物理世界理解上却力有不逮——这大概率源于其训练数据中缺乏物理知识，且未嵌入物理先验信息。
为解决这一短板，我们提出**PhysAgent**——一款融合视觉语言模型泛化优势与视觉模型专业能力的新型框架，可显著提升视觉语言模型的物理理解能力，其中GPT-4o的性能提升幅度达18.4%。
此外，我们的实验结果证实，提升视觉语言模型的物理世界理解能力，可有效助力具身智能体的落地应用，推动机器智能在物理世界理解与交互领域的边界拓展。我们认为，PhysBench与PhysAgent能够提供极具价值的研究视角，为缩小视觉语言模型与物理世界理解能力之间的差距贡献力量。

## 数据集构建
## 数据集摘要
完整的**PhysBench测试集**共包含10002条数据，作为测试集其样本更具挑战性与多样性；同时附带200条数据作为参数调优用的验证集。

<img src="assets/stat.png" width="900" />

## 加载数据集
您可通过[此处](https://github.com/USC-GVL/PhysBench/tree/main/eval)获取评测脚本，我们已将论文中使用的全部39款视觉语言模型进行了封装，以确保实验结果可复现。

shell
cd <your_path_for_dataset>
huggingface-cli download USC-GVL/PhysBench --local-dir . --local-dir-use-symlinks False --repo-type dataset
yes | unzip image.zip -d image
yes | unzip video.zip -d video

## 数据项格式
所有测试问题均收录于[test.json](https://huggingface.co/datasets/USC-GVL/PhysBench/blob/main/test.json)中。以下为各数据项键值的详细说明（尽管我们强烈建议使用我们封装的脚本进行测试）。

| 键名 | 说明 |
| ----------- | ------------------------------------------------------------ |
| scene | 描述数据的上下文信息，可留空。主要用于记录仿真数据中使用的高动态范围（HDR）配置。 |
| object | 指场景中的主要对象。例如，`glass_x`代表可能在多个场景中出现的特定实例x，而`glass`则指代通用的玻璃实例。 |
| source | `real`表示通过我们的相机采集的真实数据，`simulation`表示仿真平台生成的数据，`web`表示从网络获取的数据。 |
| file_name | 指视觉内容的顺序输入，包括`<video>`和`<image>`。需要注意的是，我们已确保该序列可按从前往后的顺序进行替换。 |
| question | 给出问题及对应的四个候选答案。 |
| description | 对于视频对话类数据（其他类型可能有所不同），其结构为列表：[<视频描述>, <详细描述>]。其中<视频描述>为人工标注，<详细描述>由VILA-1.5模型标注。 |

一个示例如下：
json
[
{
"scene": "black background",
"object": ["glass", "rubber bullet"],
"source": "web",
"file_name": ["iNINChj51Aqn.mp4", "iNINChj51Aqj.png", "iNINChj51Aqk.png", "iNINChj51Aql.png", "iNINChj51Aqm.png"],
"question": "Following the content of the <video>, which option's corresponding picture will happen first?
A. <image>
B. <image>
C. <image>
D. <image>
",
"answer": "A",
"task_type": "phenomena",
"sub_type": "collision",
"ability_type": "prediction",
"description": null
}
]

## 🏆 微型排行榜
本榜单为PhysBench测试集排行榜的子集，完整榜单请参阅[**🌐 官网**](https://physbench.github.io/)。
您可在**[EvalAI平台](https://eval.ai/web/challenges/challenge-page/2287/overview)**上提交您的模型在**测试集**上的预测结果。

| 排名 | 模型 | 整体得分 | 属性理解 | 关系推理 | 场景理解 | 动力学分析 |
| ---- | ---- | -------- | -------- | -------- | -------- | ---------- |
| - | 人类表现 | 95.87 | 97.10 | 95.67 | 94.91 | 95.68 |
| 1 | InternVL2.5-38B 🥇 | 51.94 | 58.77 | 67.51 | 39.04 | 45.00 |
| 2 | InternVL2.5-78B 🥈 | 51.16 | 60.32 | 62.13 | 37.32 | 46.11 |
| 3 | GPT-4o 🥉 | 49.49 | 56.91 | 64.80 | 30.15 | 46.99 |
| 4 | Gemini-1.5-pro | 49.11 | 57.26 | 63.61 | 36.52 | 41.56 |
| 5 | InternVL2.5-26B | 48.56 | 59.08 | 58.33 | 36.61 | 41.79 |
| 6 | NVILA-15B | 46.91 | 59.16 | 42.34 | 38.78 | 45.72 |
| 7 | InternVL2-76B | 46.77 | 57.65 | 52.43 | 38.07 | 40.12 |
| 8 | Gemini-1.5-flash | 46.07 | 57.41 | 52.24 | 34.32 | 40.93 |
| 9 | InternVL2-40B | 45.66 | 55.79 | 50.05 | 35.86 | 41.33 |
| 10 | NVILA-Lite-15B | 44.93 | 55.44 | 40.15 | 38.11 | 44.38 |
| 11 | InternVL2.5-8B | 43.88 | 55.87 | 48.67 | 29.35 | 41.20 |
| 12 | NVILA-8B | 43.82 | 55.79 | 40.29 | 33.95 | 43.43 |
| 13 | InternVL2-26B | 43.50 | 51.92 | 45.20 | 37.94 | 39.34 |
| 14 | GPT-4o-mini | 43.15 | 53.54 | 44.24 | 30.59 | 42.90 |
| 15 | mPLUG-Owl3-7B | 42.83 | 49.25 | 45.62 | 35.90 | 40.61 |
| 16 | NVILA-Lite-8B | 42.55 | 53.81 | 39.25 | 34.62 | 41.17 |
| 17 | InternVL2.5-4B | 42.44 | 51.03 | 44.77 | 31.34 | 41.79 |
| 18 | GPT-4V | 41.26 | 49.59 | 45.77 | 26.34 | 42.15 |
| 19 | LLaVA-interleave | 41.00 | 47.23 | 44.62 | 35.64 | 37.21 |
| 20 | LLaVA-interleave-dpo | 40.83 | 47.97 | 42.67 | 33.73 | 38.78 |
| 21 | InternVL2-8B | 40.00 | 49.05 | 43.58 | 27.05 | 39.47 |
| 22 | Phi-3.5V | 39.75 | 45.72 | 40.15 | 33.02 | 39.40 |
| 23 | InternVL2-4B | 39.71 | 47.12 | 39.96 | 30.94 | 39.76 |
| 24 | InternVL2.5-2B | 39.22 | 49.63 | 38.15 | 29.44 | 38.39 |
| 25 | Phi-3V | 38.42 | 43.67 | 37.92 | 34.93 | 36.92 |
| 26 | Mantis-siglip-llama3 | 37.64 | 42.47 | 32.78 | 36.83 | 37.51 |
| 27 | LLaVA-NV-dpo | 37.43 | 38.83 | 44.31 | 33.86 | 37.21 |
| 28 | Mantis-Idefics2 | 37.39 | 41.97 | 41.44 | 29.53 | 36.56 |
| 29 | VILA-1.5-13B | 37.15 | 40.53 | 40.15 | 31.96 | 36.07 |
| 30 | Mantis-clip-llama3 | 36.92 | 40.61 | 35.11 | 32.45 | 38.36 |
| 31 | Mantis-LLaVA | 36.69 | 44.48 | 30.45 | 36.25 | 34.73 |
| 32 | InternVL2-2B | 36.57 | 44.17 | 35.06 | 30.54 | 35.64 |
| 33 | InternVL2.5-1B | 36.15 | 44.25 | 33.30 | 26.87 | 38.13 |
| 34 | LLaVA-NV | 35.42 | 38.33 | 30.83 | 34.00 | 37.17 |
| 35 | mPLUG-Owl3-2B | 34.87 | 40.92 | 35.11 | 26.69 | 35.64 |
| 36 | VILA-1.5-3B | 34.11 | 32.40 | 33.02 | 34.84 | 35.78 |
| 37 | VILA-1.5-3B-s2 | 33.07 | 33.14 | 30.26 | 35.72 | 33.00 |
| 38 | VILA-1.5-8B | 32.85 | 33.41 | 29.88 | 30.85 | 35.91 |
| 39 | InternVL2-1B | 32.35 | 37.05 | 33.06 | 22.84 | 34.92 |
| 40 | mPLUG-Owl3-1B | 31.68 | 38.02 | 31.54 | 21.87 | 33.00 |

## 免责声明
PhysBench中的部分数据基于现有数据集进行标注，具体说明详见论文附录。对于取证检测任务，我们通过网络搜索手动收集了公开可用的图像。我们已尽最大努力遵守适用的版权法规，并确保对本文中使用的图像进行了适当的署名。但若您是本文中某张图像的版权持有者，并认为其使用方式与您的许可协议相悖，请直接[联系](#contact)我们。我们将致力于及时处理任何合理的诉求。

## 联系方式
- 周炜（Wei Chow）：[e-mail]

其他相关链接：
[PhysBench-media](https://huggingface.co/datasets/WeiChow/PhysBench-media)
[PhysBench训练集](https://huggingface.co/datasets/WeiChow/PhysBench-train)
[PhysBench资源包](https://huggingface.co/datasets/WeiChow/PhysBench-assets)

## 引用
**BibTeX格式：**
bibtex
@article{chow2025physbench,
title={PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding},
author={Chow, Wei and Mao, Jiageng and Li, Boyi and Seita, Daniel and Guizilini, Vitor and Wang, Yue},
journal={arXiv preprint arXiv:2501.16411},
year={2025}
}