EWMBench

<div align="center">

<h2>
EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models
</h2>

</div>

<div align="center">
<a href="https://github.com/AgibotTech/EWMBench">
<img src="https://img.shields.io/badge/GitHub-grey?logo=GitHub" alt="GitHub">
</a>

<a href="https://arxiv.org/abs/2505.09694">
<img src="https://img.shields.io/badge/arXiv-2505.09694-b31b1b.svg?logo=arxiv" alt="arXiv"/>
</a>

</div>

<img src="figs/pipe.jpg" alt="Image Alt Text" width="90%" style="display: block; margin-left: auto; margin-right: auto;" />

<img src="figs/dataset.jpg" alt="Image Alt Text" width="90%" style="display: block; margin-left: auto; margin-right: auto;" />

### Resources

- 🐙 **GitHub**: Explore the project repository to run evaluation script. [AgibotTech/EWMBench](https://github.com/AgibotTech/EWMBench).
- 📑 **arXiv**: Read our paper for detailed methodology and results at [arXiv:2505.09694](https://arxiv.org/abs/2505.09694).
- 🤗 **Data**: Discover [EWMBench Dataset](https://huggingface.co/datasets/agibot-world/EWMBench/tree/main), we sample a diverse dataset from AgiBot World for running EWMBench evaluation.
- 🤗 **Model**: Download pretrained weights used for evaluation from [EWMBench-model](https://huggingface.co/agibot-world/EWMBench-model/tree/main).

## Data Specification

### Ground Truth Data

```
gt_dataset/
├── task_1/
│ ├── episode_1/
│ │ ├── prompt/
│ │ │ ├── init_frame.png
│ │ │ └── introduction.txt
│ │ └── video/
│ │ ├── frame_00000.jpg
│ │ ├── ...
│ │ └── frame_0000n.jpg
│ ├── episode_2/
│ └── ...
├── task_2/
└── ...
```

### Generated Samples

```
generated_samples/
├── task_1/
│ ├── episode_1/
│ │ ├── 1/
│ │ │ └── video/
│ │ │ ├── frame_00000.jpg
│ │ │ ├── ...
│ │ │ └── frame_0000n.jpg
│ │ ├── 2/
│ │ └── 3/
│ ├── episode_2/
│ └── ...
├── task_2/
└── ...
```

# License and Citation
All the data and code within this repo are under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). Please consider citing our project if it helps your research.

```BibTeX
@article{hu2025ewmbench,
title={EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models},
author={Hu, Yue and Huang, Siyuan and Liao, Yue and Chen, Shengcong and Zhou, Pengfei and Chen, Liliang and Yao, Maoqing and Ren, Guanghui},
journal={arXiv preprint arXiv:2505.09694},
year={2025}
}

<div align="center">
<h2>
EWMBench：具身世界模型（Embodied World Models）的场景、运动与语义质量评估基准
</h2>
</div>

<div align="center">
<a href="https://github.com/AgibotTech/EWMBench">
<img src="https://img.shields.io/badge/GitHub-grey?logo=GitHub" alt="GitHub">
</a>

<a href="https://arxiv.org/abs/2505.09694">
<img src="https://img.shields.io/badge/arXiv-2505.09694-b31b1b.svg?logo=arxiv" alt="arXiv">
</a>

</div>

<img src="figs/pipe.jpg" alt="Image Alt Text" width="90%" style="display: block; margin-left: auto; margin-right: auto;" />

<img src="figs/dataset.jpg" alt="Image Alt Text" width="90%" style="display: block; margin-left: auto; margin-right: auto;" />

### 资源

- 🐙 **GitHub 仓库**：浏览本项目仓库以运行评估脚本，地址为 [AgibotTech/EWMBench](https://github.com/AgibotTech/EWMBench)。
- 📑 **arXiv 论文**：查阅本论文可获取详细的研究方法与实验结果，链接为 [arXiv:2505.09694](https://arxiv.org/abs/2505.09694)。
- 🤗 **数据集资源**：可从 [EWMBench 数据集](https://huggingface.co/datasets/agibot-world/EWMBench/tree/main) 获取本次评估所用的多样化数据集，该数据集从 AgiBot World 中采样得到。
- 🤗 **模型权重**：可从 [EWMBench-model](https://huggingface.co/agibot-world/EWMBench-model/tree/main) 下载评估所需的预训练模型权重。

## 数据规范

### 基准真值数据

gt_dataset/
├── task_1/
│ ├── episode_1/
│ │ ├── prompt/
│ │ │ ├── init_frame.png
│ │ │ └── introduction.txt
│ │ └── video/
│ │ ├── frame_00000.jpg
│ │ ├── ...
│ │ └── frame_0000n.jpg
│ ├── episode_2/
│ └── ...
├── task_2/
└── ...

### 生成样本数据集

generated_samples/
├── task_1/
│ ├── episode_1/
│ │ ├── 1/
│ │ │ └── video/
│ │ │ ├── frame_00000.jpg
│ │ │ ├── ...
│ │ │ └── frame_0000n.jpg
│ │ ├── 2/
│ │ └── 3/
│ ├── episode_2/
│ └── ...
├── task_2/
└── ...

## 许可与引用

本仓库内的所有数据与代码均遵循 [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) 许可协议。若本项目对您的研究有所帮助，请考虑引用我们的工作。

BibTeX
@article{hu2025ewmbench,
title={EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models},
author={Hu, Yue and Huang, Siyuan and Liao, Yue and Chen, Shengcong and Zhou, Pengfei and Chen, Liliang and Yao, Maoqing and Ren, Guanghui},
journal={arXiv preprint arXiv:2505.09694},
year={2025}
}