Physis-AI/wm-eval-gt-ewmbench

---
license: cc-by-nc-sa-4.0
task_categories:
- video-classification
- robotics
tags:
- world-model
- evaluation
- embodied-ai
- ground-truth
pretty_name: "WM-Eval GT: EWMBench Ground Truth"
size_categories:
- 1K<n<10K
---

# WM-Eval GT: EWMBench Ground Truth

Ground truth data for [EWMBench](https://github.com/AgibotTech/EWMBench) evaluation, extracted and restructured for direct use by the [wm-evaluation-harness](https://github.com/Physis-AI/wm-evaluation-harness) framework.

## Source

Extracted from [`agibot-world/EWMBench`](https://huggingface.co/datasets/agibot-world/EWMBench) (`gt_dataset.tar`). Original dataset by [AgibotTech](https://github.com/AgibotTech).

## Dataset Structure

```
{task_id}/
├── {episode_id}/
│ ├── prompt/
│ │ ├── init_frame.png # Initial frame for generation
│ │ └── prompt.txt # Task description
│ └── video/
│ ├── frame_00000.jpg # Ground truth frame sequence
│ ├── frame_00001.jpg
│ └── ...
```

## Statistics

| Item | Count |
|------|-------|
| Tasks | 7 (367, 392, 497, 511, 543, 558, 574) |
| Episodes per task | 3 |
| Total episodes | 21 |
| Total files | 3,959 |
| Size | ~129 MB |

## Usage with wm-evaluation-harness

This dataset is auto-downloaded by the framework when referenced as a HuggingFace dataset repo ID:

```yaml
# configs/ewmbench_hf.yaml
evaluation:
gt_dir: Physis-AI/wm-eval-gt-ewmbench # auto-download
benchmarks:
ewmbench:
metrics: [psnr, ssim]
```

```bash
wm-eval evaluate --config configs/ewmbench_hf.yaml --videos <your_pred_dir>
```

Or used programmatically:

```python
from wm_eval.evaluation.models.registry import ModelRegistry

registry = ModelRegistry()
gt_path = registry.resolve("Physis-AI/wm-eval-gt-ewmbench", repo_type="dataset")
```

## License

This dataset follows the original [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license from AgibotTech/EWMBench.

## Citation

```bibtex
@article{ewmbench2025,
title={EWMBench: Evaluating Scene Understanding and Generation Quality for Embodied World Models},
author={AgibotTech},
year={2025},
eprint={2505.09694},
archivePrefix={arXiv}
}
```