Physis-AI/wm-eval-samples-ewmbench

---
license: cc-by-nc-sa-4.0
task_categories:
- video-classification
- robotics
tags:
- world-model
- evaluation
- embodied-ai
- generated-video
pretty_name: "WM-Eval Samples: EWMBench Generated Videos"
size_categories:
- 1K<n<10K
---

# WM-Eval Samples: EWMBench Generated Videos

Pre-generated video samples for [EWMBench](https://github.com/AgibotTech/EWMBench) evaluation, extracted and restructured for direct use by the [wm-evaluation-harness](https://github.com/Physis-AI/wm-evaluation-harness) framework.

Use this dataset as `videos_dir` to validate the evaluation pipeline without running your own World Model.

## Source

Extracted from [`agibot-world/EWMBench`](https://huggingface.co/datasets/agibot-world/EWMBench) (`generated_samples.tar`). Original dataset by [AgibotTech](https://github.com/AgibotTech).

## Dataset Structure

```
{task_id}/
├── {episode_id}/
│ ├── 1/ # Trial 1
│ │ └── video/
│ │ ├── frame_00000.jpg
│ │ ├── frame_00001.jpg
│ │ └── ...
│ ├── 2/ # Trial 2
│ │ └── video/
│ └── 3/ # Trial 3
│ └── video/
```

## Statistics

| Item | Count |
|------|-------|
| Tasks | 3 (367, 515, 558) |
| Episodes per task | 3 |
| Trials per episode | 3 |
| Total trial videos | 27 |
| Total files | 4,644 |
| Size | ~139 MB |

## Usage with wm-evaluation-harness

Use as `--videos` argument for pipeline validation:

```bash
# Quick validation (no GPU required)
wm-eval evaluate \
--config configs/ewmbench_hf.yaml \
--videos Physis-AI/wm-eval-samples-ewmbench

# Full evaluation (requires GPU)
wm-eval evaluate \
--config configs/ewmbench.yaml \
--videos Physis-AI/wm-eval-samples-ewmbench
```

Paired with [`Physis-AI/wm-eval-gt-ewmbench`](https://huggingface.co/datasets/Physis-AI/wm-eval-gt-ewmbench) as ground truth.

## License

This dataset follows the original [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license from AgibotTech/EWMBench.

## Citation

```bibtex
@article{ewmbench2025,
title={EWMBench: Evaluating Scene Understanding and Generation Quality for Embodied World Models},
author={AgibotTech},
year={2025},
eprint={2505.09694},
archivePrefix={arXiv}
}
```