Physis-AI/wm-eval-gt-ewmbench

--- license: cc-by-nc-sa-4.0 task_categories: - video-classification - robotics tags: - world-model - evaluation - embodied-ai - ground-truth pretty_name: "WM-Eval GT: EWMBench Ground Truth" size_categories: - 1K<n<10K --- # WM-Eval GT: EWMBench Ground Truth Ground truth data for [EWMBench](https://github.com/AgibotTech/EWMBench) evaluation, extracted and restructured for direct use by the [wm-evaluation-harness](https://github.com/Physis-AI/wm-evaluation-harness) framework. ## Source Extracted from [`agibot-world/EWMBench`](https://huggingface.co/datasets/agibot-world/EWMBench) (`gt_dataset.tar`). Original dataset by [AgibotTech](https://github.com/AgibotTech). ## Dataset Structure ``` {task_id}/ ├── {episode_id}/ │ ├── prompt/ │ │ ├── init_frame.png # Initial frame for generation │ │ └── prompt.txt # Task description │ └── video/ │ ├── frame_00000.jpg # Ground truth frame sequence │ ├── frame_00001.jpg │ └── ... ``` ## Statistics | Item | Count | |------|-------| | Tasks | 7 (367, 392, 497, 511, 543, 558, 574) | | Episodes per task | 3 | | Total episodes | 21 | | Total files | 3,959 | | Size | ~129 MB | ## Usage with wm-evaluation-harness This dataset is auto-downloaded by the framework when referenced as a HuggingFace dataset repo ID: ```yaml # configs/ewmbench_hf.yaml evaluation: gt_dir: Physis-AI/wm-eval-gt-ewmbench # auto-download benchmarks: ewmbench: metrics: [psnr, ssim] ``` ```bash wm-eval evaluate --config configs/ewmbench_hf.yaml --videos <your_pred_dir> ``` Or used programmatically: ```python from wm_eval.evaluation.models.registry import ModelRegistry registry = ModelRegistry() gt_path = registry.resolve("Physis-AI/wm-eval-gt-ewmbench", repo_type="dataset") ``` ## License This dataset follows the original [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license from AgibotTech/EWMBench. ## Citation ```bibtex @article{ewmbench2025, title={EWMBench: Evaluating Scene Understanding and Generation Quality for Embodied World Models}, author={AgibotTech}, year={2025}, eprint={2505.09694}, archivePrefix={arXiv} } ```