shi-labs/physical-ai-bench-conditional-generation

---
language:
- en
pretty_name: PAI-Bench-Transfer
configs:
- config_name: benchmark
data_files:
- split: PAIBenchTransfer
path: metadata.csv
task_categories:
- video-to-video
license: mit
---

# Physical AI Bench - Conditional Generation

[Paper](https://huggingface.co/papers/2512.01989) | [Code](https://github.com/SHI-Labs/physical-ai-bench)

This dataset (Phsical AI benchmark, PAI-Bench) consisting of 600 examples across three key scenarios: robotic arm operations, driving, and ego-centric everyday life scenes, each representing a critical aspect of Physical AI. This dataset is constructed by sampling a number of videos from three different datasets. The specific details are provided below.

| Dataset | Category | Sample Nums |
| ------------------------------------------------------------ | ------------------ | ----------- |
| [Agibot World](https://github.com/OpenDriveLab/AgiBot-World) | Robotics | 200 |
| [OpenDV](https://github.com/OpenDriveLab/DriveAGI.git) | Autonomous Driving | 200 |
| [Ego-Exo4D](https://ego-exo4d-data.org/) | Ego-centric | 200 |

## Dataset Summary

- **Dataset Size**: 600 video samples
- **Video Format**: MP4 files with various processing variants
- **Annotations**: Text captions for each video
- **Processing Variants**: Blur, Canny edge detection, Depth estimation, SAM2 segmentation

## File Organization

```text
physical-ai-bench-transfer/
├── videos/ # Original video files
├── blur/ # Blur-processed videos
├── canny/ # Edge detection videos
├── depth_vids/ # Depth estimation videos
├── depth_npzs/ # Depth estimation numpy arrays
├── sam2_vids/ # SAM2 segmentation videos
├── sam2_pkls/ # SAM2 segmentation pickle files
└── captions/ # JSON files with video descriptions
```

## Citation

If you use Physical AI Bench in your research, please cite:

```bibtex
@misc{zhou2025paibenchcomprehensivebenchmarkphysical,
title={PAI-Bench: A Comprehensive Benchmark For Physical AI},
author={Fengzhe Zhou and Jiannan Huang and Jialuo Li and Deva Ramanan and Humphrey Shi},
year={2025},
eprint={2512.01989},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2512.01989},
}
```