PhysicalWorld

# PhysEditWorld **PhysEditWorld** is a large-scale dataset for studying physics-editable world models in interactive game environments. The dataset is built around a matched replay protocol: the same authored UE5 scene, initial state, character controller, action sequence, and camera policy are replayed under different physical configurations, making the effect of a physical edit directly comparable across rollouts. This ModelScope repository currently releases the `0520-lite` subset. It contains synchronized multi-view RGB videos, frame-level and event-level metadata, and explicit gravity configuration files. The release is intended for research on gravity-conditioned video generation, action-conditioned world models, physical consistency evaluation, and gravity-aware video-language understanding. ## Abstract Recent game world models can synthesize visually plausible action-conditioned rollouts, but their physical dynamics are often learned as implicit correlations rather than controllable variables. PhysEditWorld targets this gap by organizing game rollouts around explicit physical interventions. In the current release, gravity is used as the first editable physical parameter. Each replay group keeps scene content, player input, controller logic, and camera policy fixed while varying only the gravity configuration. This design supports controlled analysis of jump arcs, airtime, fall speed, landing timing, camera displacement, and object motion under counterfactual physical settings. The broader PhysEditWorld project is generated by a UE5 replay-and-rendering pipeline and is designed to support synchronized multimodal supervision. The `0520-lite` release on this branch provides an accessible public subset for initial research and benchmarking. ## Release Summary | Item | `0520-lite` release | | --- | ---: | | Samples | 7,300 | | RGB videos | 58,400 MP4 files | | Camera views per sample | 8 | | Frame metadata files | 7,300 | | Time/event metadata files | 7,300 | | Gravity configurations | 10 | | Git LFS payload size | about 1.6 TiB | Camera views in this release: ```text BK_Camera FL_Camera FP_Camera FR_Camera FW_Camera LF_Camera RT_Camera TP_Camera ``` Gravity configuration files: ```text G_0.05 G_0.1 G_0.25 G_0.5 G_0.75 G_1.0 G_5 G_10 G_15 G_20 ``` ## Repository Organization The default `master` branch contains the dataset card and a lightweight `metadata.csv` preview manifest for ModelScope indexing. The full released dataset is stored on the `0520-lite` branch and should be downloaded by explicitly setting `revision="0520-lite"`. ## Dataset Layout ```text master branch: README.md metadata.csv 0520-lite branch: 0520-lite/ Details.json Video/ <sample_id>/ BK_Camera.mp4 FL_Camera.mp4 FP_Camera.mp4 FR_Camera.mp4 FW_Camera.mp4 LF_Camera.mp4 RT_Camera.mp4 TP_Camera.mp4 Meta/ <sample_id>_meta_frame.csv <sample_id>_meta_time.csv PhysicalConfig/ G_<gravity>.json ALL_Physical_Config_Used.json PipelinePlan/ pipeline_plan.json pipeline_state.json UploadManifests/ <manifest_id>.json upload_registry.json ``` `Details.json` is the global sample index. Video paths, metadata paths, scene names, action sequence names, camera names, and physical configuration references are stored relative to the release root. Frame metadata aligns rendered frames with simulation state. Time metadata records event-level information over replay and capture time. `PhysicalConfig/` stores the explicit simulation parameters used for each gravity condition. ## Quickstart ### Install dependencies ```bash pip install -U modelscope pandas git lfs install ``` ### Download with the ModelScope SDK ```python from modelscope import dataset_snapshot_download dataset_dir = dataset_snapshot_download( dataset_id="GelerCAT/PhysicalWorld", revision="0520-lite", local_dir="./PhysicalWorld", ) print(dataset_dir) ``` ### Download with the ModelScope CLI ```bash modelscope download \ --dataset GelerCAT/PhysicalWorld \ --revision 0520-lite \ --local_dir ./PhysicalWorld ``` ### Download with Git LFS ```bash git clone --single-branch --branch 0520-lite \ https://www.modelscope.cn/datasets/GelerCAT/PhysicalWorld.git cd PhysicalWorld git lfs pull ``` For metadata-first inspection without downloading the full LFS payload, clone with LFS smudge disabled and then pull selected files: ```bash GIT_LFS_SKIP_SMUDGE=1 git clone --single-branch --branch 0520-lite \ https://www.modelscope.cn/datasets/GelerCAT/PhysicalWorld.git cd PhysicalWorld git lfs pull --include="0520-lite/Details.json,0520-lite/PhysicalConfig/*" git lfs pull --include="0520-lite/Meta/00000000_*,0520-lite/Video/00000000/*.mp4" ``` ModelScope also provides generated download snippets: :modelscope-code[]{type="sdk"} :modelscope-code[]{type="git"} ## Reading A Sample ```python import json from pathlib import Path import pandas as pd root = Path("PhysicalWorld/0520-lite") with open(root / "Details.json", "r") as f: details = json.load(f) samples = details["samples"] if isinstance(details, dict) and "samples" in details else details sample = samples[0] sample_id = sample["sample_id"] frame_meta = pd.read_csv(root / sample["frame_meta_file"]) time_meta = pd.read_csv(root / sample["time_meta_file"]) print("sample:", sample_id) print("scene:", sample.get("scene_name")) print("physical config:", sample.get("physical_config_name")) print("cameras:", sample.get("camera_names")) print(frame_meta.head()) ``` ## Recommended Use - Use matched replay groups when evaluating physical consistency. Avoid random clip-level splits that separate gravity variants from the same replay group. - Condition models on the explicit gravity label or the corresponding `PhysicalConfig` entry. - Compare generated motion within the same scene/action/camera setup whenever possible, since the dataset is designed for counterfactual physical comparisons. - Report the release branch (`0520-lite`) and the exact subset of samples used in experiments. ## Citation If you use PhysEditWorld in your research, please cite the paper and dataset release. Paper: [arXiv:2606.26694](https://arxiv.org/abs/2606.26694) ```bibtex @misc{2026physeditworld, title = {PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models}, author = {Hu, Bin and Ma, Yanwen and Huang, Jiehui and Zhang, Ziliang and Wu, Haoning and Zhang, Ruicheng and Li, Yaokun and Wang, Zijun and Zhang, Yuechen and Tseng, Chun-Mei and Li, Hanhui and Qian, Shengju and Zhou, Jun and Zhang, Kaipeng and Liang, Xiaodan and Jia, Jiaya and Li, Xiu}, year = {2026}, eprint = {2606.26694}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, doi = {10.48550/arXiv.2606.26694}, url = {https://arxiv.org/abs/2606.26694}, note = {Dataset release, branch: 0520-lite, \url{https://modelscope.cn/datasets/GelerCAT/PhysicalWorld}} } ``` ## License This dataset release is provided under the MIT License. The released branch contains processed dataset files for research use. Raw third-party UE5 scene assets are not included in this release.