360Motion-Dataset

# 360°-Motion Dataset

[Project page](http://fuxiao0719.github.io/projects/3dtrajmaster) | [Paper](https://drive.google.com/file/d/111Z5CMJZupkmg-xWpV4Tl4Nb7SRFcoWx/view) | [Code](https://github.com/kwaiVGI/3DTrajMaster)

### Acknowledgments
We thank Jinwen Cao, Yisong Guo, Haowen Ji, Jichao Wang, and Yi Wang from Kuaishou Technology for their help in constructing our 360°-Motion Dataset.


### News
- [2024-12] We release the V1 dataset (72,000 videos consists of 50 entities, 6 UE scenes, and 121 trajectory templates).

### Data structure

```
├── 360Motion-Dataset Video Number Cam-Obj Distance (m)
├── 480_720/384_672
├── Desert (desert) 18,000 [3.06, 13.39]
├── location_data.json
├── HDRI
├── loc1 (snowy street) 3,600 [3.43, 13.02]
├── loc2 (park) 3,600 [4.16, 12.22]
├── loc3 (indoor open space) 3,600 [3.62, 12.79]
├── loc11 (gymnastics room) 3,600 [4.06, 12.32]
├── loc13 (autumn forest) 3,600 [4.49 11.91]
├── location_data.json
├── RefPic
├── CharacterInfo.json
├── Hemi12_transforms.json
```

**(1) Released Dataset Information**

| Argument | Description |Argument | Description |
|-------------------------|-------------|-------------------------|-------------|
| **Video Resolution** | (1) 480×720 (2) 384×672 | **Frames/Duration/FPS** | 99/3.3s/30 |
| **UE Scenes** | 6 (1 desert+5 HDRIs) | **Video Samples** | (1) 36,000 (2) 36,000 |
| **Camera Intrinsics (fx,fy)** | (1) 1060.606 (2) 989.899 | **Sensor Width/Height (mm)** | (1) 23.76/15.84 (2) 23.76/13.365 |
| **Hemi12_transforms.json** | 12 surrounding cameras | **CharacterInfo.json** | entity prompts |
| **RefPic** | 50 animals | **1/2/3 Trajectory Templates** | 36/60/35 (121 in total) |
| **{D/N}_{locX}** | {Day/Night}_{LocationX} | **{C}_ {XX}_{35mm}** | {Close-Up Shot}_{Cam. Index(1-12)} _{Focal Length}|

**Note that** the resolution of 384×672 refers to our internal video diffusion resolution. In fact, we render the video at a resolution of 378×672 (aspect ratio 9:16), with a 3-pixel black border added to both the top and bottom.

**(2) Difference with the Dataset to Train on Our Internal Video Diffusion Model**

The release of the full dataset regarding more entities and UE scenes is still under our internal license check.

| Argument | Released Dataset | Our Internal Dataset|
|-------------------------|-------------|-------------------------|
| **Video Resolution** | (1) 480×720 (2) 384×672 | 384×672 |
| **Entities** | 50 (all animals) | 70 (20 humans+50 animals) |
| **Video Samples** | (1) 36,000 (2) 36,000 | 54,000 |
| **Scenes** | 6 | 9 (+city, forest, asian town) |
| **Trajectory Templates** | 121 | 96 |

**(3) Load Dataset Sample**

1. Change root path to `dataset`. We provide a script to load our dataset (video & entity & pose sequence) as follows. It will generate the sampled video for visualization in the same folder path.

```bash
python load_dataset.py
```

2. Visualize the 6DoF pose sequence via Open3D as follows.

```bash
python vis_trajecotry.py
```
After running the visualization script, you will get an interactive window like this. Note that we have converted the right-handed coordinate system (Open3D) to the left-handed coordinate system in order to better align with the motion trajectory of the video.

<img src="imgs/vis_objstraj.png" width="350" />

## Citation

```bibtex
@inproceedings{fu20243dtrajmaster,
author = {Fu, Xiao and Liu, Xian and Wang, Xintao and Peng, Sida and Xia, Menghan and Shi, Xiaoyu and Yuan, Ziyang and Wan, Pengfei and Zhang, Di and Lin, Dahua},
title = {3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation},
booktitle = {ICLR},
year = {2025}
}
```

## Contact

Xiao Fu: [e-mail]

# 360°运动数据集（360°-Motion Dataset）

[项目页面](http://fuxiao0719.github.io/projects/3dtrajmaster) | [论文](https://drive.google.com/file/d/111Z5CMJZupkmg-xWpV4Tl4Nb7SRFcoWx/view) | [代码](https://github.com/kwaiVGI/3DTrajMaster)

### 致谢
感谢快手科技（Kuaishou Technology）的曹金文、郭依松、纪浩文、王继超及王毅，为本360°运动数据集的构建提供协助。


### 最新动态
- [2024-12] 我们发布了V1版本数据集，包含72000段视频，涵盖50个实体、6个虚幻引擎（Unreal Engine，UE）场景及121个轨迹模板。

### 数据结构

├── 360Motion-Dataset 视频数量 相机-物体距离（米）
├── 480_720/384_672
├── Desert（沙漠） 18,000 [3.06, 13.39]
├── location_data.json
├── HDRI（高动态范围成像，High Dynamic Range Imaging，HDRI）
├── loc1（雪街） 3,600 [3.43, 13.02]
├── loc2（公园） 3,600 [4.16, 12.22]
├── loc3（室内开放空间） 3,600 [3.62, 12.79]
├── loc11（体操房） 3,600 [4.06, 12.32]
├── loc13（秋季森林） 3,600 [4.49, 11.91]
├── location_data.json
├── RefPic
├── CharacterInfo.json
├── Hemi12_transforms.json

**(1) 已发布数据集信息**

| 参数名 | 说明 |参数名 | 说明 |
|-------------------------|-------------|-------------------------|-------------|
| **视频分辨率** | (1) 480×720 (2) 384×672 | **帧数/时长/帧率** | 99帧/3.3秒/30 |
| **虚幻引擎场景** | 共6个，含1个沙漠场景与5个HDRI场景 | **视频样本数** | (1) 36,000 (2) 36,000 |
| **相机内参（fx, fy）** | (1) 1060.606 (2) 989.899 | **传感器宽/高（单位：毫米）** | (1) 23.76/15.84 (2) 23.76/13.365 |
| **Hemi12_transforms.json** | 12个环绕式相机 | **CharacterInfo.json** | 实体提示词 |
| **RefPic** | 涵盖50种动物 | **1/2/3类轨迹模板** | 分别为36、60、35个，总计121个 |
| **{D/N}_{locX}** | {白天/黑夜}_{位置X} | **{C}_ {XX}_{35mm}** | {特写镜头}_{相机索引（1-12）}_{焦距}|

**请注意**，384×672的分辨率指代我们内部视频扩散模型的训练分辨率。实际渲染时，视频分辨率为378×672（宽高比9:16），仅在上下两侧各添加3像素的黑边。

**(2) 与内部视频扩散模型训练数据集的差异**
完整数据集（包含更多实体与虚幻引擎场景）的发布仍需通过我们内部的许可审核。

| 参数名 | 已发布数据集 | 内部数据集|
|-------------------------|-------------|-------------------------|
| **视频分辨率** | (1) 480×720 (2) 384×672 | 384×672 |
| **实体数量** | 50个，均为动物 | 70个，含20个人类与50个动物 |
| **视频样本数** | (1) 36,000 (2) 36,000 | 54,000 |
| **场景数量** | 6 | 9个，新增城市、森林、亚洲城镇场景 |
| **轨迹模板数量** | 121 | 96 |

**(3) 数据集样本加载方法**
1. 将根路径切换至`dataset`目录。我们提供了如下脚本用于加载数据集（含视频、实体及姿态序列），该脚本将在同文件夹路径下生成用于可视化的采样视频。
bash
python load_dataset.py

2. 可通过Open3D可视化6自由度（6 Degrees of Freedom，6DoF）姿态序列，命令如下：
bash
python vis_trajecotry.py

运行可视化脚本后，将得到如所示的交互式窗口。请注意，为更好匹配视频中的运动轨迹，我们将Open3D默认的右手坐标系转换为了左手坐标系。

<img src="imgs/vis_objstraj.png" width="350" />

## 引用格式
bibtex
@inproceedings{fu20243dtrajmaster,
author = {Fu, Xiao and Liu, Xian and Wang, Xintao and Peng, Sida and Xia, Menghan and Shi, Xiaoyu and Yuan, Ziyang and Wan, Pengfei and Zhang, Di and Lin, Dahua},
title = {3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation},
booktitle = {国际学习表征会议（International Conference on Learning Representations，ICLR)},
year = {2025}
}

## 联系方式
傅晓：[e-mail]