KlingTeam/GameFactory-Dataset

<div align="center"> <h1>GameFactory: Creating New Games with Generative Interactive Videos</h1> <p> <a href="https://yujiwen.github.io/gamefactory">[Project page]</a> <a href="https://arxiv.org/abs/2501.08325">[ArXiv]</a> <a href="https://huggingface.co/datasets/KwaiVGI/GameFactory-Dataset">[Dataset]</a> </p> </div> <div align="center"> **[Jiwen Yu^1*&dagger;](https://yujiwen.github.io/), [Yiran Qin^1*](https://github.com/IranQin), <br> [Xintao Wang^2&ddagger;](https://xinntao.github.io/), [Pengfei Wan²](https://scholar.google.com/citations?user=P6MraaYAAAAJ&hl=en), [Di Zhang²](https://openreview.net/profile?id=~Di_ZHANG3), [Xihui Liu^1&ddagger;](https://xh-liu.github.io/)** <br> ¹The University of Hong Kong ²Kuaishou Technology <br> &dagger;: Intern at KwaiVGI, Kuaishou Technology, *: Equal Contribution, &ddagger;: Corresponding Authors </div> ## 🚀 GF-Minecraft Dataset ### 1. Dataset Introduction The [**GF-Minecraft Dataset**](https://huggingface.co/datasets/KwaiVGI/GameFactory-Dataset) is designed to meet three key requirements for action-controllable video generation: 1. **Customizable actions** for cost-effective, large-scale data collection. 2. **Unbiased action sequences** to ensure diverse and low-probability action combinations. 3. **Diverse scenes** with textual descriptions to capture scene-specific physical dynamics. We use [**Minecraft**](https://minedojo.org/) as the data collection platform due to its comprehensive API, diverse open-world environment, and extensive action space. By executing predefined, randomized action sequences, we collected **70 hours of gameplay video** with action annotations. To enhance diversity, we preconfigured three biomes (forest, plains, desert), three weather conditions (clear, rain, thunder), and six times of day (e.g., sunrise, noon, midnight), resulting in more than **2,000 video clips**. Each clip contains **2,000 frames** and is paired with textual descriptions generated by the multimodal language model [**MiniCPM-V**](https://github.com/OpenBMB/MiniCPM-V) (examples shown below). This dataset provides a strong foundation for training action-controllable and generalizable game video generation models. ### 2. File Structure The **GF-Minecraft Dataset** can be downloaded from [**HuggingFace**](https://huggingface.co/datasets/KwaiVGI/GameFactory-Dataset). Upon download, the dataset will be organized as follows: ``` GF-Minecraft ├── data_2003 │ ├── part_aa │ ├── part_ab │ ├── part_ac │ ├── part_ad │ ├── part_ae │ └── part_af └── data_269.zip ``` To prepare the dataset for use, navigate to the `data_2003` folder and merge the parts into a single zip file using the following command: ```bash cat part_* > data_2003.zip ``` After extracting `data_2003.zip` and `data_269.zip`, the dataset will be organized as follows: ``` GF-Minecraft ├── data_2003 │ ├── annotation.csv │ ├── metadata │ │ ├── seed_1_part_1.json │ │ ├── seed_2_part_2.json │ │ ├── seed_3_part_3.json │ │ └── ... │ └── video │ ├── seed_1_part_1.mp4 │ ├── seed_2_part_2.mp4 │ ├── seed_3_part_3.mp4 │ └── ... └── data_269 ├── annotation.csv ├── metadata │ ├── seed_1_part_1.json │ ├── seed_2_part_2.json │ ├── seed_3_part_3.json │ └── ... └── video ├── seed_1_part_1.mp4 ├── seed_2_part_2.mp4 ├── seed_3_part_3.mp4 └── ... ``` We have also placed a file `sample-10.zip`([link](https://huggingface.co/datasets/KwaiVGI/GameFactory-Dataset/blob/main/GF-Minecraft/sample-10.zip)) in the `GF-Minecraft/` directory, which contains 5 video files and their corresponding metadata from both `data_2003/` and `data_269/` folders. This can be used for quick reference of the file format. #### Directory Details 1. **`annotation.csv`**: A CSV file containing the textual descriptions for all video clips. Each row corresponds to a video clip and includes the following columns: - **Original video name**: The name of the original video from which the clip is extracted. - **Start frame index**: The starting frame of the clip within the original video. - **End frame index**: The ending frame of the clip within the original video. - **Prompt**: The textual description associated with the clip. 2. **`metadata/`**: A folder containing JSON files with detailed metadata for each video clip. 3. **`video/`**: A folder containing the video files in `.mp4` format. The filenames (e.g., `seed_1_part_1.mp4`) correspond to their associated metadata and annotation records. #### Explanation of Dataset Parts - **`data_2003/`**: Contains the first part of the dataset, including both mouse movement actions and keyboard actions. - **`data_269/`**: Contains the second part of the dataset, similarly structured to `data_2003/`, but includes only keyboard actions. ### 3. JSON File Details #### Example JSON: ```json { "biome": "plains", "initial_weather": "rain", "start_time": "Sunset", "actions": { "0": { "ws": 2, "ad": 0, "scs": 3, "pitch": 0.0, "yaw": 0.0, "pitch_delta": 0.0, "yaw_delta": 0.0, "pos": [-228.5, 75.0, 246.4] }, "1": { "ws": 2, "ad": 1, "scs": 3, "pitch": 0.0, "yaw": 0.0, "pitch_delta": 0.0, "yaw_delta": 0.0, "pos": [-228.43, 75.0, 246.3] } } } ``` Each JSON file in the `metadata/` folder provides detailed metadata for a corresponding video clip. The **most important information in the JSON file is the `actions` field**, which describes the sequence of actions executed during the video. Below are the key details: - **actions**: A dictionary indexed by timestamps (e.g., `"0"`, `"1"`, etc.) representing the sequence of actions. Each video contains **2,000 frames**, and the actions for frames `1` to `1,999` correspond to the information in entries `"1"` to `"1999"` in the `actions` dictionary. The information in the `"0"` entry can be ignored as it does not correspond to any frame in the video. Each action entry includes: - **`ws`**: Encodes forward (`1`), backward (`2`), or no movement (`0`) along the W/S axis. - **`ad`**: Encodes left (`1`), right (`2`), or no movement (`0`) along the A/D axis. - **`scs`**: **`scs`**: Represents special control states, including jumping (space key, `1`), sneaking (shift key, `2`), sprinting (ctrl key, `3`), or no action (`0`). - **`pitch`**: The vertical angle of the camera. - **`yaw`**: The horizontal angle of the camera. - **`pitch_delta`** and **`yaw_delta`**: Changes in pitch and yaw between consecutive frames. These values need to be multiplied by `15` to convert them into degrees. - **`pos`**: A 3D coordinate `[x, y, z]` representing the agent's position in the game world. Other fields in the JSON file provide context for the actions: - **biome**: Specifies the biome type where the video was recorded (`plains`, `forest`, or `desert`). - **initial_weather**: Describes the weather condition at the start of the video (`clear`, `rain`, or `thunder`). - **start_time**: Indicates the time of day at the start of the video (`"Starting of a day"`, `"Noon, sun is at its peak"`, `"Sunset"`, `"Beginning of night"`, `"Midnight, moon is at its peak"`, `"Beginning of sunrise"`). ### 4. Useful scripts #### Invalid Jump and Collision Detection The `detection.py` script processes all JSON files in the specified `metadata` directory to detect and mark collisions and invalid jumps. The updated JSON files are saved in a new `metadata-detection` directory. Run the script with the following command: ```bash python detection.py --dir_name Your_Directory_Root ``` Ensure the directory specified in `--dir_name` contains the following subdirectories: - `video/`: Contains the video files. - `metadata/`: Contains the JSON files to be processed. #### Why Detect Invalid Jumps and Collisions? **Invalid Jumps**: During data collection, the agent sometimes receives a jump action for several consecutive frames. However, once the agent is in the air, the jump action becomes ineffective—this is what we call an "invalid jump." By detecting and removing these invalid jump actions in the metadata, we simplify the learning process for the model by ensuring it only processes valid and meaningful actions. **Collisions**: Collision detection provides additional information about the agent's interaction with the environment. Collisions, such as the agent hitting a wall or an obstacle, can be treated as a unique action signal. Incorporating this information into the metadata helps the model better understand environmental constraints and improves its ability to learn action dynamics. Of course, it is also possible to not provide this information and let the network learn it by itself. #### Action Visualization The provided script `visualize.py` allows users to annotate input videos with action information and save the output as an annotated video. Simply run the script directly to execute the visualization process: ```bash python visualize.py ``` The script uses a predefined action format, where actions are described as a list of entries. Each entry includes: - A frame range for which the action is active. - A string encoding the specific action details. - Optionally, a list of specific frames where the space key (jump) is pressed. For example `[[25, "0 0 0 0 0 0 0 0 0.5"], [77, "1 0 0 0 0 0 0 0 0"], "15 30 50"]`: - `[25, "0 0 0 0 0 0 0 0 0.5"]` indicates an action lasting until frame 25 with specific movement and control states. - `[77, "1 0 0 0 0 0 0 0 0"]` specifies a new action starting from frame 26 and lasting until frame 77. - `"15 30 50"` lists the frames where the space key (jump) is pressed, such as frames 15, 30, and 50. The action string consists of `"w s a d shift ctrl collision delta_pitch delta_yaw"`<div align="center"> <h1>GameFactory：利用生成式交互视频创作全新游戏（GameFactory: Creating New Games with Generative Interactive Videos）</h1> <p> <a href="https://yujiwen.github.io/gamefactory">[项目页面]</a> <a href="https://arxiv.org/abs/2501.08325">[ArXiv]</a> <a href="https://huggingface.co/datasets/KwaiVGI/GameFactory-Dataset">[数据集]</a> </p> </div> <div align="center"> **[于吉文^1*&dagger;](https://yujiwen.github.io/), [秦亦然^1*](https://github.com/IranQin), <br> [王鑫涛^2&ddagger;](https://xinntao.github.io/), [万鹏飞²](https://scholar.google.com/citations?user=P6MraaYAAAAJ&hl=en), [张迪²](https://openreview.net/profile?id=~Di_ZHANG3), [刘锡辉^1&ddagger;](https://xh-liu.github.io/)** <br> ¹香港大学 ²快手科技 <br> &dagger;: 快手科技KwaiVGI实习生，*: 共同第一作者，&ddagger;: 通讯作者 </div> ## 🚀 GF-Minecraft 数据集 ### 1. 数据集介绍 本[**GF-Minecraft 数据集**](https://huggingface.co/datasets/KwaiVGI/GameFactory-Dataset)旨在满足**动作可控视频生成（action-controllable video generation）**的三大核心需求： 1. **可自定义动作**，以实现低成本、大规模的数据采集； 2. **无偏动作序列**，确保动作组合的多样性与低概率性； 3. **带文本描述的多样化场景**，以捕捉场景特有的物理动态。 我们选用[**我的世界（Minecraft）**](https://minedojo.org/)作为数据采集平台，因其拥有完备的应用程序接口（API）、多样化的开放世界环境与丰富的动作空间。通过执行预定义的随机动作序列，我们采集了**70小时的游戏玩法视频**并附带动作标注。 为提升数据多样性，我们预设了三种生物群系（森林、平原、沙漠）、三种天气状况（晴朗、降雨、雷暴）以及六种时段（如日出、正午、午夜），最终生成超过**2000个视频片段**。每个片段包含**2000帧**，并配有由多模态大语言模型[**MiniCPM-V**](https://github.com/OpenBMB/MiniCPM-V)生成的文本描述（示例见下文）。本数据集为训练动作可控且泛化性强的游戏视频生成模型提供了坚实基础。 ### 2. 文件结构 **GF-Minecraft 数据集**可从[**HuggingFace**](https://huggingface.co/datasets/KwaiVGI/GameFactory-Dataset)下载。下载后，数据集的目录结构如下： GF-Minecraft ├── data_2003 │ ├── part_aa │ ├── part_ab │ ├── part_ac │ ├── part_ad │ ├── part_ae │ └── part_af └── data_269.zip 若要准备数据集以供使用，请进入`data_2003`文件夹，执行以下命令将分卷文件合并为单个压缩包： bash cat part_* > data_2003.zip 解压`data_2003.zip`与`data_269.zip`后，数据集的目录结构如下： GF-Minecraft ├── data_2003 │ ├── annotation.csv │ ├── metadata │ │ ├── seed_1_part_1.json │ │ ├── seed_2_part_2.json │ │ ├── seed_3_part_3.json │ │ └── ... │ └── video │ ├── seed_1_part_1.mp4 │ ├── seed_2_part_2.mp4 │ ├── seed_3_part_3.mp4 │ └── ... └── data_269 ├── annotation.csv ├── metadata │ ├── seed_1_part_1.json │ ├── seed_2_part_2.json │ ├── seed_3_part_3.json │ └── ... └── video ├── seed_1_part_1.mp4 ├── seed_2_part_2.mp4 ├── seed_3_part_3.mp4 └── ... 我们还在`GF-Minecraft/`目录中放置了`sample-10.zip`文件（[链接](https://huggingface.co/datasets/KwaiVGI/GameFactory-Dataset/blob/main/GF-Minecraft/sample-10.zip)），其中包含来自`data_2003/`与`data_269/`文件夹的5个视频文件及其对应的元数据，可用于快速参考数据集的文件格式。 #### 目录细节 1. **`annotation.csv`**：一个包含所有视频片段文本描述的CSV文件。每一行对应一个视频片段，包含以下字段： - **原视频名称**：该片段所提取自的原始视频文件名； - **起始帧索引**：片段在原始视频中的起始帧位置； - **结束帧索引**：片段在原始视频中的结束帧位置； - **提示文本**：该片段对应的文本描述。 2. **`metadata/`**：存储每个视频片段详细元数据的JSON文件所在文件夹。 3. **`video/`**：存储`.mp4`格式视频文件的文件夹。文件名（如`seed_1_part_1.mp4`）与其对应的元数据和标注记录一一对应。 #### 数据集分卷说明 - **`data_2003/`**：包含数据集的第一部分，涵盖鼠标移动动作与键盘动作。 - **`data_269/`**：包含数据集的第二部分，目录结构与`data_2003/`一致，但仅包含键盘动作。 ### 3. JSON文件详情 #### 示例JSON文件： json { "biome": "plains", "initial_weather": "rain", "start_time": "Sunset", "actions": { "0": { "ws": 2, "ad": 0, "scs": 3, "pitch": 0.0, "yaw": 0.0, "pitch_delta": 0.0, "yaw_delta": 0.0, "pos": [-228.5, 75.0, 246.4] }, "1": { "ws": 2, "ad": 1, "scs": 3, "pitch": 0.0, "yaw": 0.0, "pitch_delta": 0.0, "yaw_delta": 0.0, "pos": [-228.43, 75.0, 246.3] } } } `metadata/`文件夹中的每个JSON文件均为对应视频片段提供详细元数据。JSON文件中**最为核心的字段为`actions`**，其描述了视频播放过程中执行的动作序列。以下为各字段的详细说明： - **`actions`**：以时间戳（如`"0"`、`"1"`等）为索引的字典，用于表示动作序列。每个视频包含**2000帧**，第1至1999帧的动作对应`actions`字典中`"1"`至`"1999"`条目所存储的信息。`"0"`条目对应的信息可忽略，因其未对应视频中的任何帧。每个动作条目包含以下字段： - **`ws`**：编码W/S轴方向的移动状态，1表示前进，2表示后退，0表示无移动。 - **`ad`**：编码A/D轴方向的移动状态，1表示左移，2表示右移，0表示无移动。 - **`scs`**：表示特殊控制状态，1对应跳跃（空格键），2对应潜行（Shift键），3对应冲刺（Ctrl键），0表示无特殊操作。 - **`pitch`**：相机的垂直角度。 - **`yaw`**：相机的水平角度。 - **`pitch_delta`**与**`yaw_delta`**：相邻帧之间相机俯仰角与偏航角的变化量，需将该值乘以`15`以转换为角度值。 - **`pos`**：表示游戏内玩家角色的三维坐标`[x, y, z]`。 JSON文件中的其他字段用于提供动作的上下文信息： - **`biome`**：指定视频录制所在的生物群系类型（`plains`平原、`forest`森林或`desert`沙漠）。 - **`initial_weather`**：描述视频起始时刻的天气状况（`clear`晴朗、`rain`降雨或`thunder`雷暴）。 - **`start_time`**：指示视频起始时刻的时段，可选值包括`"Starting of a day"`（一日伊始）、`"Noon, sun is at its peak"`（正午，太阳直射）、`"Sunset"`（日落）、`"Beginning of night"`（夜幕降临）、`"Midnight, moon is at its peak"`（午夜，月朗中天）、`"Beginning of sunrise"`（日出时分）。 ### 4. 实用脚本 #### 无效跳跃与碰撞检测 `detection.py`脚本可处理指定`metadata`目录下的所有JSON文件，检测并标记无效跳跃与碰撞事件，处理后的JSON文件将保存至新建的`metadata-detection`目录中。 执行该脚本的命令如下： bash python detection.py --dir_name Your_Directory_Root 请确保`--dir_name`指定的根目录包含以下子文件夹： - `video/`：包含视频文件。 - `metadata/`：包含待处理的JSON元数据文件。 #### 为何需要检测无效跳跃与碰撞？ **无效跳跃**：在数据采集过程中，玩家角色有时会收到连续多帧的跳跃动作指令，但当角色处于空中时，跳跃指令将不再生效，此类情况即为“无效跳跃”。通过在元数据中检测并移除无效跳跃动作，可确保模型仅处理有效且有意义的动作，从而简化模型的学习流程。 **碰撞事件**：碰撞检测可提供玩家角色与环境交互的额外信息。例如角色撞击墙体或障碍物的碰撞事件，可被视为一种独特的动作信号。将此类信息纳入元数据中，有助于模型更好地理解环境约束，提升其学习动作动态的能力。当然，也可以选择不提供该信息，让模型自行学习。 #### 动作可视化 `visualize.py`脚本可将动作信息标注至输入视频中，并将标注后的视频保存为输出文件。直接运行该脚本即可执行可视化流程： bash python visualize.py 该脚本采用预定义的动作格式，动作以条目列表的形式进行描述，每个条目包含以下内容： - 动作生效的帧范围。 - 编码具体动作细节的字符串。 - 可选字段：按下空格键（跳跃）的具体帧列表。 例如`[[25, "0 0 0 0 0 0 0 0 0.5"], [77, "1 0 0 0 0 0 0 0 0"], "15 30 50"]]`： - `[25, "0 0 0 0 0 0 0 0 0.5"]`表示一个动作将持续至第25帧，并包含特定的移动与控制状态。 - `[77, "1 0 0 0 0 0 0 0 0"]`表示从第26帧开始执行新的动作，持续至第77帧。 - `"15 30 50"`列出了按下空格键（跳跃）的帧，例如第15、30、50帧。 动作字符串共包含9个参数，依次为：W轴移动、S轴移动、A轴移动、D轴移动、潜行（Shift）、冲刺（Ctrl）、碰撞事件、俯仰角变化量、偏航角变化量。