Haoyuwu/MultiWorldData

--- license: apache-2.0 size_categories: - 100K<n<1M --- # MultiWorld Dataset ## Dataset Summary **MultiWorld** is a large-scale multi-agent multi-view video dataset collected for training video world models. It contains two complementary sources of data: 1. **It Takes Two Gameplay Dataset**: 100+ hours of real human gameplay from the cooperative action-adventure game *It Takes Two*, featuring dual-agent synchronized actions with distinct first-person viewpoints. 2. **RoboFactory Manipulation Dataset**: Multi-robot manipulation trajectories spanning 4 tasks with 2-4 agents and variable camera viewpoints, including both success and failure episodes. This dataset is the official release accompanying the paper *"MultiWorld: Scalable Multi-Agent Multi-View Video World Models"*. - **Homepage:** https://multi-world.github.io - **Repository:** https://github.com/CIntellifusion/MultiWorld - **Paper:** [arXiv:XXXX.XXXXX](https://arxiv.org/abs/XXXX.XXXXX) --- ## Dataset Details ### It Takes Two Gameplay | Property | Value | |----------|-------| | **Total Duration** | 100+ hours | | **Frame Rate** | 60 FPS | | **Resolution** | 480 × 960 | | **Agents** | 2 players | | **Viewpoints** | 2 distinct first-person views per episode | | **Actions** | Synchronized keyboard and mouse actions per agent | | **Modality** | RGB video + discrete/continuous action vectors | The gameplay videos are captured from real human players cooperating in the game. Each frame is accompanied by per-agent action labels capturing keyboard presses and mouse movements. ### RoboFactory Manipulation | Property | Value | |----------|-------| | **Tasks** | 4 multi-robot manipulation tasks | | **Agents** | 2–4 robots per task | | **Viewpoints** | Variable camera configurations per task | | **Resolution** | 256 × 320 | | **Success Episodes** | 1,000 per task | | **Failure Episodes** | 2,000 per task | | **Modality** | RGB video + robot proprioception + actions | Tasks include collaborative stacking, pushing, and pick-and-place scenarios. Both successful and failed trajectories are included to support learning robust world models and failure prediction. --- ### Possible Usage The dataset is intended for research in: - Video world models - Multi-agent video generation - Multi-view consistent video generation. --- ### Contact For questions about the dataset, please open an issue on the [GitHub repository](https://github.com/CIntellifusion/MultiWorld) or contact the authors.许可证：Apache-2.0 数据规模区间：100K<n<1M --- # MultiWorld 数据集 ## 数据集概览 **MultiWorld** 是专为训练视频世界模型（video world models）采集的大规模多智能体多视角视频数据集，包含两类互补数据源： 1. **《It Takes Two》游戏游玩数据集**：包含100+小时的真实人类游玩合作动作冒险游戏《It Takes Two》（中文译名为《双人成行》）的录像，具备双智能体同步动作与独特的第一人称视角。 2. **RoboFactory 操控数据集**：涵盖4项任务的多机器人操控轨迹，每项任务包含2-4个智能体，视角可变，同时包含成功与失败的轨迹片段。 本数据集为论文《MultiWorld: Scalable Multi-Agent Multi-View Video World Models》的官方配套发布版本。 - **项目主页**：https://multi-world.github.io - **代码仓库**：https://github.com/CIntellifusion/MultiWorld - **论文**：[arXiv:XXXX.XXXXX](https://arxiv.org/abs/XXXX.XXXXX) --- ## 数据集详情 ### 《It Takes Two》游戏游玩数据 | 属性 | 数值 | |--------------|-----------------------| | **总时长** | 100+小时 | | **帧率** | 60 FPS | | **分辨率** | 480 × 960 | | **智能体数量** | 2名玩家 | | **视角数量** | 每个片段包含2种不同的第一人称视角 | | **动作类型** | 每个智能体的同步键盘与鼠标操作 | | **模态** | RGB视频 + 离散/连续动作向量 | 该游玩录像采集自真实人类玩家的合作游戏过程，每一帧均附带对应每个智能体的动作标签，记录键盘按键与鼠标移动轨迹。 ### RoboFactory 操控数据 | 属性 | 数值 | |--------------|-----------------------| | **任务数量** | 4项多机器人操控任务 | | **智能体数量** | 每项任务包含2-4个机器人 | | **视角配置** | 每项任务采用可变相机配置 | | **分辨率** | 256 × 320 | | **成功轨迹数量** | 每项任务1000条 | | **失败轨迹数量** | 每项任务2000条 | | **模态** | RGB视频 + 机器人本体感知数据 + 动作 | 任务涵盖协同堆叠、推物、拾取与放置等场景，同时包含成功与失败的轨迹，以支持鲁棒世界模型与故障预测的学习。 --- ### 潜在应用场景 本数据集旨在支持以下方向的研究： - 视频世界模型 - 多智能体视频生成 - 多视角一致视频生成 --- ### 联系方式 如有关于本数据集的疑问，请在[GitHub代码仓库](https://github.com/CIntellifusion/MultiWorld)提交Issue，或联系论文作者。