mocapact-data

# MoCapAct Dataset
Control of simulated humanoid characters is a challenging benchmark for sequential decision-making methods, as it assesses a policy’s ability to drive an inherently unstable, discontinuous, and high-dimensional physical system. Motion capture (MoCap) data can be very helpful in learning sophisticated locomotion policies by teaching a humanoid agent low-level skills (e.g., standing, walking, and running) that can then be used to generate high-level behaviors. However, even with MoCap data, controlling simulated humanoids remains very hard, because this data offers only kinematic information. Finding physical control inputs to realize the MoCap-demonstrated motions has required methods like reinforcement learning that need large amounts of compute, which has effectively served as a barrier to entry for this exciting research direction.

In an effort to broaden participation and facilitate evaluation of ideas in humanoid locomotion research, we are releasing MoCapAct (Motion Capture with Actions), a library of high-quality pre-trained agents that can track over three hours of MoCap data for a simulated humanoid in the `dm_control` physics-based environment and rollouts from these experts containing proprioceptive observations and actions. MoCapAct allows researchers to sidestep the computationally intensive task of training low-level control policies from MoCap data and instead use MoCapAct's expert agents and demonstrations for learning advanced locomotion behaviors. It also allows improving on our low-level policies by using them and their demonstration data as a starting point.

In our work, we use MoCapAct to train a single hierarchical policy capable of tracking the entire MoCap dataset within `dm_control`.
We then re-use the learned low-level component to efficiently learn other high-level tasks.
Finally, we use MoCapAct to train an autoregressive GPT model and show that it can perform natural motion completion given a motion prompt.
We encourage the reader to visit our [project website](https://microsoft.github.io/MoCapAct/) to see videos of our results as well as get links to our paper and code.

## File Structure

The file structure of the dataset is:
```
├── all
│ ├── large
│ │ ├── large_1.tar.gz
│ │ ├── large_2.tar.gz
| │ ...
│ │ └── large_43.tar.gz
│ └── small
│ ├── small_1.tar.gz
│ ├── small_2.tar.gz
│ └── small_3.tar.gz
│
├── sample
│ ├── large.tar.gz
│ └── small.tar.gz
│
└── videos
├── full_clip_videos.tar.gz
└── snippet_videos.tar.gz
```

## MoCapAct Dataset Tarball Files
The dataset tarball files have the following structure:
- `all/small/small_*.tar.gz`: Contains HDF5 files with 20 rollouts per snippet. Due to file size limitations, we split the rollouts among multiple tarball files.
- `all/large/large_*.tar.gz`: Contains HDF5 files with 200 rollouts per snippet. Due to file size limitations, we split the rollouts among multiple tarball files.
- `sample/small.tar.gz`: Contains example HDF5 files with 20 rollouts per snippet.
- `sample/large.tar.gz`: Contains example HDF5 files with 200 rollouts per snippet.

The HDF5 structure is detailed in Appendix A.2 of the paper as well as https://github.com/microsoft/MoCapAct#description.

An example for loading and inspecting an HDF5 file in Python is:
```python
import h5py
dset = h5py.File("/path/to/small/CMU_083_33.hdf5", "r")
print("Expert actions from first rollout episode:")
print(dset["CMU_083_33-0-194/0/actions"][...])
```

## MoCap Videos
There are two tarball files containing videos of the MoCap clips in the dataset:
- `full_clip_videos.tar.gz` contains videos of the full MoCap clips.
- `snippet_videos.tar.gz` contains videos of the snippets that were used to train the experts.
Note that they are playbacks of the clips themselves, not rollouts of the corresponding experts.

# MoCapAct 数据集

对仿真人形角色的控制是序列决策方法领域极具挑战性的基准测试任务，因为该任务能够评估策略驱动一个本质上不稳定、非连续且高维物理系统的能力。动作捕捉（Motion Capture, MoCap）数据可通过为人形智能体教授诸如站立、行走、奔跑等低级技能，助力其学习复杂的运动策略，这些低级技能后续可用于生成高级行为。然而即便拥有MoCap数据，仿真人形角色的控制仍极具难度，因为这类数据仅提供运动学信息。要获取物理控制输入以实现MoCap演示的动作，需要依赖强化学习这类需要大量计算资源的方法，这实际上为该极具前景的研究方向设置了准入壁垒。

为了扩大参与范围并推动人形运动研究中各类思路的评估，我们发布了MoCapAct（Motion Capture with Actions）——这是一个包含高质量预训练智能体的工具库，这些智能体可在基于物理仿真的`dm_control`环境中，对仿真人形角色的超过三小时的MoCap数据进行跟踪，并生成包含本体感受观测与动作的专家轨迹。MoCapAct可让研究人员规避从MoCap数据中训练低级控制策略这一计算密集型任务，转而借助MoCapAct的专家智能体与演示数据来学习高级运动行为。同时，研究人员也可将这些专家智能体及其演示数据作为起点，对我们的低级策略进行优化改进。

在我们的研究中，我们借助MoCapAct训练了一个可在`dm_control`环境中跟踪完整MoCap数据集的分层策略。随后我们复用学到的低级组件，高效地完成了其他高级任务的学习。最后，我们使用MoCapAct训练了一个自回归GPT模型，并证明该模型可基于动作提示完成自然的动作补全任务。我们邀请读者访问我们的[项目网站](https://microsoft.github.io/MoCapAct/)查看成果视频，同时获取论文与代码的链接。

## 文件结构
数据集的文件结构如下：

├── all
│ ├── large
│ │ ├── large_1.tar.gz
│ │ ├── large_2.tar.gz
| │ ...
│ │ └── large_43.tar.gz
│ └── small
│ ├── small_1.tar.gz
│ ├── small_2.tar.gz
│ └── small_3.tar.gz
│
├── sample
│ ├── large.tar.gz
│ └── small.tar.gz
│
└── videos
├── full_clip_videos.tar.gz
└── snippet_videos.tar.gz

## MoCapAct 数据集压缩包文件
数据集压缩包的结构如下：
- `all/small/small_*.tar.gz`：包含HDF5格式文件，每个片段对应20条轨迹。受限于文件大小，我们将轨迹拆分至多个压缩包中。
- `all/large/large_*.tar.gz`：包含HDF5格式文件，每个片段对应200条轨迹。受限于文件大小，我们将轨迹拆分至多个压缩包中。
- `sample/small.tar.gz`：包含示例HDF5文件，每个片段对应20条轨迹。
- `sample/large.tar.gz`：包含示例HDF5文件，每个片段对应200条轨迹。

HDF5文件的具体结构可参见论文附录A.2，以及https://github.com/microsoft/MoCapAct#description页面。

以下为在Python中加载并检视HDF5文件的示例代码：
python
import h5py
dset = h5py.File("/path/to/small/CMU_083_33.hdf5", "r")
print("Expert actions from first rollout episode:")
print(dset["CMU_083_33-0-194/0/actions"][...])

## 动作捕捉视频
数据集中包含两个压缩包，内含MoCap片段的视频文件：
- `full_clip_videos.tar.gz`：包含完整MoCap片段的视频。
- `snippet_videos.tar.gz`：包含用于训练专家智能体的片段视频。
请注意，这些视频是片段本身的回放，而非对应专家智能体的轨迹回放。