amathislab/ESK_motion_generation

--- language: - en license: cc-by-4.0 size_categories: - 10K<n<100K tags: - motion - human-motion - motion-generation - 3d_pose - hand_pose - body_pose - language - esk pretty_name: ESK-MotionGen --- [Paper](https://arxiv.org/abs/2506.01608) | [GitHub](https://github.com/amathislab/EPFL-Smart-Kitchen) # 🍳 EPFL‑Smart‑Kitchen: Motion Generation Assets ## 📚 Introduction This folder accompanies motion generation experiments on EPFL‑Smart‑Kitchen‑30 (ESK‑30). The goal is to generate realistic human motion sequences (e.g., body and hands in 3D) optionally conditioned on text, activity context, or past motion. The assets here focus on kinematic inputs/outputs (body, hands, gaze) from ESK‑30 to enable training, fine‑tuning, and evaluation of motion generators. - Source dataset: ESK‑30 with densely annotated activities and synchronized 3D kinematics - Modalities typically used: body pose, hand pose, and gaze - Typical tasks: unconditional/conditional motion synthesis, continuation, and in‑betweening If you need the full dataset description and licensing, please refer to the paper and upstream repository linked above. ## 📦 What’s in this folder This directory contains split archives for motion data and (optionally) pretrained checkpoints: - `motion_data.z01`, `motion_data.z02`, …: multipart archive with motion training/eval data - `checkpoints.z01` (optional): multipart archive with example pretrained weights You’ll first reconstruct and unzip these archives locally. And you can see the following folders: ``` ESK_motion_generation ├── motion_data | ├── new_joint_vecs | └── holo_images ├── evaluators | ├── Comp_v6_KLD005 | └── Decomp_SP001_SM001_H512 | └── text_mot_match_[TEXT_TYPE] ├── checkpoints | ├── fullbody_[TOKENIZER] | └── fullbody_[BASELINE] | ├── fullbody_image_[TOKENIZER] | └── fullbody_image_[BASELINE] └── README.md ``` ### 🔓 Extract the archives (Linux) ```bash # Reconstruct and extract motion data cat motion_data.z* > motion_data.zip unzip motion_data.zip -d motion_data unzip evaluators.zip -d evaluators # (Optional) Reconstruct and extract pretrained checkpoints cat checkpoints.z* > checkpoints.zip unzip checkpoints.zip -d checkpoints ``` ## 🌟 Citations Please cite our work: ``` @misc{bonnetto2025epflsmartkitchen, title={EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models}, author={Andy Bonnetto and Haozhe Qi and Franklin Leong and Matea Tashkovska and Mahdi Rad and Solaiman Shokur and Friedhelm Hummel and Silvestro Micera and Marc Pollefeys and Alexander Mathis}, year={2025}, eprint={2506.01608}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2506.01608}, } ``` ❤️ Acknowledgments This work was funded by EPFL, Swiss SNF grant (320030-227871), Microsoft Swiss Joint Research Center, and a Boehringer Ingelheim Fonds PhD stipend (H.Q.). We thank the Brain Mind Institute for hardware support and the Neuro‑X Institute for services.language: - 英语 license: 知识共享署名4.0（CC BY 4.0） size_categories: - 10000 < 样本量 < 100000 tags: - 动作 - 人体动作 - 动作生成（motion generation） - 三维姿态（3D pose） - 手部姿态（hand_pose） - 躯干姿态（body_pose） - 语言 - ESK pretty_name: ESK-MotionGen [论文](https://arxiv.org/abs/2506.01608) | [GitHub](https://github.com/amathislab/EPFL-Smart-Kitchen) # 🍳 EPFL智能厨房：动作生成资源集 ## 📚 研究简介 本文件夹配套EPFL-Smart-Kitchen-30（以下简称ESK-30）上的动作生成（motion generation）实验。本项目旨在生成逼真的人体动作序列（例如三维躯干与手部姿态），支持可选地以文本、活动上下文或过往动作为条件进行生成。本资源集聚焦ESK-30中的运动学输入与输出（kinematic inputs/outputs）数据（躯干、手部、视线（gaze）），以助力动作生成（motion generation）模型的训练、微调与评估。 - 源数据集：ESK-30，包含密集标注的活动信息与同步三维运动学数据 - 常用模态：躯干姿态（body_pose）、手部姿态（hand_pose）与视线（gaze）追踪数据 - 典型任务：无条件/条件动作合成（unconditional/conditional motion synthesis）、动作延续与动作插补 如需完整数据集说明与授权信息，请参阅前文链接的论文与上游仓库。 ## 📦 目录内容说明 本目录包含动作数据分卷归档文件，以及（可选）预训练模型检查点（checkpoint）： - `motion_data.z01`、`motion_data.z02`……：存储动作训练/评估数据的多卷归档 - `checkpoints.z01`（可选）：存储示例预训练权重的多卷归档 请先在本地重建并解压这些归档文件，随后将看到以下目录结构： ESK_motion_generation ├── motion_data | ├── new_joint_vecs | └── holo_images ├── evaluators | ├── Comp_v6_KLD005 | └── Decomp_SP001_SM001_H512 | └── text_mot_match_[TEXT_TYPE] ├── checkpoints | ├── fullbody_[TOKENIZER] | └── fullbody_[BASELINE] | ├── fullbody_image_[TOKENIZER] | └── fullbody_image_[BASELINE] └── README.md ### 🔓 归档文件解压（Linux 系统） bash # 重建并解压动作数据 cat motion_data.z* > motion_data.zip unzip motion_data.zip -d motion_data unzip evaluators.zip -d evaluators # （可选）重建并解压预训练模型检查点（checkpoint） cat checkpoints.z* > checkpoints.zip unzip checkpoints.zip -d checkpoints ## 🌟 引用规范 请引用本研究： @misc{bonnetto2025epflsmartkitchen, title={EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models, author={Andy Bonnetto and Haozhe Qi and Franklin Leong and Matea Tashkovska and Mahdi Rad and Solaiman Shokur and Friedhelm Hummel and Silvestro Micera and Marc Pollefeys and Alexander Mathis}, year={2025}, eprint={2506.01608}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2506.01608}, } ❤️ 致谢 本研究受洛桑联邦理工学院（EPFL）、瑞士国家科学基金会（SNSF）项目编号320030-227871、微软瑞士联合研究中心，以及勃林格殷格翰基金博士奖学金（H.Q.）资助。感谢脑与思想研究所提供硬件支持，同时感谢Neuro-X研究所提供相关服务。