EPFL-Smart-Kitchen-30 Annotations and Poses

# The EPFL-Smart-Kitchen-30

> ⚠️ videos and other collected data can be found at https://zenodo.org/records/15535461

Understanding behavior requires datasets that capture humans while carrying out complex tasks. The kitchen is an excellent environment for assessing human motor and cognitive function, as many complex actions are naturally exhibited in kitchens from chopping to cleaning. Here, we introduce the EPFL-Smart-Kitchen-30 dataset, collected in a noninvasive motion capture platform inside a kitchen environment. Nine static RGB-D cameras, inertial measurement units (IMUs) and one head-mounted HoloLens~2 headset were used to capture 3D hand, body, and eye movements. The EPFL-Smart-Kitchen-30 dataset is a multi-view action dataset with synchronized exocentric, egocentric, depth, IMUs, eye gaze, body and hand kinematics spanning 29.7 hours of 16 subjects cooking four different recipes. Action sequences were densely annotated with 33.78 action segments per minute. Leveraging this multi-modal dataset, we propose four benchmarks to advance behavior understanding and modeling through

1) a vision-language benchmark,

2) a semantic text-to-motion generation benchmark,

3) a multi-modal action recognition benchmark,

4) a pose-based action segmentation benchmark.

## General informations

* **Authors**: Andy Bonnetto 1, Haozhe Qi 1, Franklin Leong 1, Matea Tashkovska 1, Mahdi Rad 3, Solaiman Shokur 1,3, Friedhelm Hummel 1,4,5, Silvestro Micera 1,3, Marc Pollefeys 2,6, Alexander Mathis 1

* **Affiliation**:  1 École Polytechnique Fédérale de Lausanne (EPFL), Lausanne,  2 Microsoft,  3  Scuola Superiore Sant’Anna, Pisa,  4 Swiss Federal Institute of Technology Valais (EPFL Valais), Clinique Romande de Réadaptation, Sion,  5 University of Geneva Medical School, Geneva,  6 Eidgenössische Technische Hochschule (ETH), Zürich

* **Date of collection**: 05.2023 - 01.2024 (MM.YYYY - MM.YYYY)

* **Geolocation data**: Campus Biotech, Genève, Switzerland

* **Associated publication URL**: https://arxiv.org/abs/2506.01608

* **Funding**: Our work was funded by EPFL and Microsoft Swiss Joint Research Center and a Boehringer Ingelheim Fonds PhD stipend (H.Q.). We are grateful to the Brain Mind Institute for providing funds for the cameras and to the Neuro-X Institute for providing funds to annotate data.

## Dataset availability

* **License**: This dataset is released under the non-commercial [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode) license.

* **Citation**: Please cite the associated publication when using our data.

* **Repository URL**: https://github.com/amathislab/EPFL-Smart-Kitchen

* **Repository DOI**: 10.5281/zenodo.15551913

* **Dataset version**: v1

## Data and files overview

* **Data preparation**: unzip `Public_release_pose.zip`

* **Repository structure**:

```

Public_release_pose

├── README.md

├── train

|   ├── YH2002 (participant)

|   |   ├── 2023_12_04_10_15_23 (session)

|   |   |   ├── annotations

|   |   |   |   ├── action_annotations.xlsx

|   |   |   |   └── activity_annotations.json

|   |   |   ├── pose_3d

|   |   |   |   ├── pose3d_mano.csv

|   |   |   |   └── pose3d_smpl.csv

|   |   └── ...

|   └── ...  

└── test

    └── ...

```

* `train` and `test`: Contains the train and test data for the action recognition task, the actions segmentation task and the full-body motion generation task. These folders are structured in participants and sessions. Each session contains 2 modalities:

    * **annotations**: contains the action and activity annotation data.

    * **pose_3d**: 3D pose estimation for the hand (MANO) and for the body (SMPL).

> We refer the reader to the associated publication for details about data processing and tasks description.

### Naming conventions

* Exocentric camera names are the following : output0 , Aoutput0, Aoutput1, Aoutput2, Aoutput3, Boutput0, Boutput1, Boutput2, Boutput3.

* Participant are identified with YH and a random identifier, sessions are given by the date  and time of recording.

### File characteristics

* `action_annotations.xlsx`: Table with the following fields:

    * Start : start time in second of an action

    * End : end time in second of an action

    * Verbs : annotated verb for the segment

    * Nouns : annotated noun for the segment

    * Confusion: confusion of annotator for this segment (0-1)

* `activity_annotations.json` : Json file with the following fields:  

    * datetime : time of the annotation

    * video_file : annotated session (corresponds to all cameras)

    * annotations:

        * start: start time in second of an action

        * end : end time in second of an action

        * Activities : annotated activity

* `pose3d_mano` and `pose3d_smpl`: 3D pose estimation for the hand and body, contain the following fields:

    * kp3ds : 3D pose estimation (42 keypoints for the hands (left/rights) and 17 keypoints for the body)

    * left_poses/right_poses : pose parameters of the fitted mesh model

    * left_RH/right_RH : rotation matrices for the fitted mesh model

    * left_TH/right_TH : translation matrices for the fitted mesh model

    * left_shapes/right_shapes: shape parameters for the fitted mesh model

> We refer the reader to the associated publication for details about data processing and tasks description.

## Methodological informations

**Benchmark evaluation code**: Will be available soon

> We refer the reader to the associated publication for details about data processing and tasks description.

## Acknowledgements

Our work was funded by EPFL and Microsoft Swiss Joint Research Center and a Boehringer Ingelheim Fonds PhD stipend (H.Q.). We are grateful to the Brain Mind Institute for providing funds for the cameras and to the Neuro-X Institute for providing funds to annotate data

## Change log (DD.MM.YYYY)

[03.06.2025]: First data release !

# EPFL智能厨房-30（EPFL-Smart-Kitchen-30）

⚠️ 视频及其他采集数据可于 https://zenodo.org/records/15535461 获取

理解人类行为需要能够捕捉人类执行复杂任务过程的数据集。厨房是评估人类运动与认知功能的绝佳环境，因为从切配到清洁等诸多复杂动作均可在厨房场景中自然呈现。本文介绍EPFL-Smart-Kitchen-30数据集，该数据集采集于厨房环境内的无创运动捕捉平台。我们采用9台固定式RGB-D相机、惯性测量单元（Inertial Measurement Unit, IMU）以及1台头戴式HoloLens 2设备，采集手部、躯体及眼部的三维运动数据。EPFL-Smart-Kitchen-30是一个多视角动作数据集，包含同步的外视角、第一视角、深度图像、惯性测量单元数据、视线追踪数据、躯体与手部运动学数据，总时长29.7小时，涵盖16名受试者烹饪4种不同食谱的过程。该数据集的动作序列标注密度为每分钟33.78个动作片段。依托该多模态数据集，我们提出四项基准任务以推动行为理解与建模研究：
1) 视觉语言基准任务；
2) 语义文本到运动生成基准任务；
3) 多模态动作识别基准任务；
4) 基于姿态的动作分割基准任务。

## 基本信息

* **作者**：Andy Bonnetto 1, Haozhe Qi 1, Franklin Leong 1, Matea Tashkovska 1, Mahdi Rad 3, Solaiman Shokur 1,3, Friedhelm Hummel 1,4,5, Silvestro Micera 1,3, Marc Pollefeys 2,6, Alexander Mathis 1
* **所属机构**：1 洛桑联邦理工学院（École Polytechnique Fédérale de Lausanne, EPFL），瑞士洛桑；2 微软（Microsoft）；3 圣安娜高等学校，意大利比萨；4 瑞士联邦瓦莱理工学院（EPFL Valais）、瑞士罗曼德康复诊所，瑞士锡永；5 日内瓦大学医学院，瑞士日内瓦；6 瑞士联邦理工学院（Eidgenössische Technische Hochschule, ETH），瑞士苏黎世
* **数据采集时间**：2023年5月 - 2024年1月（MM.YYYY - MM.YYYY）
* **地理位置**：瑞士日内瓦生物技术园区（Campus Biotech）
* **关联论文URL**：https://arxiv.org/abs/2506.01608
* **资助情况**：本研究由洛桑联邦理工学院、微软瑞士联合研究中心以及勃林格殷格翰基金博士奖学金（H.Q.）资助。感谢脑与思维研究所为相机设备提供经费，以及Neuro-X研究所为数据标注提供经费支持。

## 数据集可用性

* **许可证**：本数据集采用非商业用途的[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode)协议发布。
* **引用规范**：使用本数据集时请引用关联论文。
* **代码仓库URL**：https://github.com/amathislab/EPFL-Smart-Kitchen
* **仓库DOI**：10.5281/zenodo.15551913
* **数据集版本**：v1

## 数据与文件概览

* **数据预处理**：解压`Public_release_pose.zip`
* **仓库结构**：

Public_release_pose
├── README.md
├── train
| ├── YH2002（受试者）
| | ├── 2023_12_04_10_15_23（采集会话）
| | | ├── annotations
| | | | ├── action_annotations.xlsx
| | | | └── activity_annotations.json
| | | ├── pose_3d
| | | | ├── pose3d_mano.csv
| | | | └── pose3d_smpl.csv
| | └── ...
| └── ...
└── test
└── ...

* `train`与`test`：分别包含动作识别、动作分割以及全身体运动生成任务的训练与测试数据。这些文件夹按照受试者与采集会话进行组织。每个采集会话包含两种模态数据：
* **annotations**：存储动作与活动标注数据。
* **pose_3d**：手部（MANO模型）与躯体（SMPL模型）的三维姿态估计结果。

> 有关数据处理与任务描述的详细信息，请参阅关联论文。

### 命名规范
* 外视角相机命名规则如下：output0、Aoutput0、Aoutput1、Aoutput2、Aoutput3、Boutput0、Boutput1、Boutput2、Boutput3。
* 受试者以YH加随机标识符进行标识，采集会话以录制的日期与时间命名。

### 文件特征
* `action_annotations.xlsx`：包含以下字段的表格：
* Start：动作的起始时间（单位：秒）
* End：动作的结束时间（单位：秒）
* Verbs：该片段的标注动词
* Nouns：该片段的标注名词
* Confusion：标注员对该片段的标注混淆度（取值范围0-1）
* `activity_annotations.json`：包含以下字段的JSON文件：
* datetime：标注时间
* video_file：本次标注对应的采集会话（对应所有相机的视角）
* annotations:
* start：动作的起始时间（单位：秒）
* end：动作的结束时间（单位：秒）
* Activities：标注的活动类别
* `pose3d_mano`与`pose3d_smpl`：分别为手部与躯体的三维姿态估计文件，包含以下字段：
* kp3ds：三维姿态估计结果（手部包含42个关键点，左/右手各21个；躯体包含17个关键点）
* left_poses/right_poses：拟合网格模型的姿态参数
* left_RH/right_RH：拟合网格模型的旋转矩阵
* left_TH/right_TH：拟合网格模型的平移矩阵
* left_shapes/right_shapes：拟合网格模型的形状参数

> 有关数据处理与任务描述的详细信息，请参阅关联论文。

## 方法学信息
**基准任务评估代码**：即将上线

> 有关数据处理与任务描述的详细信息，请参阅关联论文。

## 致谢
本研究由洛桑联邦理工学院、微软瑞士联合研究中心以及勃林格殷格翰基金博士奖学金（H.Q.）资助。在此感谢脑与思维研究所为相机设备提供经费支持，同时感谢Neuro-X研究所为数据标注工作提供经费支持。

## 更新日志（DD.MM.YYYY）
[03.06.2025]：首次发布数据集！