IAS-Lab Collaborative Draping HAR dataset

Description This dataset contains movement data for several subjects performing actions related to human-robot collaboration in an industrial carbon fiber draping process, such as draping and collaborative transport of carbon fiber plies. The collected dataset has been used to train and evaluate skeleton-based Human Action Recognition (HAR) models developed to provide a simple and intuitive way for the human operator to interact with the robot, such as signaling start and stop of the process or requesting robot’s assistance with specific tasks (e.g., inspection of specific parts). Actions of interest The dataset includes gestures designed to provide a simple and intuitive way for the operator to interact with the robot (e.g., “OK/NEXT” and “POINT” actions), short duration actions related to the beginning and end of collaborative transport operations (e.g., “PICK” and “PLACE” actions), and long duration actions related to the draping process (e.g., “TRANSPORT” and “DRAPING” actions) or general movements of the operator (e.g., “REST” and “WALK”). The dataset also includes an additional unknown class (“UNKWN”) which represents various random movements that the operator might make during the collaborative process but that do not correspond to any of the actions of interest. Action ID Action Name Description A001 OK/NEXT Raise one arm to signal the robot to continue the draping process A002 POINT Point at a desired location with a straight right arm to trigger inspection A003 PICK Raise the ply to trigger the collaborative transport A004 PLACE Place the ply on the mold A005 TRANSPORT Collaborative transportation A006 DRAPE Manual draping of a ply A007 REST Resting position, mainly waiting for the robot to complete its task A008 WALK Walking across the workcell A009 UNKWN Operator movements not related to the draping process Dataset Data has been collected from 7 participants, 2 female and 5 male, average age 27 (SD=3.0). Each participant performed 6 repetitions of each of the 8 actions considered for a collaborative draping process, and 18 repetitions of random movements for the unknow class. This results in a collected dataset containing 462 trimmed samples, where each sample is a sequence of 3D skeletons lasting approximately 3 seconds, containing only one action being performed. For all samples, 3D skeletons were obtained by means of the camera network installed in the laboratory, providing: (i) 3D skeletons from each camera in the network and (ii) 3D skeletons obtained by fusing the detections from each camera with a tracking algorithm; the output of the tracking algorithm provides a 3D skeleton representation robust to occlusions. All the 3D skeletons acquired are composed of 15 joints, with 3D coordinates expressed with respect to the camera network reference frame. Skeleton data for all sequences are provided in the `skeleton_data` folder. The dataset consists of a folder for each action of interest, with a subfolder for each participant and an individual text file for each action repetition performed by the participant. The naming convention for these files follows a pattern of the type “AxxxPyyyRzzzCwww.skeleton”, where "Axxx" represents the action id, "Pyyy" represents the id assigned to the participant, "Rzzz" represents the repeat number, and "Cwww" represents the id of the camera from which the 3D skeleton is derived; a network of 4 cameras was used to acquire the data, so “C001” denotes the first camera, “C002” the second, and so on, while “C000” represents the 3D skeletons obtained by merging all views.Each of these files is provided as a “.skeleton” text files, similar to popular human action recognition dataset (e.g., NTU RGB+D and NTU RGB+D 120 action recognition datasets). In particular, each file includes a sequence of 3D skeletons with 25 joints following the OpenPose convention for joint numbering, but only the first 15 joints contain valid values since face keypoints were not considered during the acquisitions. Example python code to read/write and visualize the skeleton data is also provided in the `scripts` folder:```pythonpython3 plot_sequence.py --data_dir ../skeleton_data``` References Allegro, D., Terreran, M., & Ghidoni, S. (2023). METRIC—Multi-Eye to Robot Indoor Calibration Dataset. Information, 14(6), 314. https://doi.org/10.3390/info14060314 Terreran, M., Barcellona, L., & Ghidoni, S. (2023). A general skeleton-based action and gesture recognition framework for human–robot collaboration. Robotics and Autonomous Systems, 170, 104523. https://doi.org/10.1016/j.robot.2023.104523 Terreran, M., Lazzaretto, M., & Ghidoni, S. (2022, June). Skeleton-based action and gesture recognition for human-robot collaboration. In International Conference on Intelligent Autonomous Systems (pp. 29-45). Cham: Springer Nature Switzerland. Carraro, M., Munaro, M., Burke, J., & Menegatti, E. (2019). Real-time marker-less multi-person 3D pose estimation in RGB-depth camera networks. In Intelligent Autonomous Systems 15: Proceedings of the 15th International Conference IAS-15 (pp. 534-545). Springer International Publishing.### 数据集描述 本数据集包含多名受试人员在工业碳纤维铺覆工艺中开展人机协作相关动作的运动数据，例如碳纤维铺层的铺覆与协作转运。所采集的数据集已被用于训练与评估基于骨骼的人体动作识别（Human Action Recognition, HAR）模型，此类模型旨在为操作人员提供一种简洁直观的人机交互方式，例如发送流程启停信号，或请求机器人协助完成特定任务（如对指定部件进行检测）。 ## 关注动作 本数据集涵盖为实现操作人员与机器人的简洁直观交互而设计的手势动作（如"OK/NEXT"与"POINT"动作）、与协作转运流程起止相关的短时动作（如"PICK"与"PLACE"动作），以及与铺覆工艺相关的长时动作（如"TRANSPORT"与"DRAPE"动作）或操作人员的常规动作（如"REST"与"WALK"动作）。此外，数据集还包含一个额外的未知类别（"UNKWN"），用于表示操作人员在协作过程中产生的各类随机动作，此类动作不属于上述任一关注动作范畴。 ### 动作编号、动作名称及说明 | 动作编号 | 动作名称 | 动作说明 | | ---- | ---- | ---- | | A001 | OK/NEXT | 举起单臂，向机器人发出继续铺覆流程的信号 | | A002 | POINT | 伸直右臂指向目标位置，以触发检测任务 | | A003 | PICK | 抬起碳纤维铺层，启动协作转运 | | A004 | PLACE | 将铺层放置于模具之上 | | A005 | TRANSPORT | 协作转运 | | A006 | DRAPE | 手动铺覆碳纤维铺层 | | A007 | REST | 待机姿态，主要用于等待机器人完成任务 | | A008 | WALK | 在作业单元内行走 | | A009 | UNKWN | 与铺覆工艺无关的操作人员动作 | ## 数据集概况 本数据集的采集对象为7名受试人员，其中2名女性、5名男性，平均年龄27岁（标准差SD=3.0）。每名受试人员需针对协作铺覆流程中的8种关注动作各完成6次重复操作，并针对未知类别完成18次随机动作重复。最终采集得到的数据集共包含462个裁剪后的有效样本，每个样本为一段时长约3秒的三维骨骼序列，且仅包含单一种类的动作。所有样本的三维骨骼数据均通过实验室部署的相机网络采集得到，可提供两类数据：(i) 网络中每台相机单独采集的三维骨骼数据；(ii) 通过融合各相机检测结果与跟踪算法得到的三维骨骼数据，该跟踪算法的输出可生成抗遮挡的三维骨骼姿态表示。所有采集到的三维骨骼均包含15个关节，其三维坐标以相机网络参考坐标系为基准。 ## 骨骼数据说明 所有序列的骨骼数据均存放于`skeleton_data`文件夹中。数据集的组织结构为：每个关注动作对应一个独立文件夹，每个动作文件夹下包含受试人员子文件夹，每个受试人员子文件夹中包含该人员完成的每一次动作重复对应的独立文本文件。此类文件的命名规范遵循`AxxxPyyyRzzzCwww.skeleton`格式，其中：`Axxx`代表动作编号，`Pyyy`代表受试人员编号，`Rzzz`代表重复次数，`Cwww`代表采集该三维骨骼的相机编号。本数据集采用4台相机组成的网络进行数据采集，因此`C001`代表第一台相机，`C002`代表第二台相机，依此类推；而`C000`则代表融合所有相机视角得到的三维骨骼数据。所有文件均以`.skeleton`文本格式存储，与主流人体动作识别数据集（如NTU RGB+D与NTU RGB+D 120动作识别数据集）的格式一致。具体而言，每个文件包含一段遵循OpenPose关节编号约定的25关节三维骨骼序列，但由于采集过程未考虑面部关键点，仅前15个关节包含有效数值。 ## 示例代码 `scripts`文件夹中还提供了用于读取、写入并可视化骨骼数据的示例Python代码： python python3 plot_sequence.py --data_dir ../skeleton_data ## 参考文献 1. Allegro, D., Terreran, M., & Ghidoni, S. (2023). METRIC—Multi-Eye to Robot Indoor Calibration Dataset. *Information*, 14(6), 314. https://doi.org/10.3390/info14060314 2. Terreran, M., Barcellona, L., & Ghidoni, S. (2023). A general skeleton-based action and gesture recognition framework for human–robot collaboration. *Robotics and Autonomous Systems*, 170, 104523. https://doi.org/10.1016/j.robot.2023.104523 3. Terreran, M., Lazzaretto, M., & Ghidoni, S. (2022, June). Skeleton-based action and gesture recognition for human-robot collaboration. In *International Conference on Intelligent Autonomous Systems* (pp. 29-45). Cham: Springer Nature Switzerland. 4. Carraro, M., Munaro, M., Burke, J., & Menegatti, E. (2019). Real-time marker-less multi-person 3D pose estimation in RGB-depth camera networks. In *Intelligent Autonomous Systems 15: Proceedings of the 15th International Conference IAS-15* (pp. 534-545). Springer International Publishing.