IAS-Lab Collaborative Draping HAR dataset

Description

This dataset contains movement data for several subjects performing actions related to human-robot collaboration in an industrial carbon fiber draping process, such as draping and collaborative transport of carbon fiber plies. The collected dataset has been used to train and evaluate skeleton-based Human Action Recognition (HAR) models developed to provide a simple and intuitive way for the human operator to interact with the robot, such as signaling start and stop of the process or requesting robot’s assistance with specific tasks (e.g., inspection of specific parts).

Actions of interest

The dataset includes gestures designed to provide a simple and intuitive way for the operator to interact with the robot (e.g., “OK/NEXT” and “POINT” actions), short duration actions related to the beginning and end of collaborative transport operations (e.g., “PICK” and “PLACE” actions), and long duration actions related to the draping process (e.g., “TRANSPORT” and “DRAPING” actions) or general movements of the operator (e.g., “REST” and “WALK”). The dataset also includes an additional unknown class (“UNKWN”) which represents various random movements that the operator might make during the collaborative process but that do not correspond to any of the actions of interest.

Action ID
Action Name
Description

A001
OK/NEXT
Raise one arm to signal the robot to continue the draping process

A002
POINT
Point at a desired location with a straight right arm to trigger inspection

A003
PICK
Raise the ply to trigger the collaborative transport

A004
PLACE
Place the ply on the mold

A005
TRANSPORT
Collaborative transportation

A006
DRAPE
Manual draping of a ply

A007
REST
Resting position, mainly waiting for the robot to complete its task

A008
WALK
Walking across the workcell

A009
UNKWN
Operator movements not related to the draping process

Dataset

Data has been collected from 7 participants, 2 female and 5 male, average age 27 (SD=3.0). Each participant performed 6 repetitions of each of the 8 actions considered for a collaborative draping process, and 18 repetitions of random movements for the unknow class. This results in a collected dataset containing 462 trimmed samples, where each sample is a sequence of 3D skeletons lasting approximately 3 seconds, containing only one action being performed. For all samples, 3D skeletons were obtained by means of the camera network installed in the laboratory, providing: (i) 3D skeletons from each camera in the network and (ii) 3D skeletons obtained by fusing the detections from each camera with a tracking algorithm; the output of the tracking algorithm provides a 3D skeleton representation robust to occlusions. All the 3D skeletons acquired are composed of 15 joints, with 3D coordinates expressed with respect to the camera network reference frame.

Skeleton data for all sequences are provided in the `skeleton_data` folder. The dataset consists of a folder for each action of interest, with a subfolder for each participant and an individual text file for each action repetition performed by the participant. The naming convention for these files follows a pattern of the type “AxxxPyyyRzzzCwww.skeleton”, where "Axxx" represents the action id, "Pyyy" represents the id assigned to the participant, "Rzzz" represents the repeat number, and "Cwww" represents the id of the camera from which the 3D skeleton is derived; a network of 4 cameras was used to acquire the data, so “C001” denotes the first camera, “C002” the second, and so on, while “C000” represents the 3D skeletons obtained by merging all views.Each of these files is provided as a “.skeleton” text files, similar to popular human action recognition dataset (e.g., NTU RGB+D and NTU RGB+D 120 action recognition datasets). In particular, each file includes a sequence of 3D skeletons with 25 joints following the OpenPose convention for joint numbering, but only the first 15 joints contain valid values since face keypoints were not considered during the acquisitions.

Example python code to read/write and visualize the skeleton data is also provided in the `scripts` folder:```pythonpython3 plot_sequence.py --data_dir ../skeleton_data```

References

Allegro, D., Terreran, M., & Ghidoni, S. (2023). METRIC—Multi-Eye to Robot Indoor Calibration Dataset. Information, 14(6), 314. https://doi.org/10.3390/info14060314

Terreran, M., Barcellona, L., & Ghidoni, S. (2023). A general skeleton-based action and gesture recognition framework for human–robot collaboration. Robotics and Autonomous Systems, 170, 104523. https://doi.org/10.1016/j.robot.2023.104523

Terreran, M., Lazzaretto, M., & Ghidoni, S. (2022, June). Skeleton-based action and gesture recognition for human-robot collaboration. In International Conference on Intelligent Autonomous Systems (pp. 29-45). Cham: Springer Nature Switzerland.

Carraro, M., Munaro, M., Burke, J., & Menegatti, E. (2019). Real-time marker-less multi-person 3D pose estimation in RGB-depth camera networks. In Intelligent Autonomous Systems 15: Proceedings of the 15th International Conference IAS-15 (pp. 534-545). Springer International Publishing.