HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands

Please cite: Han, M., Günay, S.Y., Schirner, G. et al. HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands. Intel Serv Robotics 13, 179–185 (2020). https://doi.org/10.1007/s11370-019-00293-8 Upper limb and hand functionality is critical to many activities of daily living, and the amputation of one can lead to significant functionality loss for individuals. From this perspective, advanced prosthetic hands of the future are anticipated to benefit from improved shared control between a robotic hand and its human user, but more importantly from the improved capability to infer human intent from multimodal sensor data to provide the robotic hand perception abilities regarding the operational context. Such multimodal sensor data may include various environment sensors including vision, as well as human physiology and behavior sensors including electromyography and inertial measurement units. A fusion methodology for environmental state and human intent estimation can combine these sources of evidence in order to help prosthetic hand motion planning and control. In this paper, we present a dataset of this type that was gathered with the anticipation of cameras being built into prosthetic hands, and computer vision methods will need to assess this hand-view visual evidence in order to estimate human intent. Specifically, paired images from human eye-view and hand-view of various objects placed at different orientations have been captured at the initial state of grasping trials, followed by paired video, EMG and IMU from the arm of the human during a grasp, lift, put-down, and retract style trial structure. For each trial, based on eye-view images of the scene showing the hand and object on a table, multiple humans were asked to sort in decreasing order of preference, five grasp types appropriate for the object in its given configuration relative to the hand. The potential utility of paired eye-view and hand-view images was illustrated by training a convolutional neural network to process hand-view images in order to predict eye-view labels assigned by humans.

请引用：Han, M., Günay, S.Y., Schirner, G. 等人. HANDS：一个用于模拟假肢手的人类抓取意图推断的多模态数据集. Intel Serv Robotics 13, 179–185 (2020). https://doi.org/10.1007/s11370-019-00293-8 上肢和手的功能对于许多日常生活活动至关重要，而其截肢可能导致个体功能损失。从这个角度来看，未来先进的假肢手有望从机器人手与人类用户之间的共享控制改善中受益，更重要的是从从多模态传感器数据中推断人类意图的能力提升中受益，从而为机器人手提供关于操作环境的感知能力。此类多模态传感器数据可能包括各种环境传感器，如视觉传感器，以及人类生理和行为传感器，如肌电图和惯性测量单元。环境状态与人类意图估计的融合方法可以将这些证据来源结合在一起，以协助假肢手运动规划和控制。在本论文中，我们呈现了一个此类数据集，该数据集的收集旨在预见假肢手内置摄像头的情况，计算机视觉方法需要评估此手视视觉证据以推断人类意图。具体而言，在抓取试验的初始状态，捕获了放置在不同方向的物体的各种物体的人类眼视和手视成对图像，随后是抓取、举起、放下和缩回风格试验结构的成对视频、肌电图和惯性测量单元。对于每个试验，基于显示在桌上手和物体的场景眼视图图像，要求多位人类按偏好递减的顺序对与手相对配置的物体适当的五种抓取类型进行排序。成对眼视和手视图像的潜在效用通过训练卷积神经网络处理手视图像以预测由人类分配的眼视图标签得到了说明。