HANDS Dataset - Multimodal Dataset for Prosthetic Grasp

Limb deficiency severely affects the daily lives of amputees and drives research efforts to provide functional robotic prosthetic hands. Convolutional neural network-based computer vision systems for the classification and control of grasp have received increased attention as a method to replace or compliment physiological signals. The head- or eyeglass- mounted cameras have been commonly utilized for image data collection. Recently, placement of the camera into the palm of the prosthetic hand has become more popular, in part because it removes the need for a second device to be worn by the user. However, the grasp type labeled from the eye and hand perspective may differ as object shapes are not always symmetric; this mismatch in labels may be an artifact of perspective however approaching an object from different orientations may also necessitate different grasps. This study proposes a novel data collection procedure for representing this difference in a realistic way and presents a methodology to analyze the collected data. In particular, a dataset (called HANDS dataset) containing synchronous images from eye and hand view is labelled by multiple individuals from the eye perspective. The hand-perspective data is then used to train the convolutional neural network after augmentation and pre-processing. Added complexity results from rotating the objects to obtain images from different approaching orientations. Electromyogram (EMG) activity and dynamic data from the upper arm are also collected for the possible combination of multiple systems.