Multimodal Sensory Learning for Object Manipulation

<strong>Multimodal Manipulation Learning Database</strong> The dataset consists of data recordings for object manipulation with audio-tactile sensory feedback for object handover. It captures the auditory and tactile signals of a Kuka IIWA robot with an Allegro hand holding a plastic container containing different materials. The robot manipulates the container with vertical shaking and rotation motions. The data consists of force/pressure measurements on the Allegro hand using a Tekscan tactile skin sensor, auditory signals from a microphone, and the joints data of the IIWA robot and the Allegro hand joints. <strong>Dataset</strong> Each datafile is a rosbag file containing the data recording from one trial of a robot motion with one material, with rostopics on the following data: Kuka IIWA 7 Joint data: /iiwa/TorqueController/command /iiwa/eePose /iiwa/joint_states Allegro hand joint data: /allegro_hand_right/joint_states Tekscan sensor recording (tactile force/pressure sensor data on hand): /tekscan/frame Audio data (for microphone attached to hand): /audio/audio /audio/audio_info Experiment information: /trialInfo which contains: trial information (motion type, speed, etc.) start/stop of different phases of the trials <strong>Motion Types</strong> The database contains recordings for the robot executing two different motion types: vertical shaking of the object and rotation of the object. <strong>Materials</strong> The database contains recordings for 5 different material classes in the plastic container, as shown below: empty, vitamins, gummies, cornflakes, and rice. We used approximately the same volume of each material for each trial. We tested each material class and motion combination for a total of 10 different experimental conditions and collected 30 trials for each condition. The vertical motion dataset was entirely collected on 2021/08/25. The rotation dataset was split into two day. The empty, gummies and rice class data was collected on 2021/08/26. The vitamins and cornflakes classes were collected on 2021/09/13. <strong>Database Setup</strong> The database consists of the data in two formats: annotated ('annotated_bags_mml.zip') and unannotated/numbered filenames ('numbered_bags_mml.zip') datasets. The data in the two datasets are identical- the annotated filename dataset has the experimental descriptions in the filename directly (as described below). The annotated filenames dataset ('annotated_bags_mml.zip') consists of a single directory with all 300 rosbag datafiles (10 experimental conditions, 30 trials each). Each rosbag (<code>.bag</code>) is saved in the directory, with filename specified ('Date Recorded YYYYMMDD' + '_motion' + '_material' + '_trialID' + '.bag'). Motion Types are: {'vertical', 'rotation'}. Materials are: {'empty', 'cornflakes', 'gummies', 'rice', 'vitamins'}. For each experimental condition, there are 30 datafiles with trial IDs from 0-29. All data recordings for the vertical motion have filenames: '20210825_vertical_+ 'material' + 'trialID' +'.bag). For the rotation motion, the empty, gummy and rice classes have filenames: '20210826_rotation_+ 'material' + 'trialID' +'.bag). For cornflakes and vitamins classes, the filenames are: '20210913_rotation_+ 'material' + 'trialID' +'.bag). The numbered/unannotated file dataset ('numbered_bags_mml.zip') consists of the same 300 data files as in the annotated dataset except here the filenames are numbered '{000-299}.bag'. The directory contains a spreadsheet ('annotations.csv') listing the experimental descriptions for each file name. The columns of the xls spreadsheet are {'Bagfile name', 'Year', 'Month', 'Day', 'Motion/Movement (mvt_type)', 'Material', 'Trial ID'}, where {Year, Month, Day} refer to the date that trial data was collected (either 2021/08/25, 2021/08/26, or 2021/09/13).