Multimodal Sensory Learning for Object Manipulation

Multimodal Manipulation Learning Database The dataset consists of data recordings for object manipulation with audio-tactile sensory feedback for object handover. It captures the auditory and tactile signals of a Kuka IIWA robot with an Allegro hand holding a plastic container containing different materials. The robot manipulates the container with vertical shaking and rotation motions. The data consists of force/pressure measurements on the Allegro hand using a Tekscan tactile skin sensor, auditory signals from a microphone, and the joints data of the IIWA robot and the Allegro hand joints. Dataset Each datafile is a rosbag file containing the data recording from one trial of a robot motion with one material, with rostopics on the following data: Kuka IIWA 7 Joint data: /iiwa/TorqueController/command /iiwa/eePose /iiwa/joint_states Allegro hand joint data: /allegro_hand_right/joint_states Tekscan sensor recording (tactile force/pressure sensor data on hand): /tekscan/frame Audio data (for microphone attached to hand): /audio/audio /audio/audio_info Experiment information: /trialInfo which contains: trial information (motion type, speed, etc.) start/stop of different phases of the trials Motion Types The database contains recordings for the robot executing two different motion types: vertical shaking of the object and rotation of the object. Materials The database contains recordings for 5 different material classes in the plastic container, as shown below: empty, vitamins, gummies, cornflakes, and rice. We used approximately the same volume of each material for each trial. We tested each material class and motion combination for a total of 10 different experimental conditions and collected 30 trials for each condition. The vertical motion dataset was entirely collected on 2021/08/25. The rotation dataset was split into two day. The empty, gummies and rice class data was collected on 2021/08/26. The vitamins and cornflakes classes were collected on 2021/09/13. Database Setup The database consists of the data in two formats: annotated ('annotated_bags_mml.zip') and unannotated/numbered filenames ('numbered_bags_mml.zip') datasets. The data in the two datasets are identical- the annotated filename dataset has the experimental descriptions in the filename directly (as described below). The annotated filenames dataset ('annotated_bags_mml.zip') consists of a single directory with all 300 rosbag datafiles (10 experimental conditions, 30 trials each). Each rosbag (.bag) is saved in the directory, with filename specified ('Date Recorded YYYYMMDD' + '_motion' + '_material' + '_trialID' + '.bag'). Motion Types are: {'vertical', 'rotation'}. Materials are: {'empty', 'cornflakes', 'gummies', 'rice', 'vitamins'}. For each experimental condition, there are 30 datafiles with trial IDs from 0-29. All data recordings for the vertical motion have filenames: '20210825_vertical_+ 'material' + 'trialID' +'.bag). For the rotation motion, the empty, gummy and rice classes have filenames: '20210826_rotation_+ 'material' + 'trialID' +'.bag). For cornflakes and vitamins classes, the filenames are: '20210913_rotation_+ 'material' + 'trialID' +'.bag). The numbered/unannotated file dataset ('numbered_bags_mml.zip') consists of the same 300 data files as in the annotated dataset except here the filenames are numbered '{000-299}.bag'. The directory contains a spreadsheet ('annotations.csv') listing the experimental descriptions for each file name. The columns of the xls spreadsheet are {'Bagfile name', 'Year', 'Month', 'Day', 'Motion/Movement (mvt_type)', 'Material', 'Trial ID'}, where {Year, Month, Day} refer to the date that trial data was collected (either 2021/08/25, 2021/08/26, or 2021/09/13).

多模态操作学习数据库（Multimodal Manipulation Learning Database）
本数据集包含用于物体交接任务的、搭载声-触觉感官反馈的物体操作数据录制内容。该数据集采集了搭载Allegro灵巧手（Allegro Hand）的Kuka IIWA机器人（Kuka IIWA）握持装有不同物料的塑料容器时的听觉与触觉信号。机器人通过垂直振动与旋转动作对容器进行操作。数据涵盖以下内容：Allegro手上搭载的Tekscan触觉皮肤传感器（Tekscan tactile skin sensor）采集的力/压力测量数据、手部麦克风录制的听觉信号，以及Kuka IIWA机器人与Allegro灵巧手的关节数据。

数据集文件说明
每个数据文件均为rosbag文件（rosbag），对应单次机器人操作试验的录制数据，包含以下ROS话题（rostopic）数据：
1. Kuka IIWA 7轴关节数据：/iiwa/TorqueController/command、/iiwa/eePose、/iiwa/joint_states
2. Allegro手关节数据：/allegro_hand_right/joint_states
3. Tekscan传感器录制数据（手部触觉力/压力传感器数据）：/tekscan/frame
4. 听觉数据（手部搭载麦克风的录制信号）：/audio/audio、/audio/audio_info
5. 试验信息：/trialInfo，其中包含试验相关参数（动作类型、速度等）以及试验各阶段的启停标记。

动作类型
本数据库包含两种机器人执行的操作动作的录制数据：物体垂直振动与物体旋转。

物料类别
本数据库包含塑料容器内的5类不同物料的录制数据，分别为：空容器（empty）、维生素类物料（vitamins）、软糖（gummies）、玉米片（cornflakes）与大米（rice）。每轮试验中，我们使用体积近似一致的物料。针对每类物料与动作的组合开展试验，总计10种实验条件，每种条件下采集30轮试验数据。
垂直振动数据集的采集日期为2021/08/25。旋转数据集分两日采集：空容器、软糖与大米类物料的数据采集于2021/08/26；维生素类与玉米片类物料的数据采集于2021/09/13。

数据库构建方式
本数据库包含两种格式的数据集：带标注命名的数据集（"annotated_bags_mml.zip"）与无标注/仅编号命名的数据集（"numbered_bags_mml.zip"）。两种数据集包含完全一致的原始数据，仅标注版数据集的文件名直接包含实验描述信息（如下所述）。
标注版命名数据集（"annotated_bags_mml.zip"）仅包含一个目录，存放全部300个rosbag数据文件（10种实验条件，每种条件对应30轮试验）。每个rosbag文件的命名规则为：'录制日期YYYYMMDD' + '_motion' + '_物料' + '_试验ID' + '.bag'。其中动作类型集合为{'vertical', 'rotation'}，物料类别集合为{'empty', 'cornflakes', 'gummies', 'rice', 'vitamins'}。针对每种实验条件，包含30个数据文件，试验ID范围为0-29。
垂直动作的所有数据文件命名格式为：'20210825_vertical_' + '物料' + '_trialID' + '.bag'。
旋转动作中，空容器、软糖与大米类物料的数据文件命名格式为：'20210826_rotation_' + '物料' + '_trialID' + '.bag'。
玉米片与维生素类物料的旋转动作数据文件命名格式为：'20210913_rotation_' + '物料' + '_trialID' + '.bag'。
无标注/编号版数据集（"numbered_bags_mml.zip"）包含与标注版数据集完全一致的300个数据文件，仅文件名改为编号格式：'{000-299}.bag'。该目录下附带一份电子表格（"annotations.csv"），用于标注每个文件对应的实验信息。该电子表格的列包括：{'Bagfile name', 'Year', 'Month', 'Day', 'Motion/Movement (mvt_type)', 'Material', 'Trial ID'}，其中{Year, Month, Day}为试验数据的采集日期，可选值为2021/08/25、2021/08/26或2021/09/13。

搜集汇总

背景与挑战

背景概述

该数据集是一个多模态操作学习数据库，专注于机器人对象交接中的听觉-触觉感知学习。数据集包含300个数据文件，记录了Kuka IIWA机器人在执行垂直摇动和旋转运动时，操作装有五种不同材料的容器，同步采集触觉传感器、音频和关节数据。数据涵盖10种实验条件，每种重复30次试验，适用于多模态感知和机器人操作研究。