Dataset: Audio-visual representations of object drops

As robots perform manipulation tasks and interact with objects, it is probable that they accidentally drop objects that subsequently bounce out of their visual fields (e.g., due to an inadequate grasp of an unfamiliar object). To enable robots to recover from such errors, we draw upon the concept of object permanence—objects remain in existence even when they are not being sensed (e.g., seen) directly. This dataset was created to address the above challenge. The data was collected by having a Kinova Gen3 Robot Arm repeatedly pick up a wooden cube (3cm x 3cm x 3cm) and release the cube 0.3 meters above a table surface. A 7-channel microphone array was used to record the impact sound from when the robot gripper opened to drop the cube. A camera mounted on the robot’s wrist observed the beginning of the cube’s trajectory before the cube bounced out of view, and a ceiling camera recorded the rest of the trajectory. Audio recordings were saved as WAV files, and the cube’s trajectory was saved as NumPy arrays, with two versions: the partial trajectory observed from the robot wrist camera and the complete trajectory obtained by merging two partial trajectories.

当机器人执行操作任务并与物体交互时，有可能会意外掉落物体，而该物体随后会跳出机器人的可视视野（例如，因对陌生物体握持不足所致）。为使机器人能够从这类错误中恢复，我们借鉴了客体永久性（object permanence）的概念：即物体即便未被直接感知（例如被直接观察），也依然持续存在。本数据集正是为解决上述挑战而构建。数据采集过程为：使用Kinova Gen3机械臂反复抓取一块尺寸为3cm×3cm×3cm的木质立方体，并将该立方体在桌面上方0.3米处释放。采用7通道麦克风阵列记录从机器人夹爪张开到立方体掉落过程中的撞击声。安装在机器人腕部的相机可在立方体跳出视野前，捕捉到其运动轨迹的初始片段；而顶部相机则记录了该轨迹的剩余部分。音频录制结果以WAV格式存储，而立方体的运动轨迹则以NumPy数组格式存储，共包含两种版本：由机器人腕部相机采集的局部轨迹，以及通过合并两段局部轨迹得到的完整轨迹。