Data from Interactive Robotic Moving Cable Segmentation by Motion Correlation

Manipulating tangled hoses, cables, or ropes can be challenging for both robots and humans. Humans often approach these perceptually demanding tasks by pushing or pulling tangled cables and observing the resulting motions. We follow a similar idea to aid robotic cable manipulation. We integrate visual and proprioceptive perception to segment a grasped cable by moving it even when the robot or the grasped cable sometimes perturb neighboring cables. We formulate the cable interactive segmentation problem in such a way that our methods do not require robot arm segmentation masks. Furthermore, a novel grasp sampling method can propose new cable grasp points given a partial cable segmentation to improve the segmentation via additional cable-robot interaction. We evaluate the proposed motion correlation (MCor) method on data sequences recorded by our physical robotic setup and show that the method outperforms an earlier motion segmentation (MSeg) baseline. Here we provide the dataset of image and gripper position sequences recorded by our robotic setup (Franka Emika Panda robot, Intel RealSense D456 RGB-D camera, mounted ropes or garden hoses). We provide the Cable Motion Correlation (CMCor) dataset in a single zip archive: CMCor.zip size: 42.9 GiB sha256sum: 231f8887f87a7190b522b9b6f97cc7b46f52fb4f8d2c76d67458c7c10e7b799e In addition to the complete dataset package, we provide a sample package with only one recorded (validation) sequence of the dataset: CMCor_sample.zip size: 447 MiB sha256sum: 1e6371127e7ed8f8240ef82dd228ee8d792e7729266f6ee209403b16851f1f94 Data format The dataset files are PNG images and JSON data files. The CMCor archive has two folders: CMCor/motion_correlation_annotations contains binary images of manually created ground truth cable segmentation masks for the last image of each data sequence. Its content has the structure dataset_split/sequence_name/cable_mask_DDDDDDDD.png, where dataset_split is either test or validation, DDDDDDDD is the index of the last image in the sequence, it is zero-padded to eight digits. CMCor/motion_correlation_buffers stores the recorded data sequences. Each sequence contains the following files: actions_gripper.json - action labels (key "action_buffer"), robot end-effector positions (key "ee_point_buffer") and other numerical data such as the camera focal length or camera matrix. grasped_cable_00000000.png - a binary mask image showing the grasped cable segment in the first image of the sequence rgb_DDDDDDDD.png - color image sequence depth_DDDDDDDD.png (in all sequences except 2024-08-06-*) - depth image sequence, single channel 16-bit PNG images with the depth stored in millimeters arm_DDDDDDDD.png (not in all sequences) - robot arm binary segmentation mask sequence Corresponding rgb, depth and arm images have the same DDDDDDDD index. The same index also points to the corresponding action label in action_buffer and gripper position in ee_point_buffer. The JSON file CMCor/multigrasp_sequences.json lists the groups of multigrasp sequences. The sequences in each multigrasp group (the lowest-level list of sequence names in the JSON file) were recorded by grasping and moving the same cable. The first sequence in each group used a grasp given by a human, all the following sequences used automatically proposed grasps.