Deeptouch_manipulation_peg_in_hole

# Deeptouch_manipulation_peg_in_hole *ManipulationNet peg-in-hole on a Realman arm with Xense tactile sensing, 16 holes.* Bimanual peg-in-hole manipulation with vision-based tactile sensing, contributed to [Open-X-Tactile (OXT)](https://open-x-tactile.github.io/). The tasks follow the [ManipulationNet](https://manipulation-net.org/) Peg-in-Hole benchmark from its Physical Skills Track. The data format — key naming, pose convention, camera/tactile/action fields — is defined by OXT and FTP-1, **not** by this repository. For the format specification and for the other datasets in the collection, see: - OXT: <https://open-x-tactile.github.io/> - ManipulationNet: <https://manipulation-net.org/> ([task page](https://manipulation-net.org/tasks/peg_in_hole.html), [paper](https://arxiv.org/abs/2603.04363)) - FTP-1 data processing spec: <https://github.com/michaelyuancb/ftp1-policy/blob/main/data_processing/README.md> ## Download ```python from modelscope import snapshot_download # everything (97.0 GiB) snapshot_download("Deeptouch_lab/Deeptouch_manipulation_peg_in_hole", repo_type="dataset", local_dir="./peg_in_hole") # or just the holes you need snapshot_download("Deeptouch_lab/Deeptouch_manipulation_peg_in_hole", repo_type="dataset", local_dir="./peg_in_hole", allow_patterns=["*/hole_A1.zip", "*/hole_A2.zip"]) ``` ## Contents 16 tasks x 99 teleoperated episodes = **1,584 episodes / 209,318 frames** at a uniform 10 fps (~5.8 hours of manipulation). The tasks are drawn from the ManipulationNet Peg-in-Hole task set — 5 peg shapes x 4 clearance levels — less the four withdrawn for tactile gel damage (see [Notes on the data](#notes-on-the-data)). ### Task board The board follows the ManipulationNet Peg-in-Hole specification, which draws on the NIST Assembly Task Board (ATB) line but is a custom design rather than a NIST board: - **Board** — transparent acrylic, CNC machined to 20 µm. - **Pegs** — 304 stainless steel, CNC machined to 10 µm. - **Shapes** — circle, hexagon, triangle, rectangle, L-shape, spanning highly symmetric to highly asymmetric geometry. Symmetry matters: a circle needs no rotational alignment, an L-shape admits exactly one. - **Clearances** — 3 mm, 1 mm, 0.1 mm, 0.02 mm, from loose to extremely tight. | shape | 3 mm | 1 mm | 0.1 mm | 0.02 mm | arm | |---|---|---|---|---|---| | A circle | `hole_A1` | `hole_A2` | `hole_A3` | `hole_A4` | left | | B hexagon | `hole_B1` | `hole_B2` | `hole_B3` | — | left | | C triangle | `hole_C1` | `hole_C2` | `hole_C3` | — | left | | D rectangle | `hole_D1` | `hole_D2` | `hole_D3` | — | right | | E L-shape | `hole_E1` | `hole_E2` | `hole_E3` | — | right | The four dashes are the withdrawn holes: `hole_B4`, `hole_C4`, `hole_D4` and `hole_E4` were recorded on damaged tactile gel and are not part of this release. Clearance is the peg-to-hole gap; at 0.02 mm the fit is tight enough that vision alone cannot close the loop, which is where the tactile stream earns its place. That tightest column survives here only on the circle (`hole_A4`). Only the performing arm is exported — the idle arm moves about 1 degree per episode and carries no information. Note that the ManipulationNet benchmark scores a robot on executing the full 20 insertions live, binary per task. This dataset is the teleoperated demonstration counterpart: every episode is a human-operated success, intended for imitation learning rather than as a benchmark run. ## Format One zarr v2 store per hole, shipped as one uncompressed-container zip each (`Deeptouch_manipulation_peg_in_hole/hole_A1.zip` … `hole_E3.zip`, 4.8–10.1 GB). The chunks inside are already Blosc/zstd compressed, so the zip adds no second compression layer — which is also why **you do not need to unzip**: a zarr `ZipStore` reads the archive in place (see below). Time is the major axis: every episode is concatenated into one flat array, and episode boundaries live **only** in `meta/episode_ends` (cumulative end indices, last value == T). ``` hole_A1.zip # opens as the store below, no extraction needed hole_A1.zarr/ # wrapper dir, so archives unzipped side by side cannot collide data/ timestamps (T,) float64 seconds, restarts at 0 each episode, step 0.1 camera_main_rgb (T,224,224,3) uint8 front camera <arm>_wrist_camera_rgb (T,224,224,3) uint8 wrist camera <arm>_wrist_pose (T,6) float64 x,y,z (metres) + rotation vector (radians) <arm>_arm_joints (T,7) float64 radians <arm>_hand_joints (T,1) float64 gripper command, 0 = open, 1 = closed <arm>_hand_joints_idx (T,1) int64 constant 28 (FAAS slot for a 1-DoF gripper) <arm>_tactile_data_fingerpad (T,2,224,224,3) uint8 two tactile pad images <arm>_tactile_area_fingerpad (T,2) int64 constant [0,1] (MTTS thumb tip, index fingertip) <arm>_tactile_sensor_fingerpad (T,) <U10 constant "XenseG1-ws" <arm>_tactile_type_fingerpad (T,) <U5 constant "image" sub_task_instruction (T,) <U.. natural-language task instruction meta/ episode_ends (99,) int64 ``` `<arm>` is `left` for A/B/C and `right` for D/E. ### Reading it ```python import zarr, numpy as np # Reads straight out of the zip -- no unzip step. `path=` addresses the wrapper # directory inside the archive. z = zarr.open_group(zarr.ZipStore("hole_A1.zip", mode="r"), path="hole_A1.zarr", mode="r") # If you did extract it, open the directory instead: # z = zarr.open_group("hole_A1.zarr", mode="r") ends = np.asarray(z["meta/episode_ends"]) def episode(i): start = 0 if i == 0 else int(ends[i - 1]) stop = int(ends[i]) return dict( rgb=z["data/camera_main_rgb"][start:stop], # (T,224,224,3) tactile=z["data/left_tactile_data_fingerpad"][start:stop], # (T,2,224,224,3) joints=z["data/left_arm_joints"][start:stop], # (T,7) pose=z["data/left_wrist_pose"][start:stop], # (T,6) ) ep = episode(0) print(ep["rgb"].shape, str(z["data/sub_task_instruction"][0])) ``` ## Notes on the data **Tactile is vision-based.** The XenseG1-ws is a camera behind a deformable gel pad, so each pad is an RGB image stream, not a force array. Hence `tactile_type = "image"` and the `(T,2,H,W,3)` shape, with the two pads of the parallel gripper stacked as one `fingerpad` group. **Tactile area ids are `[0, 1]`.** FTP-1 aligns a two-finger parallel gripper with a human thumb-index pinch, so the pads carry the MTTS ids of the fingertips they stand in for: thumb tip (0) and index fingertip (1). Channel 0 is the left jaw and channel 1 the right jaw of the physical gripper. **The gripper channel is a command, not a measurement.** The robot has no gripper encoder, so `hand_joints` stores the binary open/close command that was sent. **Images are 224x224 and not aspect-preserving.** Source frames are 4:3 and were resized to square without letterboxing, matching the FTP-1 example datasets. Full resolution for 260 k frames x 5 image streams would be ~2 TB. **Rotation is a rotation vector.** Source end-effector orientation was a scalar-last unit quaternion, converted via `scipy`'s `Rotation.from_quat` / `as_rotvec`. Joint angles were degrees in the source and are radians here. **Frame rate is uniform 10 fps.** Part of the raw data was recorded at 30 fps and was decimated by taking every third frame; `timestamps` is rebuilt over the kept frames at a strict 0.1 s step. **Episodes are 99 per hole, not 100.** **Four of the benchmark's 20 holes were withdrawn for gel damage.** `hole_B4`, `hole_C4`, `hole_D4` and `hole_E4` were recorded on cracked tactile gel: the cracks are visible with the gripper open and do not heal between episodes, so those pads are partly measuring the sensor's own damage rather than contact. The thumb-tip pad (area 0) was already cracked at the first episode of all four, so no episode of those holes is unaffected and there was nothing to salvage by trimming. All four are **removed from this release** — 396 episodes and 51,078 frames — rather than shipped with a caveat, and the 16 that remain audit clean. Shape **A** (circle) therefore covers all four clearances; shapes **B**–**E** stop at 0.1 mm. Only the `*_tactile_data_fingerpad` arrays were affected — the camera streams, poses, joint angles and gripper commands in the withdrawn holes were fine — so if you need the geometry rather than the touch, ask and they can be republished separately. The audit behind this is in the OXT submission package (`notes/gel_integrity.json`, with `notes/gel_sheets/overview.png` showing every pad), and the tool that produced it compares each episode's open-gripper pad appearance against a clean reference pad for the same arm and finger, after removing any uniform brightness shift so a reseated sensor is not counted as damage. Re-run over this release it reports 0 of 16 holes affected. ## License and contact Please contact email:[e-mail]