kiratan/robocasa-pnp-mimicgen

---
license: mit
task_categories:
- robotics
- reinforcement-learning
pretty_name: RoboCasa PnP MimicGen
configs:
- config_name: metadata
data_files:
- split: train
path: metadata.csv
- config_name: actions
data_files:
- split: train
path: actions.csv
- config_name: subtasks
data_files:
- split: train
path: subtasks.csv
---

# RoboCasa PnP MimicGen

This dataset contains two RoboCasa MimicGen pick-and-place datasets in the original HDF5 format:

- `PnPCabToCounter`
- `PnPCounterToSink`

The raw HDF5 files are stored under `raw/`.

## Tables

- `metadata.csv`: one row per demonstration.
- `actions.csv`: one row per timestep with `action_0` through `action_11`, plus timestep-level gripper joint positions.
- `subtasks.csv`: one row per demonstration with the computed pick / place split boundary.

## Subtask Split Table

`subtasks.csv` adds a separate viewer table for the pick / place segmentation used for downstream conversion.

Main columns:

```text
task
demo_id
instruction
pick_start_idx
pick_end_idx
pick_num_samples
pick_instruction
place_start_idx
place_end_idx
place_num_samples
place_instruction
grasp_idx
release_idx
```

The current split heuristic uses:

- `action_6` sign changes for grasp / release candidates
- target object `obj` from `ep_meta.object_cfgs`
- RoboCasa `object-state` block layout:
- `obj_pos (3)`
- `obj_quat (4)`
- `obj_to_eef_pos (3)`
- `obj_to_eef_quat (4)`
- `pick_end_idx` is selected after grasp once the object follows the end-effector and has been lifted
- `place_end_idx` is set to the final timestep of the episode

## Action Columns

The `actions.csv` table uses the original `actions` vector from the HDF5 files.

```text
action_0: normalized relative end-effector x translation
action_1: normalized relative end-effector y translation
action_2: normalized relative end-effector z translation
action_3: normalized relative end-effector roll / axis-angle x rotation
action_4: normalized relative end-effector pitch / axis-angle y rotation
action_5: normalized relative end-effector yaw / axis-angle z rotation
action_6: gripper command
action_7: mobile base x command
action_8: mobile base y command
action_9: mobile base yaw command
action_10: mobile base / controller mode flag
action_11: arm / controller mode flag
```

For these two PnP datasets, `action_7` through `action_10` are always `0.0`, and `action_11` is always `-1.0`.
The active motion commands are therefore `action_0` through `action_6`.

## Gripper Joint Columns

The HDF5 files also contain continuous gripper joint positions under `obs/robot0_gripper_qpos`.
Those values are exposed in the viewer tables as:

```text
actions.csv:
gripper_left_qpos
gripper_right_qpos

metadata.csv:
gripper_left_qpos_start
gripper_left_qpos_end
gripper_left_qpos_min
gripper_left_qpos_max
gripper_right_qpos_start
gripper_right_qpos_end
gripper_right_qpos_min
gripper_right_qpos_max
```

These are the raw left / right finger joint positions from the dataset, not a collapsed single opening scalar.

`metadata.csv` links each demonstration to the corresponding HDF5 file and internal HDF5 group via:

- `hdf5_path`
- `hdf5_internal_path`

## Raw Files

```text
raw/PnPCabToCounter/demo_gentex_im128_randcams.hdf5
raw/PnPCounterToSink/demo_gentex_im128_randcams.hdf5
```