leesangoh/dynamics-probing

---
license: mit
task_categories:
- robotics
tags:
- physics
- dynamics
- probing
- isaac-lab
- manipulation
- physrepa
- benchmark
pretty_name: Dynamics Probing Benchmark Dataset
size_categories:
- 10K<n<100K
---

# Dynamics Probing Benchmark Dataset

Physics-varied manipulation episodes for probing whether video encoders encode contact dynamics in their intermediate representations.

## Overview

This dataset contains **12,100 episodes** across 6 manipulation tasks in Isaac Lab simulation, with **per-episode physics randomization** (friction, mass, damping, restitution). Each episode includes dual-camera RGB video (384x384), robot state/action trajectories, and dense physics ground truth annotations.

**Purpose**: Dynamics probing — train linear probes on frozen video encoder features to predict physics quantities (contact force, friction coefficient, object velocity, etc.) and identify which layers encode contact dynamics.

## Tasks

| Task | Episodes | Policy | Physics Params | Description |
|------|----------|--------|----------------|-------------|
| **Push** | 1,500 | Scripted (Step0) | mass, obj friction, surface friction | Franka pushes cube on table, random direction |
| **Strike** | 3,000 | Scripted (Step0) | mass, obj friction, surface friction, damping, restitution | Franka strikes ball, random direction |
| **PegInsert** | 2,500 | RL (rl_games) | held friction, fixed friction, held mass | Peg-in-hole insertion with pre-grasped peg |
| **NutThread** | 2,500 | RL (rl_games) | held friction, fixed friction, held mass | Nut threading onto bolt with pre-grasped nut |
| **Drawer** | 2,000 | RL (RSL-RL) | handle friction, gripper friction, drawer damping | Open cabinet drawer |
| **Reach** | 600 | Scripted (Oracle) | None (negative control) | Move EE to random target position |

## Data Format (LeRobot V2)

Each task directory contains:
```
{task}/
├── data/chunk-000/
│ ├── episode_000000.parquet
│ ├── episode_000001.parquet
│ └── ...
├── videos/chunk-000/
│ ├── observation.images.image_0/ (table camera)
│ └── observation.images.image_1/ (wrist camera)
└── meta/
├── info.json
├── episodes.jsonl
├── tasks.jsonl
└── stats.json
```

### Per-frame columns (parquet)

| Column | Shape | Description |
|--------|-------|-------------|
| `observation.state` | (8,) | EE pose [x,y,z,r,p,y,pad,gripper] |
| `action` | (7,) | IK delta [dx,dy,dz,dr,dp,dy,gripper] |
| `observation.images.image_0` | video | Table camera RGB 384x384 |
| `observation.images.image_1` | video | Wrist camera RGB 384x384 |

### Physics Ground Truth (per-frame, task-specific)

**Common (all tasks)**:
- `physics_gt.ee_position` (3), `ee_orientation` (4), `ee_velocity` (3), `ee_angular_velocity` (3), `ee_acceleration` (3)

**Push/Strike**:
- `physics_gt.object_position` (3), `object_velocity` (3), `object_angular_velocity` (3), `object_acceleration` (3)
- `physics_gt.contact_finger_l_object_flag` (1), `contact_finger_l_object_force` (3)
- `physics_gt.contact_object_surface_flag` (1), `contact_object_surface_force` (3)
- `physics_gt.object_to_target_distance` (1), `object_on_surface` (1)
- Strike only: `physics_gt.ball_planar_travel_distance` (1)

**PegInsert**:
- `physics_gt.peg_position` (3), `peg_orientation` (4), `peg_velocity` (3), `peg_angular_velocity` (3)
- `physics_gt.hole_position` (3)
- `physics_gt.insertion_depth` (1), `peg_hole_lateral_error` (1)
- Pair contacts: `contact_finger_l_peg_*`, `contact_finger_r_peg_*`, `contact_peg_socket_*`

**NutThread**:
- `physics_gt.nut_position` (3), `nut_orientation` (4), `nut_velocity` (3), `nut_angular_velocity` (3)
- `physics_gt.bolt_position` (3), `bolt_orientation` (4)
- `physics_gt.axial_progress` (1), `nut_bolt_relative_angle` (1)
- Pair contacts: `contact_finger_l_nut_*`, `contact_finger_r_nut_*`, `contact_nut_bolt_*`

**Drawer**:
- `physics_gt.drawer_joint_pos` (1), `drawer_joint_vel` (1)
- `physics_gt.handle_position` (3), `handle_velocity` (3)
- `physics_gt.drawer_opening_extent` (1)
- Pair contacts: `contact_finger_l_handle_*`, `contact_finger_r_handle_*`

**Reach** (negative control):
- `physics_gt.target_position` (3), `ee_to_target_distance` (1)

### Episode Metadata (episodes.jsonl)

Each episode records randomized physics parameters:
- Push/Strike: `object_0_mass`, `object_0_static_friction`, `surface_static_friction`
- PegInsert: `peg_static_friction`, `peg_mass`, `hole_static_friction`
- NutThread: `nut_static_friction`, `nut_mass`, `bolt_static_friction`
- Drawer: `drawer_joint_damping`, `handle_static_friction`

## Environment

- **Simulator**: Isaac Lab v2.2.1 on Isaac Sim 4.5.0
- **Robot**: Franka Panda (all tasks)
- **Control**: Differential IK (position-only for Push/Strike, pose for others)
- **Cameras**: 384x384 RGB, table_cam (third-person) + wrist_cam
- **Physics**: PhysX GPU, dt=0.01s, decimation=2 (50Hz control)

## Design Rationale

**Why physics randomization?** If a video encoder encodes physics in its intermediate layers, linear probes trained on episodes with varied friction/mass should predict these quantities from frozen features. Tasks without physics variation (Reach) serve as negative controls.

**Episode count rationale**: Proportional to `(param space dim) × (dynamics complexity) / (informative frame density)`. Strike has 3,000 episodes (most) due to 5D param space and sparse impact events (~25% informative frames).

## Reproducibility

- **Code**: [PhysREPA_Tasks](https://github.com/Leesangoh/PhysREPA_Tasks) (commit `2808943`)
- **RL Checkpoints**: Included in `checkpoints/` directory of this repo
- `peg_insert_rl_games.pth` — PegInsert (rl_games, reward 381)
- `nut_thread_rl_games.pth` — NutThread (rl_games, reward 952)
- `drawer_rsl_rl.pt` — Drawer (RSL-RL, model_4999)
- **Environment**: `environment_info.json` and `pip_freeze.txt` in this repo
- **Platform**: Isaac Sim 4.5.0, Isaac Lab v2.2.1, Python 3.10, CUDA 12.8, 4x NVIDIA A6000

### Reproduce collection

```bash
# Clone code
git clone https://github.com/Leesangoh/PhysREPA_Tasks
cd PhysREPA_Tasks

# Push (scripted, 16 parallel envs)
/isaac-sim/python.sh collect_sample_data.py --task push --num_episodes 1500 --num_envs 16 --step0 --output_dir <OUT> --headless

# PegInsert (RL, 200 parallel envs)
/isaac-sim/python.sh collect_sample_data.py --task peg_insert --num_episodes 2500 --num_envs 200 --filter_success --rl_checkpoint checkpoints/peg_insert_rl_games.pth --output_dir <OUT> --headless

# Drawer (RL, 32 parallel envs)
/isaac-sim/python.sh collect_sample_data.py --task drawer --num_episodes 2000 --num_envs 32 --filter_success --rl_checkpoint checkpoints/drawer_rsl_rl.pt --output_dir <OUT> --headless
```

## License

MIT