VincentNi/robotwin-blocks-ranking-rgb-rollouts

---
license: apache-2.0
language:
- en
tags:
- robotics
- video-generation
- robotwin
- wan22
- ti2v
- reward-model
size_categories:
- n<1K
---

# RoboTwin blocks_ranking_rgb — Wan2.2 TI2V Rollouts

160 text+image-to-video rollouts (10 initial conditions × 16 random seeds) for the
**blocks_ranking_rgb** task from RoboTwin, generated with the **Wan2.2 TI2V (5B)**
diffusion model fine-tuned with a merged Vidar LoRA adapter, and scored with the
`blocks_ranking_v2` reward (SAM3 object tracking + IDM inverse-dynamics + FK
gripper ↔ block position matching).

Companion to the [EmbodiedVideoRL / DanceGRPO](https://arxiv.org/abs/2505.07818)
reward-model work.

## Dataset summary

| | value |
|--------|-------|
| Task | `blocks_ranking_rgb` (RoboTwin) — arrange R/G/B blocks left-to-right |
| Scenes | 10 initial conditions |
| Rollouts / scene | 16 (seeds 42–57) |
| Total videos | 160 |
| Frame count | 121 @ 16 fps |
| Resolution | 640 × 736 |
| Generator | Wan2.2-TI2V-5B + merged Vidar LoRA |
| Sampler | 20-step SDE (eta=1.0, CFG=5.0, shift=5.0) |
| Reward | `blocks_ranking_v2` — FK pick/place + left-to-right order + no-duplication |

**Overall pass rate: 108 / 160 = 67.5 %**

Per-scene:

| scene id | pass / total |
|----------|--------------|
| 123500000 | 16 / 16 |
| 123500001 | 4 / 16 |
| 123500002 | 9 / 16 |
| 123500003 | 12 / 16 |
| 123500004 | 8 / 16 |
| 123500005 | 15 / 16 |
| 123500006 | 3 / 16 |
| 123500007 | 12 / 16 |
| 123500008 | 16 / 16 |
| 123500009 | 13 / 16 |

## Files

For each rollout with stem `{scene}_g{GGG}_s{SEED}`:

| file | content |
|------|---------|
| `{stem}.mp4` | raw generated video (121 frames, 640×736, libx264) |
| `{stem}_{CLEAN,FAIL}_ms{motion_score}.mp4` | annotated reward-debug video: SAM3 boxes per color, per-frame object count, FK gripper trails (left=cyan, right=magenta), pick/place match overlay, judge verdict |
| `{stem}_{CLEAN,FAIL}.txt` | one-line reward summary with failure reasons |
| `{stem}_extraction.json` | SAM3 tracking evidence — per-frame `obj_ids`, `probs`, `boxes_xywh` (xywh normalized 0–1), `num_tracked` for each of {red, green, blue} block prompts; plus `motion_score`, `max_jump`, `grip_changes`, `total_frames`, `fps`. Per-frame pixel masks have been stripped to keep the dataset compact — they are deterministically reproducible from SAM3 if needed. |
| `{stem}_idm_actions.npy` | IDM-inferred 14-DOF action trajectory `(T, 14)` — layout `[left_arm(6), left_gripper(1), right_arm(6), right_gripper(1)]` |
| `summary.json` | merged per-video summary (reward, criteria pass/fail, motion_score, max_jump, grip_changes, timings) |

## Reward criteria (`blocks_ranking_v2`)

A rollout is labelled `CLEAN` (reward=1.0) iff all of:

1. **grip_changes == 6** — exactly 3 close + 3 open cycles detected by IDM (one per block)
2. **close-match OK** — every gripper-close FK position matches an *initial* block position (pick, thr=0.05)
3. **open-match OK** — every gripper-open FK position matches a *final* block position (place, thr=0.08)
4. **order OK** — final block order R < G < B along cx (left-to-right, margin=0.05)
5. **no duplication** — block count per color never exceeds 1 throughout the trajectory

Otherwise `FAIL` (reward=0.0) and `_FAIL.txt` lists the triggered reasons.

## Loading

```python
import json, numpy as np
from huggingface_hub import hf_hub_download

summary = json.load(open(hf_hub_download(
"VincentNi/robotwin-blocks-ranking-rgb-rollouts",
"summary.json", repo_type="dataset",
)))
row = summary[0]
stem = row["video"][:-4] # drop .mp4
ext = json.load(open(hf_hub_download(
"VincentNi/robotwin-blocks-ranking-rgb-rollouts",
f"{stem}_extraction.json", repo_type="dataset",
)))
idm = np.load(hf_hub_download(
"VincentNi/robotwin-blocks-ranking-rgb-rollouts",
f"{stem}_idm_actions.npy", repo_type="dataset",
))
```

## Generation command

```bash
bash scripts/inference/rollout_score_blocks_ranking_rgb.sh
# equivalently: 8 × H100, torchrun --nproc_per_node=8,
# --num_rollouts 16 --seed 42 --eta 1.0 --sample_steps 20 --sample_guide_scale 5.0
# --reward_config blocks_ranking_v2
```