FailureSlot

# FailureSlot Dataset: Robotic Manipulation Failure Trajectories for In-Context Retry Learning <p align="center"> <a href="https://jedward225.github.io/RoboRetry/"> <img src="https://img.shields.io/badge/Project-Page-blue"> </a> <a href="https://jedward225.github.io/RoboRetry/static/pdfs/roboretry_paper.pdf"> <img src="https://img.shields.io/badge/Paper-PDF-red"> </a> <a href="https://github.com/jedward225/RoboRetry"> <img src="https://img.shields.io/badge/GitHub-Code-black"> </a> </p> A robotic manipulation failure dataset for studying failure analysis, retry learning, and in-context adaptation in vision-language-action policies. FailureSlot is organized into two subsets: `FailureSlot_FAST` and `FailureSlot_AHA`. `FailureSlot_FAST` contains naturally occurring failure trajectories collected from trained baseline VLA policy rollouts across 12 RLBench manipulation tasks. `FailureSlot_AHA` contains AHA-derived failure trajectories organized under the same task structure and taxonomy for comparison and analysis. Each trajectory stores multi-view RGB observations from three camera views, together with low-dimensional robot observations. We also provide `reviews_fast.json`, which contains human annotations for the failure trajectories, including failure keyframes, failure descriptions, and corrective feedback for retry-oriented analysis. ## 🚀 Highlights * **1,334** naturally occurring VLA failure trajectories in `FailureSlot_FAST`. * Covers **12 RLBench manipulation tasks**. * `FailureSlot_FAST` is collected from trained baseline VLA policy rollouts rather than manually scripted perturbations. * Organized into two subsets: **`FailureSlot_FAST`** and **`FailureSlot_AHA`**. * Each trajectory contains **three-view RGB frame sequences** and **low-dimensional robot observations**. * Includes **`reviews_fast.json`** with human failure annotations and corrective feedback. * Designed for studying **failure-conditioned retry**, **robot failure analysis**, and **in-context robot learning**. * Supports analysis with an **8-category failure mechanism taxonomy**. ## 📊 Dataset Statistics ### Comparison with Existing Robot Failure Datasets **Synth.** = programmatically perturbed; **Nat.** = naturally occurring during policy execution; **ICL** = structured as an in-context failure slot; **# Failures** denotes the number of failure trajectories or failure records. | Dataset | Src | Env | Ann. | # Failures | ICL | | --------------- | -------- | ------- | --------------- | ---------: | :-: | | FailSafe | Synth. | Sim | Labels | 131K | ✗ | | AHA | Synth. | Sim | QA | 49K | ✗ | | Guardian | Synth. | Both | CoT | 37K | ✗ | | RoboFAC | Synth. | Both | QA | 8.9K | ✗ | | RoboFAC | Nat. | Real | QA | 480 | ✗ | | REFLECT | Nat. | Both | LLM | 130 | ✗ | | **FailureSlot** | **Nat.** | **Sim** | **Human + VLM** | **1,334** | ✓ | ### Failure Taxonomy FailureSlot can be analyzed using a shared 8-category failure mechanism taxonomy. The table below reports the category distributions of `FailureSlot_FAST` and `FailureSlot_AHA`. | Type | Description | `FailureSlot_FAST` | `FailureSlot_AHA` | | -------------------------- | ------------------------------------------------------------------------------------------------------ | -----------------: | ----------------: | | Misalignment | Spatial alignment errors between the robot end-effector and the intended target or interaction region. | 48.1% | 51.7% | | Premature Closure | The gripper closes before the robot reaches a proper contact or grasping position. | 16.0% | 5.8% | | Contact Instability | The robot establishes contact but fails to maintain stable interaction or grasp control. | 12.4% | 7.9% | | Trajectory Deviation | The executed motion drifts away from the intended path or movement direction. | 11.9% | 0.0% | | Incorrect Target Selection | The robot grounds the instruction to the wrong target object, button, drawer, or subgoal. | 6.8% | 0.0% | | No Action / Frozen | The robot does not take a meaningful action or becomes stalled during execution. | 2.7% | 3.7% | | Rotation Error | The robot applies an incorrect orientation or rotational adjustment. | 1.1% | 22.9% | | Incorrect Sequence | The robot performs the required sub-actions in the wrong order. | 1.0% | 7.9% | <p align="center"> <img src="assets/figure2.png" alt="Figure 2: Failure type distributions across FailureSlot_FAST and FailureSlot_AHA." width="70%"> </p> ### 📈 Behavioral Visualization <p align="center"> <img src="assets/figure4.png" alt="Figure 4: Failure-distribution and behavior-space comparison." width="90%"> </p> <p align="center"> <em> Figure 4: Failure-distribution and behavior-space comparison. Left: category distributions of naturally occurring failures in <code>FailureSlot_FAST</code> and AHA-derived failures in <code>FailureSlot_AHA</code>. Right: behavioral t-SNE visualizations on representative RLBench tasks, showing that natural VLA failures and AHA-derived failures occupy different regions in behavior space. </em> </p> ## 📁 Data Format To avoid excessive file counts during dataset hosting and downloading, FailureSlot is released as task-level `.tar` archives. Each archive corresponds to one RLBench task. The dataset is organized as follows: ```text FailureSlot/ ├── README.md ├── reviews_fast.json ├── assets/ │ ├── figure2.png │ └── figure4.png │ ├── FailureSlot_FAST/ │ ├── close_box.tar │ ├── close_microwave.tar │ ├── lamp_on.tar │ ├── meat_on_grill.tar │ ├── open_drawer.tar │ ├── open_wine_bottle.tar │ ├── pick_and_lift.tar │ ├── pick_up_cup.tar │ ├── push_button.tar │ ├── push_buttons.tar │ ├── take_lid_off_saucepan.tar │ ├── turn_tap.tar │ └── ... │ └── FailureSlot_AHA/ ├── close_box.tar ├── close_microwave.tar ├── lamp_on.tar ├── meat_on_grill.tar ├── open_drawer.tar ├── open_wine_bottle.tar ├── pick_and_lift.tar ├── pick_up_cup.tar ├── push_button.tar ├── push_buttons.tar ├── take_lid_off_saucepan.tar ├── turn_tap.tar └── ... ``` ### Subsets | Subset | Description | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | `FailureSlot_FAST` | Naturally occurring failure trajectories collected from trained baseline VLA policy rollouts across RLBench manipulation tasks. | | `FailureSlot_AHA` | AHA-derived failure trajectories organized under the same task structure for comparison and analysis. | ### Extraction Each task archive can be extracted with: ```bash tar -xf FailureSlot_FAST/close_box.tar -C FailureSlot_FAST/ ``` After extraction, the task folder contains trajectory folders: ```text FailureSlot_FAST/ └── close_box/ ├── traj001/ │ ├── front_rgb/ │ ├── overhead_rgb/ │ ├── wrist_rgb/ │ └── low_dim_obs.pkl ├── traj002/ └── ... ``` The same structure is used for `FailureSlot_AHA`. ### Trajectory Format Each trajectory folder, such as `traj001`, contains synchronized observations from three camera views and a low-dimensional observation file: | File / Folder | Description | | ----------------- | --------------------------------------------------------------------------------------- | | `front_rgb/` | RGB frame sequence from the front camera view. | | `overhead_rgb/` | RGB frame sequence from the overhead camera view. | | `wrist_rgb/` | RGB frame sequence from the wrist-mounted camera view. | | `low_dim_obs.pkl` | Low-dimensional robot observations, such as robot state and proprioceptive information. | ### Annotation File | File | Description | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `reviews_fast.json` | Human annotations for `FailureSlot_FAST` failure trajectories, including key frames, failure descriptions, and corrective feedback for retry-oriented analysis. | Each entry in `reviews_fast.json` is keyed by trajectory path and contains the following fields: ```json { "FailureSlot_FAST/close_box/traj001": { "adjusted_frame": 42, "oracle_description": "...", "oracle_advice": "...", "task": "close_box", "trajectory": "traj001" } } ``` ## 📚 Citation If you find FailureSlot useful for your research, please consider citing our paper: ```bibtex @misc{failureslot2026, title = {What Do VLAs Actually Learn through In-Context Failure Conditioning?}, author = {Anonymous}, year = {2026}, note = {Anonymous submission}, } ```