LIBERO-plus

<h1 align="center">
LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
</h1>

<p align="center">
📄 <a href="https://arxiv.org/pdf/2510.13626v1"><strong>Paper</strong></a> |
🏗️ <a href="https://github.com/sylvestf/LIBERO-plus"><strong>Repo</strong></a> |
🌐 <a href="https://sylvestf.github.io/LIBERO-plus"><strong>Website</strong></a> |
🤗 <a href="https://huggingface.co/datasets/Sylvest/LIBERO-plus/tree/main"><strong>Assets</strong></a> |
🤗 <a href="https://huggingface.co/Sylvest/openvla-7b-oft-finetuned-libero-plus-mixdata"><strong>Model</strong></a> |
📁 <a href="https://huggingface.co/datasets/Sylvest/libero_plus_rlds"><strong>Training Dataset</strong></a>
</p>


## 🔥 Overview
This repository contains the official implementation and benchmark for our paper "In-depth Robustness Analysis for Vision-Language-Action Models". We systematically expose the hidden vulnerabilities of contemporary VLA models through comprehensive robustness evaluation across seven perturbation dimensions. You can simply replace the original `libero` with a `pip install -e .` without modifying your code.

## 🚀 Key Findings
- **Significant Fragility**: VLA models exhibit extreme sensitivity to camera viewpoints and robot initial states, with performance dropping from 95% to below 30% under modest perturbations
- **Language Ignorance**: Models largely ignore language instructions, functioning more like Vision-Action models
- **Negative Compositional Generalization**: Combined perturbations reveal complex interaction effects beyond independent factors

## 📊 LIBERO-plus Benchmark

### 7 Perturbation Dimensions
We introduce **LIBERO-plus**, a comprehensive benchmark with 10,030 tasks spanning:

1. **Objects Layout** - Confounding objects and target object displacement
2. **Camera Viewpoints** - Position, orientation, and field-of-view changes
3. **Robot Initial States** - Manipulator initial pose variations
4. **Language Instructions** - LLM-based instruction rewriting
5. **Light Conditions** - Intensity, direction, color, and shadow variations
6. **Background Textures** - Scene and surface appearance changes
7. **Sensor Noise** - Photometric distortions and image degradation

### Evaluated Models
- OpenVLA and variants (OFT, OFT_w, OFT_m)
- π₀ and π₀-fast
- Nora, WorldVLA, UniVLA, RIPT-VLA

## 🛠️ Installation
The usage of this project is identical to [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO). Simply replace the originally installed LIBERO repository with our repository without modifying your code.

```bash
# Clone our repository
git clone https://github.com/sylvestf/LIBERO-plus.git
cd LIBERO-plus
```

If you have LIBERO installed, please uninstall or remove it first. Please verify if the repo path in the following configuration file needs to be updated to path_to_liberoplus_repo.
Here are the default paths for the configuration files: `/root/.libero/config.yaml`. You can check your `libero_config_path` at `path_to_your_LIBERO_repo/libero/libero/__init__.py`.

Then install our new LIBERO repository
```bash
# Install the new LIBERO package
pip install -e .

# New dependencies installed on top of LIBERO
apt install libexpat1
apt install libfontconfig1-dev
apt install libpython3-stdlib
apt-get install libmagickwand-dev
pip install -r extra_requirements.txt
```

Please download our assets from [LIBERO-plus](https://huggingface.co/datasets/Sylvest/LIBERO-plus/tree/main), including hundreds of new objects, textures, and other required assets. Please unzip the `assets.zip` file to `/LIBERO-plus/libero/libero` path. You can also find the [training dataset](https://huggingface.co/datasets/Sylvest/libero_plus_rlds/tree/main) mentioned in our paper and the [OpenVLA-OFT weights after mix-SFT](https://huggingface.co/Sylvest/openvla-7b-oft-finetuned-libero-plus-mixdata/tree/main) on this dataset.

The extracted directory structure should look like:

```text
LIBERO-plus/
└── libero/
└── libero/
└── assets/
├── articulated_objects/
├── new_objects/
├── scenes/
├── stable_hope_objects/
├── stable_scanned_objects/
├── textures/
├── turbosquid_objects/
├── serving_region.xml
├── wall_frames.stl
└── wall.xml
```

## 🔧 Evaluation
The evaluation method is almost identical to `LIBERO`. The only required modification is adjusting `num_trials_per_task` from 50 to 1 in your configuration.

## 📊 LIBERO-Plus Benchmark Leaderboard
| Model | Camera | Robot | Language | Light | Background | Noise | Layout | Total |
|-------|--------|-------|----------|-------|------------|-------|--------|-------|
| [OpenVLA](https://github.com/openvla/openvla) | 0.8 | 3.5 | 23.0 | 8.1 | 50.4 | 15.2 | 28.5 | 17.3 |
| [OpenVLA-OFT](https://github.com/moojink/openvla-oft) | 56.4 | 31.9 | 79.5 | 88.7 | 97.3 | 75.8 | 74.2 | 70.0 |
| [OpenVLA-OFT_w](https://github.com/moojink/openvla-oft) | 10.4 | 38.7 | 70.5 | 76.8 | 99.2 | 49.9 | 69.9 | 56.4 |
| [NORA](https://github.com/declare-lab/nora) | 2.2 | 37.0 | 65.1 | 45.7 | 65.5 | 12.8 | 62.1 | 39.8 |
| [WorldVLA](https://github.com/alibaba-damo-academy/WorldVLA) | 0.1 | 27.9 | 41.6 | 43.7 | 19.8 | 10.9 | 38.0 | 25.3 |
| [UniVLA](https://github.com/OpenDriveLab/UniVLA) | 1.8 | 46.2 | 69.6 | 69.0 | 90.7 | 21.2 | 31.9 | 43.9 |
| [π₀](https://github.com/Physical-Intelligence/openpi) | 13.8 | 6.0 | 58.8 | 85.0 | 90.7 | 79.0 | 68.9 | 54.6 |
| [π₀-Fast](https://github.com/Physical-Intelligence/openpi) | 65.1 | 21.6 | 61.0 | 73.2 | 97.7 | 74.4 | 68.8 | 64.2 |
| [RIPT-VLA](https://github.com/Ariostgx/ript-vla) | 55.2 | 31.2 | 77.6 | 88.4 | **100.0** | 73.5 | 74.2 | 69.3 |
| [OpenVLA-OFT_m](https://github.com/moojink/openvla-oft) | 55.6 | 21.7 | 81.0 | 92.7 | 92.3 | 78.6 | 68.7 | 68.1 |
| **[OpenVLA-OFT+ (Ours)](https://github.com/moojink/openvla-oft)** | **92.8** | **30.3** | **85.8** | **94.9** | 93.9 | **89.3** | **77.6** | **79.6** |

- **OpenVLA-OFT+** shows the performance of [OpenVLA-OFT with a mix-sft on LIBERO-plus dataset](https://huggingface.co/Sylvest/openvla-7b-oft-finetuned-libero-plus-mixdata/tree/main).
- **OpenVLA-OFT_w** shows the performance of [OpenVLA-OFT without wrist observation input](https://huggingface.co/Sylvest/openvla-7b-oft-finetuned-libero-without-wrist).
- **OpenVLA-OFT_m** shows the performance of [OpenVLA-OFT with a mix-sft](https://huggingface.co/moojink/openvla-7b-oft-finetuned-libero-spatial).

### Origin LIBERO Benchmark Leaderboard

To make it easier to get all the results in one place, we've compiled the evaluation results of current VLA models on the original LIBERO benchmark in this [table](./libero_res.md).

## Citation
If you find this work useful for your research, please cite our paper:
```bibtex
@article{fei25libero-plus,
title={LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models},
author={Senyu Fei and Siyin Wang and Junhao Shi and Zihao Dai and Jikun Cai and Pengfang Qian and Li Ji and Xinzhe He and Shiduo Zhang and Zhaoye Fei and Jinlan Fu and Jingjing Gong and Xipeng Qiu},
journal = {arXiv preprint arXiv:2510.13626},
year={2025},
}
```

<h1 align="center">LIBERO-Plus：视觉-语言-动作模型的深度鲁棒性分析</h1>

<p align="center">
📄 <a href="https://arxiv.org/pdf/2510.13626v1"><strong>论文</strong></a> |
🏗️ <a href="https://github.com/sylvestf/LIBERO-plus"><strong>代码仓库</strong></a> |
🌐 <a href="https://sylvestf.github.io/LIBERO-plus"><strong>项目主页</strong></a> |
🤗 <a href="https://huggingface.co/datasets/Sylvest/LIBERO-plus/tree/main"><strong>数据集资源</strong></a> |
🤗 <a href="https://huggingface.co/Sylvest/openvla-7b-oft-finetuned-libero-plus-mixdata"><strong>模型权重</strong></a> |
📁 <a href="https://huggingface.co/datasets/Sylvest/libero_plus_rlds"><strong>训练数据集</strong></a>
</p>


## 🔥 项目概述
本仓库包含我们论文《视觉-语言-动作模型的深度鲁棒性分析》的官方实现与基准测试套件。我们通过覆盖7种扰动维度的全面鲁棒性评估，系统性地揭示了当前视觉-语言-动作（Vision-Language-Action, VLA）模型所隐藏的安全漏洞。用户仅需执行`pip install -e .`即可替换原有的`libero`库，无需修改现有代码。

## 🚀 核心发现
- **显著脆弱性**：VLA模型对相机视角与机器人初始状态具有极强敏感性，在轻度扰动下性能可从95%骤降至30%以下
- **语言失效问题**：模型大多忽略语言指令，其表现更接近视觉-动作模型而非完整的视觉-语言-动作模型
- **负向组合泛化问题**：联合扰动实验揭示了独立因素之外的复杂交互效应

## 📊 LIBERO-Plus 基准测试套件

### 7种扰动维度
我们提出了**LIBERO-Plus**，这一综合性基准测试套件包含10030个任务，覆盖以下7类扰动：

1. **物体布局扰动**：干扰物体与目标物体的位置偏移
2. **相机视角扰动**：相机位置、朝向与视场角变化
3. **机器人初始状态扰动**：机械臂初始位姿变化
4. **语言指令扰动**：基于大语言模型（Large Language Model, LLM）的指令重写
5. **光照条件扰动**：光照强度、方向、色彩与阴影变化
6. **背景纹理扰动**：场景与物体表面外观变化
7. **传感器噪声扰动**：光度畸变与图像降质

### 测试模型
- OpenVLA 及其变体（OFT、OFT_w、OFT_m）
- π₀ 与 π₀-Fast
- Nora、WorldVLA、UniVLA、RIPT-VLA

## 🛠️ 安装指南
本项目的使用方式与[LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO)完全一致，仅需将原安装的LIBERO仓库替换为本仓库即可，无需修改现有代码。

bash
# 克隆本仓库
git clone https://github.com/sylvestf/LIBERO-plus.git
cd LIBERO-plus

若您已安装LIBERO，请先卸载或移除原有版本。请检查配置文件中的仓库路径是否需要更新为`path_to_liberoplus_repo`。配置文件的默认路径为：`/root/.libero/config.yaml`。您可通过`path_to_your_LIBERO_repo/libero/libero/__init__.py`查看您的`libero_config_path`。

随后安装本仓库的新版LIBERO包：
bash
# 安装新版LIBERO包
pip install -e .

# 安装LIBERO之上的新增依赖
apt install libexpat1
apt install libfontconfig1-dev
apt install libpython3-stdlib
apt-get install libmagickwand-dev
pip install -r extra_requirements.txt

请从[LIBERO-Plus](https://huggingface.co/datasets/Sylvest/LIBERO-plus/tree/main)下载我们的资源包，其中包含数百个新增物体、纹理与其他必要资源。请将`assets.zip`解压至`/LIBERO-plus/libero/libero`路径下。您还可在该平台找到本文提及的[训练数据集](https://huggingface.co/datasets/Sylvest/libero_plus_rlds/tree/main)，以及在该数据集上经过混合监督微调（Mix-SFT）后的[OpenVLA-OFT模型权重](https://huggingface.co/Sylvest/openvla-7b-oft-finetuned-libero-plus-mixdata/tree/main)。

解压后的目录结构应如下所示：
text
LIBERO-plus/
└── libero/
└── libero/
└── assets/
├── articulated_objects/
├── new_objects/
├── scenes/
├── stable_hope_objects/
├── stable_scanned_objects/
├── textures/
├── turbosquid_objects/
├── serving_region.xml
├── wall_frames.stl
└── wall.xml

## 🔧 评估方法
评估方法与`LIBERO`基本一致，仅需在配置文件中将`num_trials_per_task`从50调整为1即可。

## 📊 LIBERO-Plus 基准测试排行榜
| 模型 | 相机视角 | 机器人初始状态 | 语言指令 | 光照条件 | 背景纹理 | 传感器噪声 | 物体布局 | 综合得分 |
|-------|--------|-------|----------|-------|------------|-------|--------|-------|
| [OpenVLA](https://github.com/openvla/openvla) | 0.8 | 3.5 | 23.0 | 8.1 | 50.4 | 15.2 | 28.5 | 17.3 |
| [OpenVLA-OFT](https://github.com/moojink/openvla-oft) | 56.4 | 31.9 | 79.5 | 88.7 | 97.3 | 75.8 | 74.2 | 70.0 |
| [OpenVLA-OFT_w](https://github.com/moojink/openvla-oft) | 10.4 | 38.7 | 70.5 | 76.8 | 99.2 | 49.9 | 69.9 | 56.4 |
| [NORA](https://github.com/declare-lab/nora) | 2.2 | 37.0 | 65.1 | 45.7 | 65.5 | 12.8 | 62.1 | 39.8 |
| [WorldVLA](https://github.com/alibaba-damo-academy/WorldVLA) | 0.1 | 27.9 | 41.6 | 43.7 | 19.8 | 10.9 | 38.0 | 25.3 |
| [UniVLA](https://github.com/OpenDriveLab/UniVLA) | 1.8 | 46.2 | 69.6 | 69.0 | 90.7 | 21.2 | 31.9 | 43.9 |
| [π₀](https://github.com/Physical-Intelligence/openpi) | 13.8 | 6.0 | 58.8 | 85.0 | 90.7 | 79.0 | 68.9 | 54.6 |
| [π₀-Fast](https://github.com/Physical-Intelligence/openpi) | 65.1 | 21.6 | 61.0 | 73.2 | 97.7 | 74.4 | 68.8 | 64.2 |
| [RIPT-VLA](https://github.com/Ariostgx/ript-vla) | 55.2 | 31.2 | 77.6 | 88.4 | **100.0** | 73.5 | 74.2 | 69.3 |
| [OpenVLA-OFT_m](https://github.com/moojink/openvla-oft) | 55.6 | 21.7 | 81.0 | 92.7 | 92.3 | 78.6 | 68.7 | 68.1 |
| **[OpenVLA-OFT+ (本文提出)](https://github.com/moojink/openvla-oft)** | **92.8** | **30.3** | **85.8** | **94.9** | 93.9 | **89.3** | **77.6** | **79.6** |

- **OpenVLA-OFT+** 为在LIBERO-Plus数据集上经过混合监督微调的OpenVLA-OFT模型性能结果。
- **OpenVLA-OFT_w** 为未使用腕部视觉输入的OpenVLA-OFT模型性能结果。
- **OpenVLA-OFT_m** 为经过混合监督微调的OpenVLA-OFT模型性能结果。

### 原始LIBERO基准测试排行榜
为方便用户一站式获取全部结果，我们将当前VLA模型在原始LIBERO基准测试上的评估结果汇总至该[表格](./libero_res.md)中。

## 引用声明
若本工作对您的研究有所帮助，请引用我们的论文：
bibtex
@article{fei25libero-plus,
title={LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models},
author={Senyu Fei and Siyin Wang and Junhao Shi and Zihao Dai and Jikun Cai and Pengfang Qian and Li Ji and Xinzhe He and Shiduo Zhang and Zhaoye Fei and Jinlan Fu and Jingjing Gong and Xipeng Qiu},
journal = {arXiv preprint arXiv:2510.13626},
year={2025},
}