sureheremarv/ipi_arena_attacks

---
license: mit
language:
- en
tags:
- agent
- security
- prompt-injection
size_categories:
- n<1K
---

# IPI Arena Attacks

Attack strings from the [IPI Arena](https://github.com/GraySwanAI/ipi_arena_os) benchmark for evaluating model robustness to indirect prompt injection (IPI), from [How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition](https://arxiv.org/abs/2603.15714).

## Dataset

95 attack strings across 28 behaviors, sourced from Qwen (`qwen/qwen3-vl-235b-a22b-instruct`). These attacks succeeded on open-source models but did **not** transfer to any closed-source model in the arena.

Each row contains:
- `behavior_id`: the benchmark behavior the attack targets
- `attack`: the injection string

## Usage

```python
from datasets import load_dataset
ds = load_dataset("sureheremarv/ipi_arena_attacks")
```

Or with [ipi-arena-bench](https://github.com/GraySwanAI/ipi_arena_os):

```yaml
behaviors:
hf_dataset: sureheremarv/ipi_arena_attacks
behavior_ids:
- garage-door-email
- book-hotel
```

## Citation

```bibtex
@misc{dziemian2026vulnerableaiagentsindirect,
title={How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition},
author={Mateusz Dziemian and Maxwell Lin and Xiaohan Fu and Micha Nowak and Nick Winter and Eliot Jones and Andy Zou and Lama Ahmad and Kamalika Chaudhuri and Sahana Chennabasappa and Xander Davies and Lauren Deason and Benjamin L. Edelman and Tanner Emek and Ivan Evtimov and Jim Gust and Maia Hamin and Kat He and Klaudia Krawiecka and Riccardo Patana and Neil Perry and Troy Peterson and Xiangyu Qi and Javier Rando and Zifan Wang and Zihan Wang and Spencer Whitman and Eric Winsor and Arman Zharmagambetov and Matt Fredrikson and Zico Kolter},
year={2026},
eprint={2603.15714},
archivePrefix={arXiv},
primaryClass={cs.CR},
url={https://arxiv.org/abs/2603.15714},
}
```

## Links

- Benchmark code: [GraySwanAI/ipi_arena_os](https://github.com/GraySwanAI/ipi_arena_os)