ScaleAI/aspi

ASPI（Ambiguous State Prompt Injection）是一个基准测试数据集，用于衡量大型语言模型（LLM）代理在澄清状态下对提示注入攻击的脆弱性。它扩展了AgentDojo（v1.2.2），采用8条件设计，涵盖状态（执行vs澄清）、通道（工具输出vs首次用户vs后续用户）和包装器（原始攻击者文本vs ImportantInstructionsAttack包装），以配对比较状态效应与通道效应和包装效应。当用户提示模糊时，代理会询问澄清问题，用户的响应开启新的注入通道——攻击者可在澄清答案中嵌入恶意指令。数据集包含728行数据，分为四个套件：banking（130行）、slack（97行）、travel（94行）和workspace（407行）。每行数据包括一个来自AgentDojo的（用户任务，注入任务）对，以及实例化所有8个ASPI条件所需的材料：模糊的base_prompt、缺失槽位、预期的澄清问题、良性澄清答案和三个操作符框架的对抗性澄清答案（HR/GS/CP）。数据使用gemini-3.1-pro-preview模型生成并经过人工验证，旨在用于使用工具的LLM代理的安全评估，例如测量代理在进入澄清状态时对提示注入的敏感性，并通过配对引导置信区间和McNemar测试分解状态效应，但不可用于训练，且仅限于英语和合成环境。

ASPI (Ambiguous State Prompt Injection) is a benchmark dataset for measuring the vulnerability of large language model (LLM) agents to prompt injection attacks under clarification states. It extends AgentDojo (v1.2.2) with an 8-condition design covering state (execution vs. clarification), channel (tool output vs. first user vs. subsequent user), and wrapper (raw attacker text vs. ImportantInstructionsAttack wrapper), to enable paired comparisons of state effects against channel effects and wrapper effects. When user prompts are ambiguous, agents will ask clarification questions, and the user's response opens a new injection channel—attackers can embed malicious instructions within clarification responses. The dataset contains 728 rows of data, divided into four suites: banking (130 rows), slack (97 rows), travel (94 rows), and workspace (407 rows). Each row includes a (user task, injection task) pair sourced from AgentDojo, plus materials required to instantiate all 8 ASPI conditions: ambiguous base_prompt, missing slots, expected clarification question, benign clarification answer, and three operator-framed adversarial clarification answers (HR/GS/CP). The data was generated using the gemini-3.1-pro-preview model and manually verified. It is intended for security evaluation of tool-using LLM agents, such as measuring agents' sensitivity to prompt injection when entering clarification states, and decomposing state effects via paired bootstrap confidence intervals and McNemar's test. The dataset cannot be used for training, and is limited to English and synthetic environments only.