hww123/MADBench
收藏资源简介：
--- license: mit task_categories: - other language: - en tags: - multi-agent - anomaly-detection - agent-collaboration - reasoning - benchmark - llm-evaluation pretty_name: MADBench — Multi-Agent System Anomaly Detection Benchmark size_categories: - 10K<n<100K configs: - config_name: default data_files: - split: train path: metadata.jsonl --- # MADBench: Multi-Agent System Anomaly Detection Benchmark MADBench is a dataset of execution traces from a collaborative multi-agent system (MAS) solving procedurally generated escape room puzzles. It is designed to support research on **anomaly detection in multi-agent systems** — identifying when and where agent collaboration breaks down, how errors propagate across agents, and what distinguishes healthy from faulty execution. ## Motivation Multi-agent systems introduce failure modes that go beyond single-agent errors: one agent's wrong output silently corrupts the inputs to downstream agents, making the root cause of a final failure hard to trace. MADBench provides a controlled environment where: - The ground truth at **every step** is known - Agents must **collaborate sequentially** — errors propagate downstream - **Multiple trap types** create realistic distractors that can mislead agents - **Six models** and **three temperature settings** produce a diverse failure distribution across difficulty levels ## Task Environment The task is a multi-step escape room puzzle. Each room contains instruments (e.g. thermometer, barometer), clues containing math word problems, and deliberate distractors: | Trap type | Description | |---|---| | `fake_item` | An instrument with a physically impossible reading | | `fake_clue` | A clue referencing a nonsensical unit or instrument | | `trap_item_desc` | A real instrument with a misleading description | To escape, the agent team must identify the correct clue, solve the embedded math problem, locate the correct instrument, and apply the computed delta (with unit conversion) to produce the right final reading. ## Multi-Agent Pipeline The MAS uses a fixed sequential architecture with three specialized agents: ``` [Observer] → OBSERVE_CLUE → identifies real clue among fakes [Clue Solver] → SOLVE_CLUE → solves the math word problem [Observer] → OBSERVE_ITEM → locates the correct instrument [Item Manager] → APPLY_DELTA → computes new instrument reading (with unit conversion) [Observer] → OBSERVE_PUZZLE → verifies whether the puzzle is solved ``` Each agent receives only the output of the previous step — a reasoning error by any agent propagates forward without correction. ### Agents | Role | Input | Output | |---|---|---| | `observer` | Room state | Identified clue / item / solved status | | `clue_solver` | Math word problem + unit hint | Numerical answer + unit | | `item_manager` | Instrument state, delta, units | New instrument reading | ## Dataset Structure ``` traces/ ├── rooms.jsonl # 200 room definitions (easy / medium / hard) ├── nightmare_rooms.jsonl # 50 room definitions (nightmare, 4 puzzles each) ├── easy_nightmares/ # Preliminary nightmare runs (gpt-4.1, claude-sonnet-4) ├── run_{date}_{model}_{temp}/ # 18 regular runs — 200 traces each └── run_{date}_{model}_{temp}_nightmare/ # 16 nightmare runs — 50 traces each ``` Each `run_*` folder contains one JSON file per room: ``` {timestamp}_EscapeRoom_{room_id}_{escaped|failed}.json ``` ### Models Evaluated | Model | Temperatures | |---|---| | `gpt-4.1-mini` | 0.0, 0.3, 0.6 | | `gpt-4.1` | 0.0, 0.3, 0.6 | | `claude-sonnet-4-20250514` | 0.0, 0.3, 0.6 | | `Qwen2.5-14B-Instruct` | 0.0, 0.3, 0.6 | | `deepseek-reasoner` | 0.0, 0.3, 0.6 | | `gpt-5.4` | 0.0, 0.3, 0.6 | ## Trace JSON Schema ```jsonc { "config": { "system": { "architecture": "sequential" }, "llm": { "provider": "...", "model": "...", "temperature": 0.0, "max_tokens": 4096 }, "agents": [ /* per-role system prompts */ ] }, "room": { "room_id": "room_0000", "difficulty": "easy", // easy | medium | hard | nightmare "n_puzzles": 1, // nightmare rooms have 4 sequential puzzles "scenery": [ /* distractor items */ ], "puzzles": [ { "puzzle_id": "P1", "clue": { "hint": "...", // natural-language hint for instrument + unit "item_type": "thermometer", "problem": "...", // math word problem "answer": 32.0, // correct solution "delta_unit": "kelvin" }, "item": { "type": "thermometer", "state": 88, "unit": "celsius" }, "fake_items": [ /* impossible readings */ ], "fake_clues": [ /* nonsensical hints */ ], "traps": ["fake_item", "fake_clue"], "ground_truth": 120 // expected final instrument reading } ] }, "trace": [ { "action": "OBSERVE_CLUE | SOLVE_CLUE | OBSERVE_ITEM | APPLY_DELTA | OBSERVE_PUZZLE", "agent": "observer | clue_solver | item_manager", "message": "...", // free-text agent reasoning "structured": { ... }, // structured JSON output "schema_errors": {}, // anomaly signal: output format violations "confidence": 0.95, "call_statistic": { "duration": 2.57, "input_tokens": 613, "output_tokens": 169, "timed_out": false, // anomaly signal: timeout "superlong_reasoning": false // anomaly signal: runaway chain-of-thought }, "attempt": 1, "verification": { "status": "correct | wrong", // ground-truth label for this step ... } } ], "failure_report": { "schema_failures": [], // steps where structured output was malformed "eval_results": ["correct"] // per-puzzle outcome }, "escaped": true, // overall outcome "puzzles_solved": 1, "puzzles_total": 1, "timing_sec": 24.1 } ``` ### Anomaly Signals in Each Trace | Field | Type | Description | |---|---|---| | `verification.status` | step-level label | `"wrong"` marks a faulty agent output with known ground truth | | `schema_errors` | structural anomaly | Agent output did not conform to the required JSON schema | | `call_statistic.timed_out` | behavioral anomaly | Agent call exceeded the time limit | | `call_statistic.superlong_reasoning` | behavioral anomaly | Runaway chain-of-thought | | `failure_report.schema_failures` | run-level summary | Aggregated schema violations for the room | | `escaped` | run-level label | Whether the full pipeline succeeded | ## Escape Rate Results Escape rate = fraction of rooms where the MAS produced the correct final instrument reading for all puzzles. ### Regular Rooms (easy / medium / hard, N=200) | Model | Temp 0.0 | Temp 0.3 | Temp 0.6 | |---|---|---|---| | deepseek-reasoner | **27%** | **30%** | **28%** | | gpt-5.4 | 23% | 22% | 25% | | claude-sonnet-4-20250514 | 22% | 23% | 22% | | gpt-4.1-mini | 18% | 20% | 18% | | gpt-4.1 | 16% | 16% | 18% | | Qwen2.5-14B-Instruct | 4% | 5% | 3% | ### Nightmare Rooms (4 sequential puzzles, N=50) | Model | Temp 0.0 | Temp 0.3 | Temp 0.6 | |---|---|---|---| | deepseek-reasoner | **10%** | **6%** | **12%** | | gpt-5.4 | 2% | 2% | 2% | | gpt-4.1 | 0% | 0% | 0% | | claude-sonnet-4-20250514 | 0% | 0% | 0% | | Qwen2.5-14B-Instruct | 0% | 0% | 0% | Nightmare rooms chain 4 puzzles sequentially with no partial credit — anomalies compound across puzzles, making the nightmare split particularly suitable for studying error propagation. ## How to Load ```python import json, glob, os, re def load_run(run_dir): traces = [] for path in sorted(glob.glob(os.path.join(run_dir, "*.json"))): with open(path) as f: traces.append(json.load(f)) return traces # Iterate all steps and collect anomaly labels def iter_steps(trace): for step in trace["trace"]: yield { "room_id": trace["room"]["room_id"], "difficulty": trace["room"]["difficulty"], "action": step["action"], "agent": step["agent"], "status": step["verification"].get("status"), # "correct" | "wrong" "schema_errors": bool(step.get("schema_errors")), "timed_out": step["call_statistic"]["timed_out"], "superlong": step["call_statistic"]["superlong_reasoning"], } # Example: load one run and collect all anomalous steps run = load_run("run_20260316_174255_claude-sonnet-4-20250514_0.0") anomalies = [s for trace in run for s in iter_steps(trace) if s["status"] == "wrong"] ```
许可证：MIT 任务类别： - 其他 语言： - 英语 标签： - 多智能体 - 异常检测 - 智能体协作 - 推理 - 基准测试 - 大语言模型评估 展示名称：MADBench——多智能体系统异常检测基准 样本量范围： - 10K < n < 100K 配置项： - 配置名称：default 数据文件： - 拆分：训练集 路径：metadata.jsonl # MADBench：多智能体系统异常检测基准 MADBench是一个来自协作式多智能体系统（Multi-Agent System, MAS）执行轨迹的数据集，该系统用于解决程序化生成的密室逃脱谜题。其设计初衷是支撑**多智能体系统异常检测**领域的研究——即识别智能体协作在何时何地失效、错误如何在智能体间传播，以及区分健康执行与故障执行的特征。 ## 研究动机 多智能体系统会引入单智能体场景中不存在的故障模式：某一智能体的错误输出会静默污染下游智能体的输入，使得最终故障的根本原因难以追溯。MADBench提供了一个可控的实验环境，其中： - 每一步的**真实标签（ground truth）**均已知 - 智能体必须**按序协作**——错误会沿链路向下传播 - **多种陷阱类型**可生成逼真的干扰项，误导智能体决策 - **六款模型**与**三种温度参数设置**可生成覆盖不同难度层级的多样化故障分布 ## 任务环境 任务为多阶段密室逃脱谜题。每个房间包含各类仪器（如温度计、气压计）、内嵌数学应用题的线索，以及刻意设置的干扰项： | 陷阱类型 | 描述 | |---|---| | `fake_item` | 带有物理上不可能的读数的仪器 | | `fake_clue` | 引用无意义单位或仪器的线索 | | `trap_item_desc` | 带有误导性描述的真实仪器 | 要逃脱，智能体团队必须识别正确线索、求解内嵌的数学题、定位正确仪器，并应用计算得到的差值（需完成单位转换）以得到正确的最终读数。 ## 多智能体流水线 MAS采用固定的序贯架构，包含三类专业化智能体： [观察者] → OBSERVE_CLUE → 从伪造线索中识别真实线索 [线索求解器] → SOLVE_CLUE → 求解数学应用题 [观察者] → OBSERVE_ITEM → 定位正确仪器 [仪器管理器] → APPLY_DELTA → 计算新的仪器读数（需完成单位转换） [观察者] → OBSERVE_PUZZLE → 验证谜题是否已解决 每个智能体仅接收前序步骤的输出——任一智能体的推理错误都会未经修正地向下传播。 ### 智能体角色 | 角色 | 输入 | 输出 | |---|---|---| | `observer`（观察者） | 房间状态 | 识别出的线索/仪器/谜题解决状态 | | `clue_solver`（线索求解器） | 数学应用题 + 单位提示 | 数值答案 + 单位 | | `item_manager`（仪器管理器） | 仪器状态、差值、单位 | 新的仪器读数 | ## 数据集结构 traces/ ├── rooms.jsonl # 200条房间定义（分为简单/中等/困难难度） ├── nightmare_rooms.jsonl # 50条房间定义（噩梦难度，每个房间含4个谜题） ├── easy_nightmares/ # 初步噩梦模式运行结果（使用gpt-4.1、claude-sonnet-4） ├── run_{date}_{model}_{temp}/ # 18次常规运行 — 每次包含200条轨迹 └── run_{date}_{model}_{temp}_nightmare/ # 16次噩梦模式运行 — 每次包含50条轨迹 每个`run_*`文件夹包含每个房间对应的一个JSON文件： {timestamp}_EscapeRoom_{room_id}_{escaped|failed}.json ### 评估模型 | 模型 | 温度参数 | |---|---| | `gpt-4.1-mini` | 0.0、0.3、0.6 | | `gpt-4.1` | 0.0、0.3、0.6 | | `claude-sonnet-4-20250514` | 0.0、0.3、0.6 | | `Qwen2.5-14B-Instruct` | 0.0、0.3、0.6 | | `deepseek-reasoner` | 0.0、0.3、0.6 | | `gpt-5.4` | 0.0、0.3、0.6 | ## 轨迹JSON Schema jsonc { "config": { "system": { "architecture": "sequential" }, "llm": { "provider": "...", "model": "...", "temperature": 0.0, "max_tokens": 4096 }, "agents": [ /* 各角色的系统提示词 */ ] }, "room": { "room_id": "room_0000", "difficulty": "easy", // 可选值：easy | medium | hard | nightmare "n_puzzles": 1, // 噩梦模式房间包含4个连续谜题 "scenery": [ /* 干扰项物品列表 */ ], "puzzles": [ { "puzzle_id": "P1", "clue": { "hint": "...", // 仪器与单位的自然语言提示 "item_type": "thermometer", "problem": "...", // 数学应用题 "answer": 32.0, // 正确解 "delta_unit": "kelvin" }, "item": { "type": "thermometer", "state": 88, "unit": "celsius" }, "fake_items": [ /* 异常读数仪器 */ ], "fake_clues": [ /* 无意义提示线索 */ ], "traps": ["fake_item", "fake_clue"], "ground_truth": 120 // 预期最终仪器读数 } ] }, "trace": [ { "action": "OBSERVE_CLUE | SOLVE_CLUE | OBSERVE_ITEM | APPLY_DELTA | OBSERVE_PUZZLE", "agent": "observer | clue_solver | item_manager", "message": "...", // 智能体自由文本推理过程 "structured": { ... }, // 结构化JSON输出 "schema_errors": {}, // 异常信号：输出格式违规 "confidence": 0.95, "call_statistic": { "duration": 2.57, "input_tokens": 613, "output_tokens": 169, "timed_out": false, // 异常信号：调用超时 "superlong_reasoning": false // 异常信号：思维链失控（推理链路过长） }, "attempt": 1, "verification": { "status": "correct | wrong", // 该步骤的真实标签 ... } } ], "failure_report": { "schema_failures": [], // 结构化输出格式错误的步骤 "eval_results": ["correct"] // 各谜题的最终结果 }, "escaped": true, // 整体任务结果 "puzzles_solved": 1, "puzzles_total": 1, "timing_sec": 24.1 } ### 每条轨迹中的异常信号 | 字段 | 类型 | 描述 | |---|---|---| | `verification.status` | 步骤级标签 | `"wrong"`表示存在已知真实标签的智能体输出故障 | | `schema_errors` | 结构异常 | 智能体输出未符合要求的JSON Schema规范 | | `call_statistic.timed_out` | 行为异常 | 智能体调用超时 | | `call_statistic.superlong_reasoning` | 行为异常 | 思维链失控（推理链路过长） | | `failure_report.schema_failures` | 运行级汇总 | 该房间内的结构化格式违规聚合统计 | | `escaped` | 运行级标签 | 完整流水线是否成功完成任务 | ## 逃脱率结果 逃脱率 = 多智能体系统为所有谜题生成正确最终仪器读数的房间占比。 ### 常规房间（简单/中等/困难，共200个） | 模型 | 温度0.0 | 温度0.3 | 温度0.6 | |---|---|---|---| | deepseek-reasoner | **27%** | **30%** | **28%** | | gpt-5.4 | 23% | 22% | 25% | | claude-sonnet-4-20250514 | 22% | 23% | 22% | | gpt-4.1-mini | 18% | 20% | 18% | | gpt-4.1 | 16% | 16% | 18% | | Qwen2.5-14B-Instruct | 4% | 5% | 3% | ### 噩梦模式房间（4个连续谜题，共50个） | 模型 | 温度0.0 | 温度0.3 | 温度0.6 | |---|---|---|---| | deepseek-reasoner | **10%** | **6%** | **12%** | | gpt-5.4 | 2% | 2% | 2% | | gpt-4.1 | 0% | 0% | 0% | | claude-sonnet-4-20250514 | 0% | 0% | 0% | | Qwen2.5-14B-Instruct | 0% | 0% | 0% | 噩梦模式房间将4个谜题按序串联且不提供部分得分——异常会在各谜题间累积传播，因此该子集特别适合研究错误传播机制。 ## 加载方法 python import json, glob, os, re def load_run(run_dir): traces = [] for path in sorted(glob.glob(os.path.join(run_dir, "*.json"))): with open(path) as f: traces.append(json.load(f)) return traces # 遍历所有步骤并收集异常标签 def iter_steps(trace): for step in trace["trace"]: yield { "room_id": trace["room"]["room_id"], "difficulty": trace["room"]["difficulty"], "action": step["action"], "agent": step["agent"], "status": step["verification"].get("status"), // "correct" | "wrong" "schema_errors": bool(step.get("schema_errors")), "timed_out": step["call_statistic"]["timed_out"], "superlong": step["call_statistic"]["superlong_reasoning"], } # 示例：加载一次运行并收集所有异常步骤 run = load_run("run_20260316_174255_claude-sonnet-4-20250514_0.0") anomalies = [s for trace in run for s in iter_steps(trace) if s["status"] == "wrong"]
