Be2Jay/hallumaze-benchmark

--- license: mit task_categories: - text-generation language: - en tags: - hallucination - metacognition - benchmark - llm-evaluation - maze - error-recovery pretty_name: HalluMaze Benchmark size_categories: - n<1K configs: - config_name: trials data_files: - split: train path: experiment_results/or_haiku.json - split: train path: experiment_results/or_maverick.json - split: train path: experiment_results/or_gptmini.json - split: train path: experiment_results/or_qwen.json - split: train path: experiment_results/or_phaseB.json - split: train path: experiment_results/checkpoint_rerun.json - split: train path: experiment_results/or_phaseC.json - config_name: analysis data_files: - split: train path: experiment_results/analysis_final2.json - config_name: failure_modes data_files: - split: train path: experiment_results/failure_modes.json - config_name: calibration data_files: - split: train path: experiment_results/calibration.json --- # HalluMaze Benchmark Dataset > **All 10 tested LLMs score significantly below a random walk on metacognitive recovery (p<0.001, Glass's δ=0.6–2.1). Frontier cost does not predict performance: GPT-4o ranks last (MEI=0.315), Claude-3.7-Sonnet ranks first (MEI=0.774).** ## Dataset Description HalluMaze measures **metacognitive error recovery** in LLMs through maze navigation. Models are exposed to "mirage" walls — passages that appear blocked but are traversable — testing real-time belief updating. **Key finding**: A random walk agent (MEI=0.900) outperforms all 10 tested LLMs (best: Claude-3.7-Sonnet, MEI=0.774), revealing a systematic deficit in metacognitive error recovery across all model families and cost tiers. - **Paper**: [HalluMaze: A Maze Navigation Benchmark for LLM Metacognitive Error Recovery](https://github.com/jaytoone/HalluMaze) - **Demo**: [HuggingFace Space](https://huggingface.co/spaces/Be2Jay/hallumaze) - **GitHub**: [jaytoone/HalluMaze](https://github.com/jaytoone/HalluMaze) ## Leaderboard (MEI ↑, n=60 per model) | Rank | Model | MEI [95% CI] | SR | HRR | Glass's δ | |------|-------|--------------|-----|-----|-----------| | — | Random Walk ★ | 0.900 | 100% | 100% | — | | 1 | **Claude-3.7-Sonnet** | **0.774** [0.715, 0.830] | 56.7% | 87.5% | 0.554 | | 2 | GLM-4.7 | 0.615 [0.551, 0.681] | 8.3% | 71.8% | 1.102 | | 3 | Llama-4-Maverick | 0.600 [0.541, 0.660] | 13.3% | 81.1% | 1.254 | | 4 | MiniMax-M2.5 | 0.593 [0.500, 0.682] | 53.3% | 60.0% | 0.847 | | 5 | Llama-4-Scout | 0.589 [0.525, 0.649] | 8.3% | 81.0% | 1.230 | | 6 | Qwen-2.5-72B | 0.559 [0.488, 0.629] | 10.0% | 60.7% | 1.223 | | 7 | Gemini-2.0-Flash-Lite | 0.432 [0.352, 0.507] | 8.3% | 40.3% | 1.557 | | 8 | Claude-3-Haiku | 0.398 [0.341, 0.457] | 5.0% | 36.3% | 2.129 | | 9 | GPT-4o-mini | 0.391 [0.310, 0.467] | 5.0% | 38.2% | 1.620 | | 10 | **GPT-4o** | **0.315** [0.239, 0.394] | 6.7% | 35.3% | 1.917 | ★ Deterministic baseline. All LLMs vs Random Walk: one-sample Wilcoxon signed-rank test, Bonferroni k=10, all p<0.001. ## Dataset Structure ### Files | File | Description | Records | |------|-------------|---------| | `experiment_results/or_haiku.json` | Claude-3-Haiku trials | 60 | | `experiment_results/or_maverick.json` | Llama-4-Maverick trials | 60 | | `experiment_results/or_gptmini.json` | GPT-4o-mini trials | 60 | | `experiment_results/or_qwen.json` | Qwen-2.5-72B trials | 60 | | `experiment_results/or_phaseB.json` | Llama-4-Scout + Gemini trials | 120 | | `experiment_results/checkpoint_rerun.json` | MiniMax-M2.5 + GLM-4.7 trials | 120 | | `experiment_results/or_phaseC.json` | Claude-3.7-Sonnet + GPT-4o trials | 120 | | `experiment_results/analysis_final2.json` | Final aggregated stats (Bootstrap CI + Wilcoxon, k=10) | — | | `experiment_results/baselines.json` | Random Walk / A* / BFS baselines | — | | `experiment_results/failure_modes.json` | Failure taxonomy (TYPE_A/B/C/S) | 480 | | `experiment_results/calibration.json` | Confidence calibration (ECE, Brier) | — | | `experiment_results/mei_sensitivity.json` | 625-config weight sensitivity analysis | — | ### Trial Record Schema ```json { "seed": 1001, "size": 5, "or_model_id": "anthropic/claude-3-haiku", "solved": false, "mei": 0.412, "sr": 0, "hrr": 0.4, "etr": 0.6, "aw": 0.5, "hr": 0.2, "brs": 0.8, "hallucination_count": 2, "backtrack_count": 1, "loop_count": 0, "path": [[0,0], [0,1], "..."], "ce": 0.75 } ``` ## Metrics **MEI (Metacognitive Escape Index)** — primary composite metric: ``` MEI = 0.4 × HRR + 0.3 × ETR + 0.2 × AW − 0.1 × HR ``` | Metric | Full Name | Description | |--------|-----------|-------------| | MEI | Metacognitive Escape Index | Primary composite metric | | HRR | Hallucination Recovery Rate | P(correct backtrack \| hallucination detected) | | ETR | Efficiency Ratio | Path quality relative to optimal | | AW | Awareness | Loop detection and redundancy avoidance | | HR | Hallucination Rate | Rate of erroneous wall belief | | SR | Solve Rate | P(reach goal within step budget) | | BRS | Backtrack Rationality Score | Quality of backtrack decisions | Weight sensitivity: 625-configuration grid search (±50% per weight) confirms random walk > all LLMs in 100% of configurations. ## Experimental Setup - **Evaluation design**: Single-call — LLMs generate the complete navigation path in one API call - **Maze algorithm**: Recursive DFS with 2 mirage positions per maze - **Seeds**: 1001, 2002, 3003, 4004, 5005 (×2 sizes = 10 mazes/seed group × 6 = 60 trials/model) - **Maze sizes**: 5×5 and 7×7 - **Random walk baseline**: N²×100 step budget; ETR normalization uses N² - **Bootstrap CI**: n_boot=2000, ci=0.95, seed=42 - **Statistical test**: One-sample Wilcoxon signed-rank test vs μ₀=0.9, Bonferroni k=10 - **Effect size**: Glass's delta (constant baseline, zero variance) ## Citation ```bibtex @misc{hallumaze2026, title = {HalluMaze: A Maze Navigation Benchmark for LLM Metacognitive Error Recovery}, author = {Jayone}, year = {2026}, url = {https://github.com/jaytoone/HalluMaze} } ``` ## License MIT Licenselicense: MIT 许可证 task_categories: - 文本生成 language: - 英语 tags: - 幻觉（hallucination） - 元认知（metacognition） - 基准测试（benchmark） - 大语言模型评估（llm-evaluation） - 迷宫（maze） - 错误恢复（error-recovery） pretty_name: HalluMaze基准测试 size_categories: - 样本数量少于1000 configs: - config_name: trials data_files: - split: 训练集 path: experiment_results/or_haiku.json - split: 训练集 path: experiment_results/or_maverick.json - split: 训练集 path: experiment_results/or_gptmini.json - split: 训练集 path: experiment_results/or_qwen.json - split: 训练集 path: experiment_results/or_phaseB.json - split: 训练集 path: experiment_results/checkpoint_rerun.json - split: 训练集 path: experiment_results/or_phaseC.json - config_name: analysis data_files: - split: 训练集 path: experiment_results/analysis_final2.json - config_name: failure_modes data_files: - split: 训练集 path: experiment_results/failure_modes.json - config_name: calibration data_files: - split: 训练集 path: experiment_results/calibration.json # HalluMaze基准测试数据集 > **经测试的10款大语言模型（LLM）在元认知恢复任务中的表现均显著劣于随机游走策略（p<0.001，格拉斯Δ值（Glass's δ）=0.6–2.1）。模型的前沿成本无法预测其表现：GPT-4o排名垫底（MEI=0.315），Claude-3.7-Sonnet位居榜首（MEI=0.774）。** ## 数据集说明 HalluMaze通过迷宫导航任务，评估大语言模型的**元认知错误恢复（metacognitive error recovery）**能力。模型会遭遇“幻影墙壁”——看似被阻挡实则可通行的通道——以此测试模型的实时信念更新能力。 **核心发现**：随机游走智能体（MEI=0.900）的表现优于所有10款经测试的大语言模型（最优者为Claude-3.7-Sonnet，MEI=0.774），这表明所有模型家族与成本层级的大语言模型均存在元认知错误恢复的系统性缺陷。 - **论文**：[HalluMaze: A Maze Navigation Benchmark for LLM Metacognitive Error Recovery](https://github.com/jaytoone/HalluMaze) - **演示Demo**：[HuggingFace Space](https://huggingface.co/spaces/Be2Jay/hallumaze) - **GitHub仓库**：[jaytoone/HalluMaze](https://github.com/jaytoone/HalluMaze) ## 排行榜（MEI ↑，每个模型n=60） | 排名 | 模型 | MEI [95%置信区间] | 通关率（SR） | 幻觉恢复率（HRR） | 格拉斯Δ值（Glass's δ） | |------|-------|--------------|-----|-----|-----------| | — | 随机游走 ★ | 0.900 | 100% | 100% | — | | 1 | **Claude-3.7-Sonnet** | **0.774** [0.715, 0.830] | 56.7% | 87.5% | 0.554 | | 2 | GLM-4.7 | 0.615 [0.551, 0.681] | 8.3% | 71.8% | 1.102 | | 3 | Llama-4-Maverick | 0.600 [0.541, 0.660] | 13.3% | 81.1% | 1.254 | | 4 | MiniMax-M2.5 | 0.593 [0.500, 0.682] | 53.3% | 60.0% | 0.847 | | 5 | Llama-4-Scout | 0.589 [0.525, 0.649] | 8.3% | 81.0% | 1.230 | | 6 | Qwen-2.5-72B | 0.559 [0.488, 0.629] | 10.0% | 60.7% | 1.223 | | 7 | Gemini-2.0-Flash-Lite | 0.432 [0.352, 0.507] | 8.3% | 40.3% | 1.557 | | 8 | Claude-3-Haiku | 0.398 [0.341, 0.457] | 5.0% | 36.3% | 2.129 | | 9 | GPT-4o-mini | 0.391 [0.310, 0.467] | 5.0% | 38.2% | 1.620 | | 10 | **GPT-4o** | **0.315** [0.239, 0.394] | 6.7% | 35.3% | 1.917 | ★ 确定性基准。所有大语言模型与随机游走策略对比：采用单样本Wilcoxon符号秩检验，Bonferroni校正k=10，所有p<0.001。 ## 数据集结构 ### 数据文件 | 文件路径 | 描述 | 记录数 | |------|-------------|---------| | `experiment_results/or_haiku.json` | Claude-3-Haiku的试验数据 | 60 | | `experiment_results/or_maverick.json` | Llama-4-Maverick的试验数据 | 60 | | `experiment_results/or_gptmini.json` | GPT-4o-mini的试验数据 | 60 | | `experiment_results/or_qwen.json` | Qwen-2.5-72B的试验数据 | 60 | | `experiment_results/or_phaseB.json` | Llama-4-Scout与Gemini的试验数据 | 120 | | `experiment_results/checkpoint_rerun.json` | MiniMax-M2.5与GLM-4.7的试验数据 | 120 | | `experiment_results/or_phaseC.json` | Claude-3.7-Sonnet与GPT-4o的试验数据 | 120 | | `experiment_results/analysis_final2.json` | 最终聚合统计数据（Bootstrap置信区间 + Wilcoxon检验，k=10） | — | | `experiment_results/baselines.json` | 随机游走、A*、BFS基准数据 | — | | `experiment_results/failure_modes.json` | 错误分类体系（TYPE_A/B/C/S） | 480 | | `experiment_results/calibration.json` | 置信度校准数据（ECE、Brier评分） | — | | `experiment_results/mei_sensitivity.json` | 625组权重敏感性分析数据 | — | ### 试验记录Schema json { "seed": 1001, "size": 5, "or_model_id": "anthropic/claude-3-haiku", "solved": false, "mei": 0.412, "sr": 0, "hrr": 0.4, "etr": 0.6, "aw": 0.5, "hr": 0.2, "brs": 0.8, "hallucination_count": 2, "backtrack_count": 1, "loop_count": 0, "path": [[0,0], [0,1], "..."], "ce": 0.75 } ## 评估指标 **元认知逃逸指数（Metacognitive Escape Index, MEI）**——核心复合指标： MEI = 0.4 × HRR + 0.3 × ETR + 0.2 × AW − 0.1 × HR | 指标缩写 | 指标全称 | 指标说明 | |--------|-----------|-------------| | MEI | 元认知逃逸指数 | 核心复合评估指标 | | HRR | 幻觉恢复率（Hallucination Recovery Rate） | P(正确回溯 | 检测到幻觉) | | ETR | 效率比（Efficiency Ratio） | 路径质量相对于最优路径的比值 | | AW | 觉知度（Awareness） | 环路检测与冗余路径规避能力 | | HR | 幻觉率（Hallucination Rate） | 错误墙体信念的发生比例 | | SR | 通关率（Solve Rate） | P(在步长预算内抵达目标) | | BRS | 回溯合理性评分（Backtrack Rationality Score） | 回溯决策的质量 | 权重敏感性分析：通过625组配置的网格搜索（每个权重±50%范围）验证，在100%的配置下，随机游走策略的表现均优于所有大语言模型。 ## 实验设置 - **评估设计**：单轮API调用——大语言模型通过一次接口请求生成完整的导航路径 - **迷宫算法**：每座迷宫设置2处幻影墙壁的递归深度优先搜索（DFS） - **随机种子**：1001、2002、3003、4004、5005（搭配2种迷宫尺寸 = 每组种子对应10座迷宫 ×6 = 每个模型60次试验） - **迷宫尺寸**：5×5与7×7 - **随机游走基准**：步长预算为N²×100；ETR归一化采用N²（N为迷宫边长） - **Bootstrap置信区间**：重采样次数n_boot=2000，置信水平=0.95，随机种子=42 - **统计检验**：以μ₀=0.9为基准的单样本Wilcoxon符号秩检验，Bonferroni校正k=10 - **效应量**：格拉斯Δ值（固定基准，零方差） ## 引用格式 bibtex @misc{hallumaze2026, title = {HalluMaze: A Maze Navigation Benchmark for LLM Metacognitive Error Recovery}, author = {Jayone}, year = {2026}, url = {https://github.com/jaytoone/HalluMaze} } ## 许可证 MIT 许可证