synthiumjp/metacognitive-monitoring-battery

--- license: cc-by-4.0 task_categories: - text-classification language: - en tags: - benchmark - nelson-narens - cognitive-science - metacognition - llm-evaluation - signal-detection-theory - cognitive pretty_name: Metacognitive Monitoring Battery size_categories: - 10K<n<100K configs: - config_name: responses data_files: - split: train path: responses.csv - config_name: leaderboard data_files: - split: train path: leaderboard.csv default_config_name: responses --- # Metacognitive Monitoring Battery A cross-domain behavioural assay of monitoring-control coupling in LLMs, grounded in the Nelson and Narens (1990) metacognitive framework. **Paper:** [The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring](https://huggingface.co/papers/2604.15702) **Code:** [github.com/synthiumjp/metacognitive-monitoring-battery](https://github.com/synthiumjp/metacognitive-monitoring-battery) **Author:** Jon-Paul Cacioli (Independent Researcher, Melbourne, Australia) ## Overview The battery comprises **524 items** across **six cognitive domains**, each grounded in an established experimental paradigm. After every forced-choice response, dual probes adapted from Koriat and Goldsmith (1996) ask the model to KEEP or WITHDRAW its answer and to BET or decline. Applied to **20 frontier LLMs** (10,480 evaluations), the battery discriminates three behavioural profiles consistent with the Nelson-Narens monitoring-control architecture: - **Profile A — Blanket Confidence:** KEEP on 95%+ of items regardless of correctness - **Profile B — Blanket Withdrawal:** WITHDRAW on 91-99% of items (DeepSeek R1 only) - **Profile C — Selective Sensitivity:** Withdraw delta 15%+ (coupled monitoring-control) ## Tracks | Track | Domain | Items | Paradigm | |---|---|---|---| | T1 | Learning | 98 | Overhypothesis induction (Kemp et al., 2007) | | T2 | Metacognition | 90 | SDT calibration (Green & Swets, 1966) | | T3 | Social Cognition | 116 | Mutual exclusivity & pragmatics (Markman & Wachtel, 1988) | | T4 | Attention | 60 | Biased competition (Desimone & Duncan, 1995) | | T5 | Executive Function | 88 | Weber's Law & flexibility (Dehaene, 2003; Diamond, 2013) | | T6 | Prospective Regulation | 72 | Help-seeking (Metcalfe & Kornell, 2005) | ## Key columns - `model` — Canonical model name (20 frontier LLMs) - `track` — T1 through T6 - `correct` — Whether the forced-choice answer matched ground truth - `keep_withdraw` — KEEP (commit to answer) or WITHDRAW (retract) - `bet_nobet` — BET (high confidence) or NO_BET (low confidence) - `item_type` — Track-specific condition label - `path_choice` — T6 only: ANSWER_DIRECTLY, REQUEST_HINT, or DECLINE ## Usage ```python from datasets import load_dataset ds = load_dataset("synthiumjp/metacognitive-monitoring-battery") df = ds['train'].to_pandas() # Compute withdraw delta for Sonnet on T2 sonnet_t2 = df[(df['model'] == 'Claude Sonnet 4.6') & (df['track'] == 'T2')] correct = sonnet_t2[sonnet_t2['correct'] == 'True'] incorrect = sonnet_t2[sonnet_t2['correct'] == 'False'] keep_c = (correct['keep_withdraw'] == 'KEEP').mean() * 100 keep_i = (incorrect['keep_withdraw'] == 'KEEP').mean() * 100 print(f"Sonnet T2 WD = {keep_c - keep_i:+.1f}%") # +14.3% ``` ## Citation ```bibtex @article{cacioli2026mmb, title={The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring}, author={Cacioli, Jon-Paul}, journal={arXiv preprint arXiv:2604.15702}, year={2026} } ``` ## License CC-BY-4.0 (data). MIT (analysis code in the GitHub repository).license: CC BY 4.0 task_categories: - 文本分类（Text Classification） language: 英语 tags: - 基准测试（Benchmark） - 纳尔逊-纳伦斯（Nelson-Narens） - 认知科学（Cognitive Science） - 元认知（Metacognition） - 大语言模型（Large Language Model）评估 - 信号检测论（Signal Detection Theory） - 认知（Cognitive） pretty_name: 元认知监测电池（Metacognitive Monitoring Battery） size_categories: - 10000 < 样本数 < 100000 configs: - config_name: responses data_files: - split: train path: responses.csv - config_name: leaderboard data_files: - split: train path: leaderboard.csv default_config_name: responses # 元认知监测电池（Metacognitive Monitoring Battery） 本数据集是针对大语言模型（Large Language Model）中监测-控制耦合的跨领域行为测试，基于纳尔逊与纳伦斯（1990）提出的元认知框架构建。 **论文：** [《元认知监测电池：面向大语言模型自我监测的跨领域基准测试》](https://huggingface.co/papers/2604.15702) **代码：** [github.com/synthiumjp/metacognitive-monitoring-battery](https://github.com/synthiumjp/metacognitive-monitoring-battery) **作者：** 乔恩-保罗·卡乔利（澳大利亚墨尔本独立研究员） ## 概述 该任务集包含覆盖**6个认知领域**的**524道测试项**，每一类均基于成熟的实验范式构建。在每一次迫选作答后，采用改编自克里亚特与戈德史密斯（1996）的双探针设计，要求模型选择**保留（KEEP）或撤回（WITHDRAW）** 其作答结果，并选择**下注（BET）或放弃下注（decline）**。 本数据集已在**20款前沿大语言模型**上完成测试（总计10480次评估），能够区分出符合纳尔逊-纳伦斯监测-控制架构的三类行为模式： - **模式A — 盲目自信：** 无论作答正确与否，在95%以上的测试项中选择保留作答 - **模式B — 全盘撤回：** 仅在91%-99%的测试项中选择撤回作答（仅DeepSeek R1出现该模式） - **模式C — 选择性敏感：** 撤回率差值≥15%（符合耦合式监测-控制机制） ## 测试赛道 | 赛道编号 | 领域 | 测试项数量 | 实验范式 | |---|---|---|---| | T1 | 学习 | 98 | 超假设归纳（Kemp等，2007） | | T2 | 元认知 | 90 | 信号检测论（Signal Detection Theory）校准（Green & Swets，1966） | | T3 | 社会认知 | 116 | 互斥性与语用学（Markman & Wachtel，1988） | | T4 | 注意力 | 60 | 偏向性竞争（Desimone & Duncan，1995） | | T5 | 执行功能 | 88 | 韦伯定律与认知灵活性（Dehaene，2003；Diamond，2013） | | T6 | 前瞻调控 | 72 | 求助行为（Metcalfe & Kornell，2005） | ## 关键字段 - `model`：模型标准名称（对应20款前沿大语言模型） - `track`：测试赛道编号（T1至T6） - `correct`：迫选作答是否与标准答案一致 - `keep_withdraw`：保留作答（KEEP，即确认答案）或撤回作答（WITHDRAW，即收回答案） - `bet_nobet`：下注（BET，即高置信度）或不下注（NO_BET，即低置信度） - `item_type`：赛道专属的条件标签 - `path_choice`：仅T6赛道可用：直接作答（ANSWER_DIRECTLY）、请求提示（REQUEST_HINT）或拒绝作答（DECLINE） ## 使用方法 python from datasets import load_dataset ds = load_dataset("synthiumjp/metacognitive-monitoring-battery") df = ds['train'].to_pandas() # 计算Claude Sonnet 4.6在T2赛道的撤回率差值 sonnet_t2 = df[(df['model'] == 'Claude Sonnet 4.6') & (df['track'] == 'T2')] correct = sonnet_t2[sonnet_t2['correct'] == 'True'] incorrect = sonnet_t2[sonnet_t2['correct'] == 'False'] keep_c = (correct['keep_withdraw'] == 'KEEP').mean() * 100 keep_i = (incorrect['keep_withdraw'] == 'KEEP').mean() * 100 print(f"Sonnet T2 WD = {keep_c - keep_i:+.1f}%") # 示例输出：+14.3% ## 引用格式 bibtex @article{cacioli2026mmb, title={The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring}, author={Cacioli, Jon-Paul}, journal={arXiv preprint arXiv:2604.15702}, year={2026} } ## 许可证 数据集采用CC BY 4.0许可证，GitHub仓库中的分析代码采用MIT许可证。