synthiumjp/metacognitive-profile-atlas

元认知轮廓图集提供了33种前沿大型语言模型（LLMs）在六个认知领域上的47,151个（答案，置信度）观察数据，用于评估LLMs如何监控其自身准确性。每个观察结果包含模型的答案、其表达的置信度（0-100）以及答案是否正确。数据集支持对Type-2 AUROC（置信度是否区分正确和错误响应）的领域级分解，揭示了聚合指标所掩盖的模型内部变化。数据集结构包括33个CSV文件，每个文件对应一个模型，包含该模型尝试的每个项目的一行数据。此外，还包括一个引导置信区间文件（atlas_bootstrap_cis.csv），包含所有模型-领域单元的198行引导95%置信区间数据。

The Metacognitive Profile Atlas provides item-level verbalized-confidence data for evaluating how well LLMs monitor their own accuracy, decomposed by cognitive domain. Each observation is one (model, item) pair containing the models answer, its verbalized confidence (0-100), and whether the answer was correct. The dataset enables domain-level decomposition of Type-2 AUROC (whether confidence discriminates correct from incorrect responses), revealing within-model variation that aggregate metrics obscure. The dataset structure includes 33 CSVs, one per model, each containing one row per item attempted by that model. Additionally, there is a bootstrap confidence intervals file (atlas_bootstrap_cis.csv) containing 198 rows of bootstrap 95% CIs for all model-domain cells.