iamseungpil/metacognition-behavior-uncertainty-snapshot

# Four Habits Mechanism Lab Project root: `/home/v-seungplee/metacognition-behavior-uncertainty` This repository studies one question: **Why do the Four Habits improve reasoning performance?** ## First Read There are two papers in scope, and they are not the same experiment: 1. `Four Habits` paper - this repository's main target - exact-paper question: are data generation, SFT, PPO, and behavioral evaluation being run the same way? 2. `epistemic analysis` paper - used here as a separate analysis layer - fixed-prefix and token-suppression interventions belong here, not to the original Four Habits training recipe ## Current Answer The repository now reconstructs the Four Habits public experiment structure correctly, but the current local pipeline is **not yet an exact paper-method rerun**. Why: 1. the released priming generator uses `claude-3-5-sonnet-20241022` 2. the current shell does not expose `ANTHROPIC_API_KEY` 3. paper-style raw priming assets are not present locally 4. the local evaluation path is a portable wrapper rather than the released `gpt-4o-mini` batch path 5. the current derivative priming plan uses `TRAPI + gpt-5.4`, which is not the paper's exact Claude generator ## What Is Valid Right Now ### A. Exactness and Release Audits The repository can now audit: 1. the Four Habits dataset and condition structure 2. the released SFT, PPO, and behavioral-eval chain 3. the gap between exact-paper execution and the current local setup 4. the gap between the released script paths and public Hugging Face assets ### B. Public Executable Baseline The repository also has one valid executable public baseline: 1. model: `obiwan96/qwen-cd-100` 2. dataset: `obiwan96/countdown-env` `eval` 3. node: reserved 4-GPU analysis node Current synced summary: 1. `n_samples = 100` 2. `accuracy = 0.22` 3. `mean_avg_logprob = -0.0737` 4. `approx_mean_entropy = 0.1847` ### C. Public Intervention Analysis The repository has also executed an epistemic-style intervention sweep on that same public checkpoint: 1. `baseline`: `accuracy=0.22`, `entropy=0.1847` 2. `fixed_prefix_okay_so_i`: `accuracy=0.11`, `entropy=0.5693` 3. `suppress_epistemic_tokens`: `accuracy=0.22`, `entropy=0.1847` Current read: 1. the fixed prefix damages the released public model 2. the currently tracked epistemic lexical tokens are not carrying the public baseline 3. the strongest visible useful behavior is lightweight verification ### D. Derivative TRAPI Priming Path The repository now supports a derivative priming path based on: 1. original Four Habits condition prompts 2. `TRAPI` as the API transport 3. `gpt-5.4` as the generator model This path is useful for a controlled follow-up study, but it is not an exact-paper priming run. Current smoke status: 1. all five core habit conditions now have derivative raw JSON outputs 2. all five core habit conditions now have derivative `train.parquet` and `test.parquet` outputs 3. this confirms derivative infrastructure readiness, not paper-faithful learning-stage reproduction ## What Is Not Valid To Claim Yet Do not currently claim: 1. exact Four Habits data generation 2. exact Four Habits learning-stage rerun 3. exact Four Habits behavioral evaluation rerun 4. learning-stage causal conclusions about why each habit improves performance ## Repository Layout Core documents: 1. `PLAN.md` - full experiment plan in `Intent / Hypothesis / Validation Method / Current Result` form 2. `CURRENT_STATUS.md` - current exactness and execution state 3. `NODE_POLICY.md` - node policy and runtime notes 4. `docs/EXPERIMENT_DESIGN.md` - experiment-stage design 5. `docs/TRAINING_TRACKS.md` - exact-vs-derivative training split 6. `docs/EPISTEMIC_ANALYSIS_PLAN.md` - entropy and intervention analysis plan 7. `docs/EXTERNAL_SOURCES.md` - upstream provenance Core scripts: 1. `scripts/run_smoke.py` 2. `scripts/audit_four_habits_repro.py` 3. `scripts/audit_public_release_closure.py` 4. `scripts/audit_exact_method_alignment.py` 5. `scripts/prepare_training_study.py` 6. `scripts/prepare_epistemic_analysis.py` 7. `scripts/run_critic.py` 8. `scripts/render_report.py` 9. `scripts/build_working_note_pdf.sh` ## External Repositories The local source-of-truth repositories are: 1. `external/cognitive-behaviors` 2. `external/strategic-information-allocation-llm-reasoning` ## Recommended Audit Loop ```bash cd /home/v-seungplee/metacognition-behavior-uncertainty bash scripts/run_loop.sh bash scripts/build_working_note_pdf.sh ``` ## Exact Training Gate Before any honest exact-paper learning-stage run, the repository still needs: 1. exact paper-style priming assets or exact Claude-backed priming access 2. exact learning-stage inputs with explicit provenance 3. an exact behavioral-eval path or an explicitly documented reason for any deviation Until then, this repository should be read as: 1. an exact-structure audit 2. a public-baseline mechanism study 3. a guarded derivative-training scaffold# 四习惯（Four Habits）机制实验室 项目根目录：`/home/v-seungplee/metacognition-behavior-uncertainty` 本仓库旨在探究一个核心问题： **四习惯为何能够提升推理性能？** ## 初始导读 本次研究涉及两篇论文，且二者并非同一实验： 1. **四习惯论文**：本仓库的核心研究目标；其核心探究问题为：数据生成、监督微调（Supervised Fine-Tuning, SFT）、近端策略优化（Proximal Policy Optimization, PPO）以及行为评估的执行流程是否完全一致？ 2. **认知分析论文**：本仓库将其作为独立的分析层；固定前缀（fixed-prefix）与Token（Token）抑制干预手段归属于该论文范畴，而非原始四习惯训练流程。 ## 当前研究结论 本仓库现已正确复现四习惯公开实验的结构框架，但当前本地流水线**尚未实现与论文方法完全一致的重跑**。 原因如下： 1. 公开的提示生成器采用了`claude-3-5-sonnet-20241022`模型 2. 当前脚本未配置`ANTHROPIC_API_KEY`密钥 3. 本地未存储符合论文规范的原始提示资源 4. 本地评估路径采用可移植封装脚本，而非公开的`gpt-4o-mini`批量评估路径 5. 当前衍生提示生成方案采用`TRAPI + gpt-5.4`，与论文中使用的Claude生成模型不完全一致。 ## 当前有效可行的研究环节 ### A. 精准性与公开版本审计 本仓库现已支持以下审计任务： 1. 四习惯数据集与实验条件结构 2. 公开的监督微调、近端策略优化与行为评估流水线 3. 论文标准执行流程与当前本地配置之间的差异 4. 公开脚本路径与Hugging Face公开资源之间的差异 ### B. 公开可执行基线模型 本仓库还提供一套可直接运行的公开基线方案： 1. 模型：`obiwan96/qwen-cd-100` 2. 数据集：`obiwan96/countdown-env` 的评估子集 3. 运行节点：预留的4GPU分析节点 当前同步的实验统计结果： 1. 样本量：`n_samples = 100` 2. 准确率：`accuracy = 0.22` 3. 平均对数概率均值：`mean_avg_logprob = -0.0737` 4. 近似平均熵：`approx_mean_entropy = 0.1847` ### C. 公开干预分析 本仓库还针对该公开基线模型完成了认知风格的干预遍历实验： 1. 基线组：`baseline`，准确率`accuracy=0.22`，熵值`entropy=0.1847` 2. 固定前缀组：`fixed_prefix_okay_so_i`，准确率`accuracy=0.11`，熵值`entropy=0.5693` 3. 认知Token抑制组：`suppress_epistemic_tokens`，准确率`accuracy=0.22`，熵值`entropy=0.1847` 当前实验解读： 1. 固定前缀会损害公开基线模型的性能 2. 当前追踪的认知类词汇Token并未承载基线模型的核心行为模式 3. 目前观测到的最有效的行为为轻量级验证操作。 ### D. 衍生TRAPI提示生成路径 本仓库现已支持基于以下框架的衍生提示生成路径： 1. 原始四习惯实验条件提示词 2. `TRAPI`作为API传输协议 3. `gpt-5.4`作为生成模型 该路径可用于可控的后续研究，但并非符合论文规范的提示生成重跑流程。 当前冒烟测试状态： 1. 五大核心习惯条件均已生成衍生原始JSON输出 2. 五大核心习惯条件均已生成衍生的`train.parquet`与`test.parquet`数据集文件 3. 上述结果证明衍生实验基础设施已就绪，但尚未实现符合论文规范的训练阶段复现。 ## 当前尚未可宣称的研究结论 目前不应宣称以下内容： 1. 已实现完全符合论文规范的四习惯数据生成 2. 已实现完全符合论文规范的四习惯训练阶段重跑 3. 已实现完全符合论文规范的四习惯行为评估重跑 4. 已得出关于各习惯提升性能的训练阶段因果性结论。 ## 仓库文件结构 ### 核心文档 1. `PLAN.md`：以“目标 / 假设 / 验证方法 / 当前结果”格式撰写的完整实验方案 2. `CURRENT_STATUS.md`：当前实验精准性与执行状态说明 3. `NODE_POLICY.md`：运行节点策略与运行时注意事项 4. `docs/EXPERIMENT_DESIGN.md`：实验阶段设计文档 5. `docs/TRAINING_TRACKS.md`：精准复现与衍生训练的分支说明 6. `docs/EPISTEMIC_ANALYSIS_PLAN.md`：熵值与干预分析方案 7. `docs/EXTERNAL_SOURCES.md`：上游资源溯源说明 ### 核心脚本 1. `scripts/run_smoke.py` 2. `scripts/audit_four_habits_repro.py` 3. `scripts/audit_public_release_closure.py` 4. `scripts/audit_exact_method_alignment.py` 5. `scripts/prepare_training_study.py` 6. `scripts/prepare_epistemic_analysis.py` 7. `scripts/run_critic.py` 8. `scripts/render_report.py` 9. `scripts/build_working_note_pdf.sh` ## 外部依赖仓库 本地权威源仓库包括： 1. `external/cognitive-behaviors` 2. `external/strategic-information-allocation-llm-reasoning` ## 推荐审计循环流程 bash cd /home/v-seungplee/metacognition-behavior-uncertainty bash scripts/run_loop.sh bash scripts/build_working_note_pdf.sh ## 精准训练准入门槛 在开展任何符合论文规范的训练阶段重跑之前，本仓库仍需完成以下准备： 1. 获取符合论文规范的原始提示资源，或获得基于Claude的提示生成权限 2. 获取带有明确溯源信息的标准训练阶段输入数据 3. 部署符合论文规范的行为评估路径，或对所有偏离规范的操作提供书面说明 在此之前，本仓库的定位应为： 1. 精准结构审计工具 2. 公开基线模型机制研究平台 3. 受管控的衍生训练脚手架。