eduardofarina/seed-of-thought-healthcare

--- license: mit tags: - medical - healthcare - differential-diagnosis - prompting - seed-of-thought - simplestrat - medqa size_categories: - n<1K --- # 🌱 Seed of Thought for Healthcare Differential Diagnosis Implementation of the **SimpleStrat / Seed-of-Thought** methodology ([Wong et al., 2024, arXiv:2410.09038](https://arxiv.org/abs/2410.09038)) applied to clinical differential diagnosis on [MedQA-USMLE](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options). ## 🧬 The Challenge LLMs exhibit **mode collapse** in clinical diagnosis — they repeatedly predict the same top-1 diagnosis, ignoring the full differential. This is dangerous in medicine where rare but serious conditions must be considered. Standard temperature sampling doesn't help because it adds noise, not systematic diversity. ## 🌱 Seed of Thought Methodology The pipeline has 3 stages: ### Stage 1: Auto-Stratification The LLM identifies **discriminating clinical properties** that split the answer space into roughly equal halves. For 4-option MCQs, ideal properties split 2:2. - Example: "Does the mechanism involve direct cellular damage?" → Options A,C satisfy; B,D don't ### Stage 2: Heuristic Estimation For each property, the LLM estimates the probability using calibrated Bayesian reasoning (inspired by superforecasting techniques from Tetlock). ### Stage 3: Probabilistic Prompting A stratum (combination of property True/False values) is **sampled** from the joint distribution. The LLM generates an answer **constrained** to that stratum. Different samples explore different regions of the answer space. ## 📊 Results | Metric | Baseline (temp=0.9) | Seed of Thought | Δ | |--------|---------------------|-----------------|---| | accuracy | 0.9333 | 0.9333 | +0.0000 | | diversity_ratio | 0.1300 | 0.2533 | +0.1233 | | unique_count | 1.3000 | 2.5333 | +1.2333 | | coverage | 0.3417 | 0.5667 | +0.2250 | | kl_divergence_from_uniform | 1.2773 | 0.8354 | -0.4419 | ### Setup - **Model**: Qwen/Qwen2.5-72B-Instruct - **Dataset**: MedQA-USMLE test split (30 questions) - **Samples/question**: 10 - **Strata**: 3 - **Baseline temp**: 0.9 | **SoT temp**: 0.7 - **Total calls**: 840 | **Time**: 202.8min ## 🔑 Key Insight By partitioning the diagnostic space into orthogonal dimensions and sampling systematically, Seed-of-Thought achieves: 1. **Higher coverage** — more plausible diagnoses considered 2. **Better diversity** — genuinely different answers per sample 3. **Lower KL divergence** — more uniform distribution across valid options 4. **Maintained accuracy** — correct answer still appears in generated set ## Citation ```bibtex @article{wong2024simplestrat, title={SimpleStrat: Diversifying Language Model Generation with Stratification}, author={Wong, Justin and Orlovskiy, Yury and Luo, Michael and Seshia, Sanjit A. and Gonzalez, Joseph E.}, journal={arXiv preprint arXiv:2410.09038}, year={2024} } ```--- license: 麻省理工学院（MIT）协议 tags: - 医疗 - 医疗保健 - 鉴别诊断 - 提示工程 - 思维种子（Seed-of-Thought） - SimpleStrat - MedQA size_categories: - 样本数少于1000 --- # 🌱 面向医疗鉴别诊断的思维种子（Seed-of-Thought）方法 本数据集实现了**SimpleStrat/思维种子（Seed-of-Thought）**方法论（Wong等，2024，arXiv:2410.09038），并将其应用于[MedQA-USMLE](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options)数据集的临床鉴别诊断任务。 ## 🧬 任务挑战 大语言模型（LLM）在临床诊断任务中存在**模式坍缩（mode collapse）**问题：模型会反复预测同一排名第一的诊断结果，忽略完整的鉴别诊断列表。在医学场景中，该问题极具危险性，因为临床实践中必须考虑罕见但危重的病症。标准的温度采样（temperature sampling）无法解决该问题，因为其仅会引入随机噪声，而非系统性的答案多样性。 ## 🌱 思维种子方法论 该流程包含三个阶段： ### 阶段1：自动分层（Auto-Stratification） 大语言模型会识别出**具有区分性的临床特征（discriminating clinical properties）**，将答案空间大致划分为两个相等的部分。对于4选项的多项选择题，理想的特征划分比例应为2:2。示例：「该致病机制是否涉及直接细胞损伤？」→ 选项A、C符合该条件，选项B、D则不符合。 ### 阶段2：启发式估计（Heuristic Estimation） 针对每个特征，大语言模型会采用校准后的贝叶斯推理进行概率估计（灵感源自Tetlock提出的超级预测技术）。 ### 阶段3：概率提示（Probabilistic Prompting） 从联合分布中**采样**得到一个层级（即各特征取真/假值的组合），大语言模型会被**约束**在该层级内生成答案。不同的采样会探索答案空间的不同区域。 ## 📊 实验结果 | 评价指标 | 基线模型（温度系数=0.9） | 思维种子方法 | 性能变化Δ | |--------|---------------------|-----------------|---| | 准确率 | 0.9333 | 0.9333 | +0.0000 | | 多样性比率 | 0.1300 | 0.2533 | +0.1233 | | 唯一答案数 | 1.3000 | 2.5333 | +1.2333 | | 覆盖率 | 0.3417 | 0.5667 | +0.2250 | | 相对于均匀分布的KL散度 | 1.2773 | 0.8354 | -0.4419 | ### 实验设置 - **模型**：Qwen/Qwen2.5-72B-Instruct - **数据集**：MedQA-USMLE测试集（共30道题目） - **单问题采样数**：10 - **层级数**：3 - **基线温度系数**：0.9 | **思维种子方法温度系数**：0.7 - **总调用次数**：840 | **总耗时**：202.8分钟 ## 🔑 核心见解 通过将诊断空间划分为正交维度并进行系统性采样，思维种子方法实现了以下优势： 1. **更高的覆盖率**：可考虑更多合理的诊断结果 2. **更优的多样性**：不同采样生成的答案存在实质性差异 3. **更低的KL散度**：在有效选项上的分布更接近均匀分布 4. **准确率保持不变**：正确答案仍会出现在生成的答案集合中 ## 引用 bibtex @article{wong2024simplestrat, title={SimpleStrat: Diversifying Language Model Generation with Stratification}, author={Wong, Justin and Orlovskiy, Yury and Luo, Michael and Seshia, Sanjit A. and Gonzalez, Joseph E.}, journal={arXiv preprint arXiv:2410.09038}, year={2024} }