MedSP1000
收藏资源简介：
MedSP1000是一个基于标准化病人（Standardized Patient, SP）衍生的交互式基准数据集，旨在评估大语言模型作为临床代理的能力。与静态、单轮医疗问答数据集不同，该数据集中的每个项目都是一个可执行的多轮交互式临床诊疗场景。在这种场景中，临床医生代理与病人代理以及环境控制器在闭环中进行多轮交互，并遵循标准化的状态转换协议。临床医生代理在整个诊疗过程中的行为，会依据一个专家定义且经过人工验证的评分细则进行持续评估和打分。该基准数据集直接构建于经过同行评审的MedEdPORTAL标准化病人教学材料之上，通过一个智能数据处理流程，将异质的原始文章转化为特定角色（如临床医生、病人、环境、评估者）的场景数据包。数据集规模庞大，包含1,638个交互式案例，覆盖17个临床专业，并包含24,602个评分细则项。评分体系严格遵循美国毕业后医学教育认证委员会（ACGME）的六大核心能力：病人照护、医学知识、基于系统的实践、人际与沟通技巧、基于实践的学习与改进以及职业精神。每个临床医生的行动都会根据这六大能力对应的固定评分细则进行打分，最终得分是完成整个诊疗轨迹中专家定义的评分细则项的比例。所有案例和轨迹都经过临床医生的独立双重评分验证，确保了数据质量。数据集适用于文本生成、问答等任务，尤其专注于医疗、临床、健康护理、交互式评估和基准测试等应用场景。
MedSP1000 is an interactive benchmark dataset derived from Standardized Patients (SP), aimed at evaluating the capabilities of Large Language Models (LLMs) as clinical agents. Unlike static, single-turn medical question answering datasets, each entry in this dataset is an executable multi-turn interactive clinical consultation scenario. In such scenarios, the clinician agent, patient agent, and environment controller conduct closed-loop multi-round interactions in accordance with a standardized state transition protocol. The actions of the clinician agent throughout the entire consultation process are continuously assessed and scored against an expert-defined and manually validated scoring rubric. This benchmark dataset is directly constructed on peer-reviewed Standardized Patient teaching materials from MedEdPORTAL, and converts heterogeneous original articles into scenario data packages for specific roles (e.g., clinician, patient, environment, evaluator) through an intelligent data processing workflow. The dataset is large-scale, containing 1,638 interactive cases covering 17 clinical specialties, and includes 24,602 scoring rubric items. The scoring system strictly follows the six core competencies of the Accreditation Council for Graduate Medical Education (ACGME): Patient Care, Medical Knowledge, System-Based Practice, Interpersonal and Communication Skills, Practice-Based Learning and Improvement, and Professionalism. Each action taken by the clinician is scored against the fixed scoring rubrics corresponding to these six competencies, and the final score is the proportion of expert-defined scoring rubric items completed across the entire clinical consultation trajectory. All cases and trajectories have undergone independent double rating and verification by clinicians to ensure data quality. The dataset is applicable to tasks such as text generation and question answering, with a particular focus on application scenarios including medical, clinical, healthcare, interactive evaluation and benchmark testing.
MedSP1000 数据集概述
基本信息
- 许可证: MIT
- 语言: 英文
- 任务类别: 文本生成、问答
- 数据集规模: 1K < n < 10K
- 标签: 医疗、临床、医疗保健、标准化病人、智能体、交互式评估、基准测试、ACGME
数据集简介
MedSP1000 是一个基于标准化病人（Standardized Patient）构建的交互式基准测试，用于评估大语言模型作为临床智能体的动态临床决策能力。与静态的单轮医学问答不同，每个样本是一个可执行的多轮问诊场景：临床智能体与病人智能体、环境控制器在连续的患者状态下进行闭环交互，其行为根据专家定义并经过人工验证的评分标准进行评分。
数据来源
基准测试建立在经同行评审的 MedEdPORTAL 标准化病人教学材料之上，通过智能体数据处理流水线将异构源文章转换为角色特定的场景包。每个发布的病例都保留到原始 MedEdPORTAL 文章的可追溯映射。
主要特点
- 🏥 标准化病人基础：源自同行评审的 MedEdPORTAL 教学材料
- 🔁 交互式多轮：临床智能体、病人智能体和环境控制器之间的闭环问诊，具有标准化的状态转换协议
- 📊 规模与广度：覆盖 17 个临床专科，共 1,638 个交互式病例，包含 24,602 个评分项
- 🧭 ACGME 对齐评分：每个动作根据 6 项 ACGME 核心能力（患者护理、医学知识、基于系统的实践、人际沟通技能、基于实践的学习与改进、专业精神）的固定评分表进行评分
- 👩⚕️ 人工验证：病例和轨迹由临床医生检查并进行独立双重评分
数据集结构
每个场景包包含四个角色智能体（临床医生/病人/环境/评估者）的材料以及用于评分的固定 ACGME 评分表。可通过以下方式加载： python from datasets import load_dataset ds = load_dataset("byrLLCC/MedSP1000")
评分方式
评分遵循 ACGME 核心能力 框架。每个临床动作根据六项能力对应的固定评分表进行评分。一次运行的得分是完整轨迹中完成的专家定义评分项的比例。
引用
bibtex @article{liang2026medsp1000, title = {MedSP1000: Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases}, author = {Liang, Cheng and Qiu, Pengcheng and Zhang, Ya and Wang, Yanfeng and Wu, Chaoyi and Xie, Weidi}, journal = {<Venue>}, year = {2026}, url = {https://arxiv.org/abs/XXXX.XXXXX} }
