EHRNoteQA: A Patient-Specific Question Answering Benchmark for Evaluating Large Language Models in Clinical Settings

We introduce EHRNoteQA, a patient-specific question answering benchmark tailored for evaluating Large Language Models (LLMs) in clinical environments. Based on MIMIC-IV Electronic Health Record (EHR), a team of three medical professionals has curated the dataset comprising 962 unique questions, each linked to a specific patient's EHR clinical notes. Our comprehensive evaluation on various large language models showed that their scores on EHRNoteQA correlate more closely with their performance in addressing real-world medical questions evaluated by clinicians than their scores from other LLM benchmarks. This emphasizes the importance of EHRNoteQA in assessing Large Language Models (LLMs) for medical purposes and underscores its contribution to incorporating LLMs into healthcare infrastructures.

本团队推出EHRNoteQA，一项针对评估临床环境中大语言模型（LLM）的个性化患者问答基准。该基准基于MIMIC-IV电子健康记录（EHR）构建，由三位医疗专业人员精心编纂，包含962个独特的提问，每个问题均与特定患者的EHR临床笔记相连接。我们对多种大型语言模型进行了全面评估，结果显示，模型在EHRNoteQA上的得分与其在临床医师评估的现实世界医学问题解答中的表现密切相关，远超其他LLM基准的得分。此发现凸显了EHRNoteQA在评估大语言模型（LLM）用于医学目的中的重要性，并强调了其在将LLM融入医疗基础设施中的贡献。