EHRNoteQA: A Patient-Specific Question Answering Benchmark for Evaluating Large Language Models in Clinical Settings

We introduce EHRNoteQA, a patient-specific question answering benchmark
tailored for evaluating Large Language Models (LLMs) in clinical environments.
Based on MIMIC-IV Electronic Health Record (EHR), a team of three medical
professionals has curated the dataset comprising 962 unique questions, each
linked to a specific patient's EHR clinical notes. Our comprehensive
evaluation on various large language models showed that their scores on
EHRNoteQA correlate more closely with their performance in addressing real-
world medical questions evaluated by clinicians than their scores from other
LLM benchmarks. This emphasizes the importance of EHRNoteQA in assessing Large
Language Models (LLMs) for medical purposes and underscores its contribution
to incorporating LLMs into healthcare infrastructures.