EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries

Discharge summaries in Electronic Health Records (EHRs) are crucial for
clinical decision-making, but their length and complexity make information
extraction challenging, especially when dealing with accumulated summaries
across multiple patient admissions. Large Language Models (LLMs) show promise
in addressing this challenge by efficiently analyzing vast and complex data.
Existing benchmarks, however, fall short in properly evaluating LLMs'
capabilities in this context, as they typically focus on single-note
information or limited topics, failing to reflect the real-world inquiries
required by clinicians. To bridge this gap, we introduce EHRNoteQA, a novel
benchmark built on the MIMIC-IV EHR, comprising 962 different QA pairs each
linked to distinct patients' discharge summaries. Every QA pair is initially
generated using GPT-4 and then manually reviewed and refined by three
clinicians to ensure clinical relevance. EHRNoteQA includes questions that
require information across multiple discharge summaries and covers eight
diverse topics, mirroring the complexity and diversity of real clinical
inquiries.