ER-REASON: A Benchmark Dataset for LLM-Based Clinical Reasoning in the Emergency Room

The ER-Reason dataset is a benchmark designed to evaluate LLM-based clinical
reasoning and decision-making in the emergency room (ER), a high-stakes
setting where clinicians make rapid, consequential decisions across diverse
patient presentations and medical specialties under time pressure. This
longitudinal collection of de-identified clinical notes encompasses 3,437
patients admitted to the ER at a large academic medical center between March
1, 2022, and March 31, 2024. ER-Reason contains 25,174 notes spanning
discharge summaries, progress notes, history and physical exams, consults,
echocardiography reports, imaging notes, and ER provider documentation across
3,984 encounters. The benchmark includes evaluation tasks from key stages of
the ER workflow: triage intake, initial assessment, treatment selection,
disposition planning, and final diagnosis, each structured to reflect core
clinical reasoning processes such as differential diagnosis via rule-out
reasoning. We also collected 72 full physician-authored rationales explaining
reasoning processes that mimic the teaching process used in residency
training, and are typically absent from ER documentation. This retrospective
dataset captures unstructured, multi-encounter clinical notes reflecting the
real-world complexity of ER patient care.