MIMIC-IV-Ext Clinical Decision Making: A MIMIC-IV Derived Dataset for Evaluation of Large Language Models on the Task of Clinical Decision Making for Abdominal Pathologies

Clinical decision making is one of the most impactful parts of a physician's
responsibilities and stands to benefit greatly from AI solutions such as large
language models (LLMs). However, while many datasets exist to test the
performance of AI models on constructed case vignettes, such as medical
licensing exams, these tests fail to assess many skills that are necessary for
deployment in a realistic clinical decision making environment. To understand
how useful LLMs are in real-world settings, we must evaluate them _in the
wild,_ i.e. on real-world data under realistic conditions. To address this
need, we have created a curated dataset based on the MIMIC-IV database,
spanning 2400 real patient cases and four common abdominal pathologies:
appendicitis, cholecystitis, diverticulitis, and pancreatitis. Each patient
case contains the filtered and curated information necessary to arrive at the
delivered diagnosis of the physician and can be used in an interactive manner
to test the information gathering, synthesizing, and diagnostic capabilities
of AI models.