MIMIC-III-Ext-VeriFact-BHC: Labeled Propositions From Brief Hospital Course Summaries for Long-form Clinical Text Evaluation

The _VeriFact-BHC_ dataset is designed to verify the factuality of long-form
text written about a patient against their own electronic health record. There
is increasing interest in using large language models (LLMs) to generate
clinical text in patient care applications, yet this text needs to be
evaluated for factual errors and hallucinations prior to committing text to a
patient 's permanent medical record. Text written about a patient should be
internally consistent with information already known about the patient, such
as that stored in their medical records.

_VeriFact-BHC_ contains long-form Brief Hospital Course (BHC) clinical
narratives typically found in a discharge summary that have been decomposed
into text proposition statements. From 100 patients in the MIMIC-III Clinical
Database v1.4, we consider two types of BHC text: a human-written BHC and a
LLM-generated BHC. The original human clinician-written BHC is extracted from
the discharge summary note. The LLM-generated BHC is composed by a LLM using
the patient 's longitudinal clinical notes from the hospital admission. Each
BHC is decomposed in two ways: sentence propositions and atomic claim
propositions. The remaining electronic health record (EHR) notes for each
patient serves as a patient-specific reference of facts that is used by
clinicians and _VeriFact_ to assign labels. A total of 13,070 propositions are
annotated by multiple clinicians with a ground truth established via majority
voting and manual adjudication. Also provided are labels assigned by the
_VeriFact_ artificial intelligence system and labels assessing whether
propositions are valid from a first-order logic standpoint. The reference EHR
for each patient is provided in both machine-readable and PDF formats.

By offering this dataset, we hope to spur further investigation and creation
of computational systems for automatic chart review and patient-specific fact
verification. We invite the research community to utilize this dataset to
develop better methods to guardrail patient-specific LLM-generated clinical
text.