MIMIC-IV-Ext-Apixaban-Trial-Criteria-Questions

Large-language models (LLMs) show promise for extracting information from
clinical notes. Deploying these models at scale can be challenging due to high
computational costs, regulatory constraints, and privacy concerns. To address
these challenges, synthetic data distillation can be used to fine-tune
smaller, open-source LLMs that achieve performance similar to the teacher
model. These smaller models can be run on less expensive local hardware or at
a vastly reduced cost in cloud deployments. In our recent study, we used
Llama-3.1-70B-Instruct to generate synthetic training examples in the form of
question-answer pairs along with supporting information. We then used these
questions to fine-tune smaller versions of Llama to improve their ability to
extract clinical information from notes.

To evaluate the resulting models, we created 23 questions resembling
eligibility criteria from the apixaban clinical trial and evaluated them on a
random sample of 100 patient notes from MIMIC-IV. Notes from MIMIC-IV were
taken from after 2012 to ensure no overlap with any of the notes from MIMIC-
III which were used to generate the data used to finetune the models. We
release the 2300 total question-answer pairs as a dataset here.