MIMIC-Eye: Integrating MIMIC Datasets with REFLACX and Eye Gaze for Multimodal Deep Learning Applications

Deep learning technologies have been widely adopted in medical imaging due to
their ability to extract features from images and make accurate diagnoses
automatically. Medical imaging technologies are particularly useful because
they can be trained to detect subtle differences in images that are hard to
detect for human radiologists. In the real world, radiologists must rely on
various types of patient information to assess medical images confidently.
However, most DL applications in medical imaging only utilize image data,
mainly because the literature on medical datasets combining different data
modalities is scarce. In this study, we present MIMIC-EYE, a dataset that
encompasses a comprehensive integration of several datasets related to MIMIC.
This dataset includes a comprehensive range of patient information, including
medical images and reports (MIMIC CXR and MIMIC JPG), clinical data (MIMIC IV
ED), a detailed account of the patient's hospital journey (MIMIC IV), and eye
tracking data containing gaze information and pupil dilations together with
image annotations (REFLACX and EYE GAZE). Integrating eye tracking data with
the various MIMIC modalities may provide a more comprehensive understanding of
radiologists' visual search behavior patterns and facilitate the development
of more robust, accurate, and reproducible deep-learning models for medical
imaging diagnosis.