MIMIC-Eye: Integrating MIMIC Datasets with REFLACX and Eye Gaze for Multimodal Deep Learning Applications

Deep learning technologies have been widely adopted in medical imaging due to their ability to extract features from images and make accurate diagnoses automatically. Medical imaging technologies are particularly useful because they can be trained to detect subtle differences in images that are hard to detect for human radiologists. In the real world, radiologists must rely on various types of patient information to assess medical images confidently. However, most DL applications in medical imaging only utilize image data, mainly because the literature on medical datasets combining different data modalities is scarce. In this study, we present MIMIC-EYE, a dataset that encompasses a comprehensive integration of several datasets related to MIMIC. This dataset includes a comprehensive range of patient information, including medical images and reports (MIMIC CXR and MIMIC JPG), clinical data (MIMIC IV ED), a detailed account of the patient's hospital journey (MIMIC IV), and eye tracking data containing gaze information and pupil dilations together with image annotations (REFLACX and EYE GAZE). Integrating eye tracking data with the various MIMIC modalities may provide a more comprehensive understanding of radiologists' visual search behavior patterns and facilitate the development of more robust, accurate, and reproducible deep-learning models for medical imaging diagnosis.

深度学习技术在医学影像领域的应用已得到广泛推广，得益于其从图像中提取特征并实现自动诊断的能力。医学影像技术尤为适用，因其可被训练以探测图像中难以被人类放射科医生察觉的细微差异。在现实世界中，放射科医生必须依赖多种患者信息来自信地评估医学影像。然而，大多数深度学习在医学影像中的应用仅利用图像数据，主要原因是关于结合不同数据模态的医学数据集的文献稀缺。在本研究中，我们提出了MIMIC-EYE数据集，该数据集综合了与MIMIC相关的多个数据集。该数据集包含了一系列全面的病人信息，包括医学图像和报告（MIMIC CXR和MIMIC JPG）、临床数据（MIMIC IV ED）、患者住院过程的详细记录（MIMIC IV），以及包含注视信息、瞳孔扩张和图像标注的眼动数据（REFLACX和EYE GAZE）。将眼动数据与MIMIC的多种模态相结合，可能有助于更全面地理解放射科医生的视觉搜索行为模式，并促进开发更为稳健、精确且可重复的医学影像诊断深度学习模型。

搜集汇总

背景与挑战

背景概述

MIMIC-EYE是一个多模态医学数据集，旨在整合多种MIMIC相关数据源，包括医学影像、报告、临床数据、患者住院记录以及眼动追踪数据。该数据集的特点在于首次结合了放射科医生的眼动信息（如注视和瞳孔扩张）与丰富的临床背景，以支持对视觉搜索行为的深入研究。通过提供这种综合数据，MIMIC-EYE有助于开发更准确、可靠的深度学习模型，用于医学影像诊断，弥补了当前多数应用仅依赖图像数据的不足。