EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images

Electronic Health Records (EHRs), which contain patients' medical histories in various multi-modal formats, often overlook the potential for joint reasoning across imaging and table modalities underexplored in current EHR Question Answering (QA) systems. In this paper, we introduce EHRXQA, a novel multi-modal question answering dataset combining structured EHRs and chest X-ray images. To develop our dataset, we first construct two uni-modal resources: 1) The MIMIC-CXR-VQA dataset, our newly created medical visual question answering (VQA) benchmark, specifically designed to augment the imaging modality in EHR QA, and 2) EHRSQL (MIMIC-IV), a refashioned version of a previously established table-based EHR QA dataset. By integrating these two uni-modal resources, we successfully construct a multi-modal EHR QA dataset that necessitates both uni-modal and cross-modal reasoning. To address the unique challenges of multi-modal questions within EHRs, we propose a NeuralSQL-based strategy equipped with an external VQA API. This pioneering endeavor enhances engagement with multi-modal EHR sources and we believe that our dataset can catalyze advances in real-world medical scenarios such as clinical decision-making and research.

电子健康记录（EHRs），其中包含患者多样化的医疗历史，以多种多模态格式呈现，常被忽视在当前电子健康记录问答（QA）系统中对成像和表格模态的联合推理潜力。在本文中，我们提出了EHRXQA，这是一个创新的多模态问答数据集，它结合了结构化EHR和胸部X射线图像。为了构建我们的数据集，我们首先构建了两个单模态资源：1）MIMIC-CXR-VQA数据集，这是我们新创建的医疗视觉问答（VQA）基准，专门设计用于增强EHR QA中的成像模态，2）EHRSQL（MIMIC-IV），这是之前建立的一个基于表格的EHR QA数据集的改进版本。通过整合这两个单模态资源，我们成功地构建了一个需要单模态和跨模态推理的多模态EHR QA数据集。为了解决EHR中多模态问题的独特挑战，我们提出了一种基于NeuralSQL的策略，并配备外部VQA API。这一开创性努力增强了与多模态EHR资源的互动，我们相信我们的数据集可以催化现实世界医疗场景如临床决策和研究等方面的进步。