MIMIC-Ext-MIMIC-CXR-VQA: A Complex, Diverse, And Large-Scale Visual Question Answering Dataset for Chest X-ray Images

We introduce MIMIC-Ext- _MIMIC-CXR-VQA_ (i.e., Extended from MIMIC database),
a complex, diverse, and large-scale dataset designed for Visual Question
Answering (VQA) tasks within the medical domain, focusing primarily on chest
radiographs. This dataset includes approximately 377K entries derived from the
MIMIC-CXR-JPG, MIMIC-IV, and Chest ImaGenome datasets, all sourced from
Physionet. It features questions generated from 48 unique templates across
seven content types: presence, anatomy, attribute, abnormality, size, plane,
and gender. Each template, developed under the guidance of a board-certified
medical expert to ensure clinical relevance, addresses both standard content
from previous medical VQA tasks and more complex scenarios involving set and
logical operations. To further enhance linguistic diversity while maintaining
a medical context, we implemented a paraphrasing strategy with an average of
16.5 paraphrases per template, developed through carefully designed prompts
based on GPT-4.

The primary aim of MIMIC-Ext- _MIMIC-CXR-VQA_ is to serve as a comprehensive
benchmark for evaluating medical VQA methodologies. However, the significance
of this dataset extends far beyond just medical VQA benchmarking. It not only
provides a foundational tool for developing and testing VQA methods but also
acts as a valuable resource for instruction tuning of medical Vision-and-
Language Models (VLMs), addressing the scarcity of medical instruction
datasets. Furthermore, the integration of structured EHRs (i.e., MIMIC-IV)
with our dataset, MIMIC-Ext- _MIMIC-CXR-VQA_ , opens new avenues for the
development of multi-modal AI frameworks that leverage both imaging and
tabular modalities of patient records. By making this dataset publicly
accessible, we aim to improve the understanding of medical images and
stimulate further innovation within the realm of medical AI.