MIMIC-Ext-MIMIC-CXR-VQA: A Complex, Diverse, And Large-Scale Visual Question Answering Dataset for Chest X-ray Images

We introduce MIMIC-Ext- _MIMIC-CXR-VQA_ (i.e., Extended from MIMIC database), a complex, diverse, and large-scale dataset designed for Visual Question Answering (VQA) tasks within the medical domain, focusing primarily on chest radiographs. This dataset includes approximately 377K entries derived from the MIMIC-CXR-JPG, MIMIC-IV, and Chest ImaGenome datasets, all sourced from Physionet. It features questions generated from 48 unique templates across seven content types: presence, anatomy, attribute, abnormality, size, plane, and gender. Each template, developed under the guidance of a board-certified medical expert to ensure clinical relevance, addresses both standard content from previous medical VQA tasks and more complex scenarios involving set and logical operations. To further enhance linguistic diversity while maintaining a medical context, we implemented a paraphrasing strategy with an average of 16.5 paraphrases per template, developed through carefully designed prompts based on GPT-4. The primary aim of MIMIC-Ext- _MIMIC-CXR-VQA_ is to serve as a comprehensive benchmark for evaluating medical VQA methodologies. However, the significance of this dataset extends far beyond just medical VQA benchmarking. It not only provides a foundational tool for developing and testing VQA methods but also acts as a valuable resource for instruction tuning of medical Vision-and- Language Models (VLMs), addressing the scarcity of medical instruction datasets. Furthermore, the integration of structured EHRs (i.e., MIMIC-IV) with our dataset, MIMIC-Ext- _MIMIC-CXR-VQA_ , opens new avenues for the development of multi-modal AI frameworks that leverage both imaging and tabular modalities of patient records. By making this dataset publicly accessible, we aim to improve the understanding of medical images and stimulate further innovation within the realm of medical AI.