Chest ImaGenome Dataset

In recent years, with the release of multiple large datasets, automatic
interpretation of chest X-ray (CXR) images with deep learning models have
become feasible for specific abnormalities or for generating preliminary
reports. However, despite reports of performance reaching similar levels to
that of radiologists, a quantitative evaluation of the explainability of these
models is hampered by the lack of locally labeled datasets for different
findings. With the exception of a few human-labeled small-scale datasets for
specific findings, such as pneumonia and pneumothorax, most of the CXR deep
learning models to date are trained on global "weak" labels extracted from
text reports, or trained via a joint image and unstructured text learning
strategy. In our work, a joint rule-based natural language processing (NLP)
and CXR atlas-based bounding box detection pipeline are used to automatically
label 242072 frontal MIMIC CXRs locally. Inspired by the Visual Genome effort
in the computer vision community [20], we constructed the first Chest
ImaGenome dataset with a scene graph data structure to describe the data.
Through a radiologist constructed CXR ontology, the annotations for each CXR
are connected as an anatomy-centered scene graph, useful for image-level
reasoning and multimodal fusion applications. Overall, our dataset contributes
significantly to the research community by providing 1) 1,256 combinations of
relation annotations between 29 CXR anatomical locations (objects with
bounding box coordinates) and their attributes, structured as a scene graph
per image, 2) over 670,000 localized comparison relations (for improved,
worsened, or no change) between the anatomical locations across sequential
exams, as well as 3) a manually annotated gold standard scene graph dataset
from 500 unique patients.