LATTE-CXR: Locally Aligned TexT and imagE, Explainable dataset for Chest X-Rays

Local annotation of medical data is both expensive and time-consuming due to the high cost of expert annotators, the precision required for accurate annotation, and the inherent challenges of medical diagnosis. To address these problems, we developed LATTE-CXR, a chest X-ray dataset with locally aligned image-text pairs, derived from the REFLACX dataset. LATTE-CXR supports tasks requiring local image-text annotations, such as phrase grounding, caption- guided object detection, and image captioning with region-level descriptions. By extracting statements from radiology reports corresponding to REFLACX annotated abnormalities, this dataset includes 3926 bounding box-statement pairs (with repeating statements) from 1668 MIMIC-CXR image readings in the REFLACX dataset. Additionally, we automatically generated 13751 bounding box- sentence pairs from 2,742 chest X-ray readings, utilizing timestamped eye- tracking data and transcribed reports from REFLACX. The eye-tracking bounding boxes are linked to corresponding annotated bounding boxes if they share a sentence, providing a comprehensive framework for assessing model explainability.