MS-CXR: Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing

We release a new dataset, MS-CXR, with locally-aligned phrase grounding
annotations by board-certified radiologists to facilitate the study of complex
semantic modelling in biomedical vision-language processing. The MS-CXR
dataset provides 1162 image-sentence pairs of bounding boxes and corresponding
phrases, collected across eight different cardiopulmonary radiological
findings, with an approximately equal number of pairs for each finding. This
dataset complements the existing MIMIC-CXR v.2 dataset and comprises: 1.
Reviewed and edited bounding boxes and phrases (1026 pairs of bounding
box/sentence); and 2. Manual bounding box labels from scratch (136 pairs of
bounding box/sentence).

This large, well-balanced phrase grounding benchmark dataset contains
carefully curated image regions annotated with descriptions of eight radiology
findings, as verified by radiologists. Unlike existing chest X-ray benchmarks,
this challenging phrase grounding task evaluates joint, local image-text
reasoning while requiring real-world language understanding, e.g. to parse
domain-specific location references, complex negations, and bias in reporting
style. This data accompany work showing that principled textual semantic
modelling can improve contrastive learning in self-supervised vision-language
processing.