MS-CXR-T: Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing

MS-CXR-T is a multi-modal benchmark dataset for evaluating biomedical vision- language processing (VLP) models on two distinct temporal tasks in radiology: image classification and sentence similarity. The former comprises multi-image frontal chest X-rays with ground-truth labels (N=1326) across 5 findings, with classes corresponding to 3 states of disease progression for each finding: {'Improving', 'Stable', 'Worsening'}, expanding on the Chest ImaGenome progression dataset. The latter quantifies the temporal-semantic similarity of text embeddings extracted from pairs of sentences (N=361). The pairs can be either paraphrases or contradictions in terms of disease progression. The data for both tasks was manually annotated and reviewed by a board-certified radiologist. The dataset provides researchers an opportunity to evaluate both image and text models on these biomedical temporal tasks and reproduce experiments reported in the corresponding literature.