Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretation

Radiology reports convey detailed clinical observations and capture diagnostic
reasoning that evolves over time. However, existing evaluation methods are
limited to single-report settings and rely on coarse metrics that fail to
capture fine-grained clinical semantics and temporal dependencies. We
introduce **LUNGUAGE** , a benchmark dataset of structured radiology reports
that serves as a gold standard for evaluating structured report frameworks. It
is designed to support comprehensive assessment of both single-report
interpretation and longitudinal reasoning. Constructed from a subset of the
MIMIC-CXR test set, LUNGUAGE comprises 1,473 chest X-ray reports from 230
patients, annotated with over 17,000 expert-verified entities and 23,000
relation-attribute pairs across 18 relation types. An additional subset of 80
sequential reports from 10 patients captures disease progression across 3 to
14 studies per patient, covering time intervals from 1 to 1,200 days. These
are annotated with over 41,000 pairwise comparisons, grouped into semantically
and temporally coherent groups. The dataset also includes a schema-aligned
vocabulary covering diagnostic entities and attributes. All annotations were
conducted and verified by board-certified radiologists, resulting in a
clinically grounded resource for structured understanding and temporal
reasoning in radiology.