ReFiSco: Report Fix and Score Dataset for Radiology Report Generation

Automated generation of clinically accurate radiology reports can improve
patient care. In order to improve automatic report generation, it is helpful
to understand what types of errors are common in generated reports. Thus, we
introduce the Report Fix and Score Dataset for Radiology Reports (ReFiSco-v0),
which was collected through an institutional review board-approved study. In
our study, we recruit radiologists to provide expert evaluations on a subset
of 60 studies from MIMIC-CXR. For each radiology image, we compile three
reports: one generated from the model X-REM, one from the model CXR-RePaiR
trained on the same MIMIC-CXR training set, and one taken from a human
benchmark (MIMIC-CXR). To each radiologist, we present one image and one
report for each of the 60 studies. Each report is randomly and independently
chosen from one of the three sources. The radiologist is blinded to the
source. We ask each radiologist to assess the error severity of their assigned
reports.