ReFiSco: Report Fix and Score Dataset for Radiology Report Generation

Automated generation of clinically accurate radiology reports can improve patient care. In order to improve automatic report generation, it is helpful to understand what types of errors are common in generated reports. Thus, we introduce the Report Fix and Score Dataset for Radiology Reports (ReFiSco-v0), which was collected through an institutional review board-approved study. In our study, we recruit radiologists to provide expert evaluations on a subset of 60 studies from MIMIC-CXR. For each radiology image, we compile three reports: one generated from the model X-REM, one from the model CXR-RePaiR trained on the same MIMIC-CXR training set, and one taken from a human benchmark (MIMIC-CXR). To each radiologist, we present one image and one report for each of the 60 studies. Each report is randomly and independently chosen from one of the three sources. The radiologist is blinded to the source. We ask each radiologist to assess the error severity of their assigned reports.