RadCases evaluation results: Evaluating acute image ordering for real-world patient cases via language model alignment with radiological guidelines

Background: Diagnostic imaging studies are increasingly important in the management of acutely presenting patients. However, ordering appropriate imaging studies in the emergency department is a challenging task with a high degree of variability between healthcare providers. To address this issue, recent work has investigated whether generative AI and large language models can be leveraged to recommend diagnostic imaging studies in accordance with evidence-based medical guidelines. However, it remains challenging to ensure that these tools can provide recommendations that correctly align with medical guidelines, especially given the limited diagnostic information available in acute care settings.
Methods: In this study, we introduce a framework to intelligently leverage language models by recommending imaging studies for patient cases that are aligned with the American College of Radiology’s Appropriateness Criteria, a set of evidence-based guidelines. To power our experiments, we make ..., This dataset was collected using custom code made publicly available at https://github.com/michael-s-yao/radGPT. No additional post-processing steps were performed., , # RadCases evaluation results: Evaluating acute image ordering for real-world patient cases via language model alignment with radiological guidelines

Dataset DOI: [10.5061/dryad.p8cz8wb0b](10.5061/dryad.p8cz8wb0b)

## Description of the Data and File Structure

The ZIP dataset contains all of the raw experimental results for our main experiments. Each individual file in the dataset is a [JSON Lines](https://jsonlines.org/) that contains the prediction made by the language model as well as the ground truth answer. Each line in a JSON Lines file represents one patient case evaluation. One JSON Lines file represents one experimental evaluation using one partition of the RadCases dataset, one language model, and one random seed. All raw outputs were generated using our custom source code.### Files and variables

#### File: radGPT-LLM-Evaluation.zip

The unzipped dataset is organized as follows (not all sub-directories are shown for brevity):

```
radGPT-LLM-Evaluation/
├── Baseline/
│ ├─...,