LLM performance on ASTRO practice exam

The 2023 ASTRO Radiation and Cancer Biology Exam Study Guide was used to evaluate the accuracy of GPT-3.5, GPT-4, and Llama-2. The multiple-choice questions were obtained from the official ASTRO website and are free and publicly available [18]. Questions containing graphical or image-based content were excluded (2/337 questions excluded).The exam questions were classified into the ten foundational topics listed on ASTRO’s study guide, including: 1) Interaction of radiation with matter, 2) Molecular and cellular damage and repair, 3) Cellular response to radiation, 4) Linear energy transfer (LET) and oxygen effect, 5) Tumor biology and microenvironment, 6) Cancer biology, 7) Radiobiology of normal tissues, 8) Dose delivery, 9) Combined modality therapy, and 10) Late effects and radiation protection. Questions were also categorized based on whether they required the use of math.