CORAL: expert-Curated medical Oncology Reports to Advance Language model inference

Both medical care and observational studies in oncology require a thorough understanding of a patient's disease progression and treatment history, often elaborately documented within clinical notes. As large language models (LLMs) are becoming more popular, it becomes important to evaluate their potential in oncology. However, no current information representation schema fully encapsulates the diversity of oncology information within clinical notes, and no comprehensively annotated oncology notes exist publicly, thereby limiting a thorough evaluation. We curated a new fine-grained, expert-labeled dataset of 40 de-identified breast and pancreatic cancer progress notes at University of California, San Francisco, and assessed three recent LLMs (GPT-4, GPT-3.5-turbo, and FLAN-UL2) in _zero-shot_ extraction of detailed oncological information from two narrative sections of clinical progress notes. Model performance was quantified with BLEU-4, ROUGE-1, and exact match (EM) F1-score evaluation metrics. Our team of oncology fellows and medical students manually annotated 9028 entities, 9986 modifiers, and 5312 relationships. The GPT-4 model exhibited overall best performance, with an average BLEU score of 0.73, an average ROUGE score of 0.72, an average EM-F1-score of 0.51, and an average accuracy of 68% (expert manual evaluation on 20 notes). GPT-4 was proficient in tumor characteristics and medication extraction, and demonstrated superior performance in inferring symptoms due to cancer and considerations of future medications. Common errors included partial responses with missing information and hallucinations with note-specific information. LLMs are promising for performing reliable information extraction for clinical research, complex population management, and documenting quality patient care, but there is a need for further improvements.