Natural language processing systems for pathology parsing in limited data environments with uncertainty estimation

Objective: Cancer is a leading cause of death, but much of the diagnostic information is stored as unstructured data in pathology reports. We aim to improve uncertainty estimates of machine-learning based pathology parsers and evaluate performance in low data settings.

Materials and Methods: Our data comes from the Urologic Outcomes Database at UCSF which includes 3,232 annotated prostate cancer pathology reports from 2001-2018. We approach 17 separate information extraction tasks, involving a wide range of pathologic features. To handle the diverse range of fields we required two statistical models, a document classification method for pathologic features with a small set of possible values and a token extraction method for pathologic features with a large set of values. For each model, we used isotonic calibration to improve the model’s estimates of its likelihood of being correct.

Results: Our best document classifier method, a convolutional neural network, achieves a weighted ...