Benchmarking LLM-based Information Extraction Tools for Medical Documents

Motivation: Medical documents are a crucial resource for medical research around the world. While troves of valuable health data exist, they are largely computationally inaccessible as hard copies of unstructured text, often with degraded quality due to the persistent prevalence of fax machines in medical settings. Digitization of these resources through manual data extraction is time-consuming and resource intensive. However, large language models (LLMs) have recently shown great promise for automated digitization and information extraction (IE), greatly improving upon previous tools in terms of speed and accuracy.Results: We reviewed recent LLM-based tools for named entity recognition (NER) and IE from the literature and assessed them with respect to their suitability for use in a clinical setting. We found only two of these tools to be usable out of the box and compared them to LLM foundation models prompted to perform extractions. Using 1000 mock medical documents with paired reference data, we evaluated the tools’ performance in different scenarios, comparing zero-shot and one-shot prompts as well as unimodal and multimodal (image and text) inputs where possible. The most effective model was OpenAI’s GPT 4.1-mini with an average F1 score of 55.6. The best performing local model was Google's Gemma3 with 27B parameters, given image inputs and a zero-shot prompt, with an average F1 score of 41.3. We found the choice of prompting strategy to have minimal impact on extraction performances. We also assessed the effects of image distortions commonly introduced by fax machines and found a significant impact on extraction performance.Availability: Source code and data are available on Github at https://github.com/courtotlab/PDF_benchmarking.