Supplementary Material for: A Pipeline for the Automatic Identification of Randomized Controlled Oncology Trials and Assignment of Tumor Entities Using Natural Language Processing

Background: Most tools trying to automatically extract information from medical publications are domain agnostic and process publications from any field. However, only retrieving trials from dedicated fields could have advantages for further processing of the data.
Methods: We trained a small transformer model to classify trials into randomized controlled trials (RCTs) vs. non-RCTs and oncology publications vs. non-oncology publications. In addition, we used two large language models (GPT-4o and GPT-4o mini) for the same task. We assessed the performance of the three models and then developed a simple set of rules to extract the tumor entity from the retrieved oncology RCTs.
Results: On the unseen test set consisting of 100 publications, the small transformer achieved an F1-score of 0.96 (95% CI: 0.92 - 1.00) with a precision of 1.00 and a recall of 0.92 for predicting whether a publication was an RCT. For predicting whether a publication covered an oncology topic, the F1-score was 0.84 (0.77 - 0.91) with a precision of 0.75 and a recall of 0.95. GPT-4o achieved an F1-score of 0.94 (95% CI: 0.90 - 0.99) with a precision of 0.89 and a recall of 1.00 for predicting whether a publication was an RCT. For predicting whether a publication covered an oncology topic the F1-score was 0.91 (0.85 - 0.97) with a precision of 0.91 and a recall of 0.91.
The rule-based system was able to correctly assign every oncology RCT in the test set to a tumor entity.
Conclusion: In conclusion, classifying publications depending on whether they were randomized controlled oncology trials or not was feasible and enabled further processing using more specialized tools such as rule-based systems and potentially dedicated machine learning models.