Supplementary Material for the paper: Automatic Document Screening of Medical Literature Using Word and Text Embeddings in an Active Learning Setting

This is the dataset used in the paper: Automatic Document Screening of Medical Literature Using Word and Text Embeddings in an Active Learning Setting. It is composed of: - Pre-trained models using active learning for document screening on HealthCLEF and Epistemonikos datasets. - Epistemonikos and HealthCLEF datasets containing medical questions and relevant/non relevant articles. - Embeddings and Document Representations used for experiments on both datasets. Scripts to run experiments can be found at: https://github.com/afcarvallo/active_learning_document_screening Paper abstract: Document screening is a fundamental task within Evidence-based Medicine (EBM), a practice that provides scientific evidence to support medical decisions. Several approaches have tried to reduce physicians' workload of screening and labeling vast amounts of documents to answer clinical questions. Previous works tried to semi-automate document screening, reporting promising results, but their evaluation was conducted on small datasets, which hinders generalization. Moreover, recent works in natural language processing have introduced neural language models, but none have compared their performance in EBM. In this paper, we evaluate the impact of several document representations such as TF-IDF along with neural language models (BioBERT, BERT, Word2vec, and GloVe) on an active learning-based setting for document screening in EBM. Our goal is to reduce the number of documents that physicians need to label to answer clinical questions. We evaluate these methods using both a small challenging dataset (HealthCLEF 2017) as well as a larger one but easier to rank (Epistemonikos). Our results indicate that word as well as textual neural embeddings always outperform the traditional TF-IDF representation. When comparing among neural and textual embeddings, in the HealthCLEF dataset the models BERT and BioBERT yielded the best results. On the larger dataset, Epistemonikos, Word2Vec and BERT were the most competitive, showing that BERT was the most consistent model across different corpuses. In term of active learning, an uncertainty sampling strategy combined with logistic regression achieved the best performance overall, above other methods under evaluation, and in fewer iterations.