Supplementary Material for the paper: Automatic Document Screening of Medical Literature Using Word and Text Embeddings in an Active Learning Setting

This is the dataset used in the paper: Automatic Document Screening of Medical Literature Using Word and Text Embeddings in an Active Learning Setting. It is composed of: - Pre-trained models using active learning for document screening on HealthCLEF and Epistemonikos datasets. - Epistemonikos and HealthCLEF datasets containing medical questions and relevant/non relevant articles. - Embeddings and Document Representations used for experiments on both datasets. Scripts to run experiments can be found at: <strong>https://github.com/afcarvallo/automatic_document_screening</strong> <strong>Paper abstract:</strong> Document screening is a fundamental task within Evidence-based Medicine (EBM), a practice that provides scientific evidence to support medical decisions. Several approaches have tried to reduce physicians' workload of screening and labeling vast amounts of documents to answer clinical questions. Previous works tried to semi-automate document screening, reporting promising results, but their evaluation was conducted on small datasets, which hinders generalization. Moreover, recent works in natural language processing have introduced neural language models, but none have compared their performance in EBM. In this paper, we evaluate the impact of several document representations such as TF-IDF along with neural language models (BioBERT, BERT, Word2vec, and GloVe) on an active learning-based setting for document screening in EBM. Our goal is to reduce the number of documents that physicians need to label to answer clinical questions. We evaluate these methods using both a small challenging dataset (HealthCLEF 2017) as well as a larger one but easier to rank (Epistemonikos). Our results indicate that word as well as textual neural embeddings always outperform the traditional TF-IDF representation. When comparing among neural and textual embeddings, in the HealthCLEF dataset the models BERT and BioBERT yielded the best results. On the larger dataset, Epistemonikos, Glove and BioBERT were the most competitive, showing that BioBERT was the most consistent model across different corpuses. In term of active learning, an uncertainty sampling strategy combined with logistic regression achieved the best performance overall, above other methods under evaluation, and in fewer iterations.本数据集源自论文《主动学习场景下结合词与文本嵌入的医学文献自动筛选》（原标题：Automatic Document Screening of Medical Literature Using Word and Text Embeddings in an Active Learning Setting）。该数据集包含以下内容： - 基于主动学习（Active Learning），针对HealthCLEF与Epistemonikos数据集开展文献筛选任务所训练的预训练模型； - Epistemonikos与HealthCLEF数据集，其中涵盖医学问题以及相关/非相关文献； - 用于两类数据集实验的嵌入向量与文档表征。 可用于运行实验的脚本可于以下链接获取：<strong>https://github.com/afcarvallo/automatic_document_screening</strong> <strong>论文摘要：</strong> 文献筛选是循证医学（Evidence-based Medicine, EBM）的核心任务，该学科通过提供科学证据以辅助临床决策。当前已有多种方法致力于减轻医师筛选、标注海量文献以解答临床问题的工作负担。既往研究尝试实现文献筛选的半自动化并取得了可观成果，但相关实验均基于小型数据集开展，这限制了方法的泛化能力。此外，近年来自然语言处理领域涌现出诸多神经语言模型，但尚无研究对其在循证医学场景下的表现开展对比评估。 本文针对循证医学中的文献筛选任务，在主动学习框架下评估了多种文档表征方法的效果，包括TF-IDF以及BioBERT、BERT、Word2vec、GloVe等神经语言模型。本研究旨在减少医师为解答临床问题所需标注的文献数量。我们分别采用小型挑战性数据集HealthCLEF 2017，以及规模更大但更易于排序的Epistemonikos数据集对上述方法进行评估。 实验结果表明，词嵌入与文本神经嵌入的表现始终优于传统TF-IDF表征。在神经文本嵌入方法的对比中，HealthCLEF数据集上BERT与BioBERT的效果最优；而在更大规模的Epistemonikos数据集上，GloVe与BioBERT表现最为亮眼，其中BioBERT在不同数据集上均展现出最佳的稳定性。在主动学习策略方面，结合逻辑回归的不确定性采样策略整体性能最优，优于其余参评方法，且所需迭代次数更少。