Supplementary Material for the paper: Automatic Document Screening of Medical Literature Using Word and Text Embeddings in an Active Learning Setting

This is the dataset used in the paper: Automatic Document Screening of Medical Literature Using Word and Text Embeddings in an Active Learning Setting. It is composed of: - Pre-trained models using active learning for document screening on HealthCLEF and Epistemonikos datasets. - Epistemonikos and HealthCLEF datasets containing medical questions and relevant/non relevant articles. - Embeddings and Document Representations used for experiments on both datasets. Scripts to run experiments can be found at: https://github.com/afcarvallo/active_learning_document_screening <strong>Paper abstract:</strong> Document screening is a fundamental task within Evidence-based Medicine (EBM), a practice that provides scientific evidence to support medical decisions. Several approaches have tried to reduce physicians' workload of screening and labeling vast amounts of documents to answer clinical questions. Previous works tried to semi-automate document screening, reporting promising results, but their evaluation was conducted on small datasets, which hinders generalization. Moreover, recent works in natural language processing have introduced neural language models, but none have compared their performance in EBM. In this paper, we evaluate the impact of several document representations such as TF-IDF along with neural language models (BioBERT, BERT, Word2vec, and GloVe) on an active learning-based setting for document screening in EBM. Our goal is to reduce the number of documents that physicians need to label to answer clinical questions. We evaluate these methods using both a small challenging dataset (HealthCLEF 2017) as well as a larger one but easier to rank (Epistemonikos). Our results indicate that word as well as textual neural embeddings always outperform the traditional TF-IDF representation. When comparing among neural and textual embeddings, in the HealthCLEF dataset the models BERT and BioBERT yielded the best results. On the larger dataset, Epistemonikos, Word2Vec and BERT were the most competitive, showing that BERT was the most consistent model across different corpuses. In term of active learning, an uncertainty sampling strategy combined with logistic regression achieved the best performance overall, above other methods under evaluation, and in fewer iterations.本数据集源自论文《基于主动学习场景下词与文本嵌入的医学文献自动筛选》（Automatic Document Screening of Medical Literature Using Word and Text Embeddings in an Active Learning Setting）。该数据集包含以下组成部分： - 针对HealthCLEF与Epistemonikos数据集的文献筛选任务，基于主动学习方法训练得到的预训练模型； - 收录医学问题与相关、非相关文献的Epistemonikos与HealthCLEF数据集； - 用于上述两类数据集实验的嵌入向量与文档表征。 实验运行脚本可访问：https://github.com/afcarvallo/active_learning_document_screening **论文摘要：** 文献筛选是循证医学（Evidence-based Medicine, EBM）中的核心任务，该学科通过提供科学证据支撑临床诊疗决策。既往诸多研究致力于减轻医师为解答临床问题而筛选、标注海量文献的工作负担。早期研究尝试实现文献筛选的半自动化并取得了可观成果，但相关评估仅在小数据集上开展，限制了方法的泛化能力。此外，近年来自然语言处理领域涌现出诸多神经语言模型，但尚无研究针对循证医学场景下的模型性能展开对比。 本文针对循证医学文献筛选的主动学习场景，评估了TF-IDF等多种文档表征方法，以及BioBERT、BERT、Word2vec、GloVe等神经语言模型的应用效果，旨在减少医师为解答临床问题所需标注的文献数量。我们分别在两类数据集上开展实验：规模较小但颇具挑战性的HealthCLEF 2017数据集，以及规模更大但排序难度更低的Epistemonikos数据集。实验结果表明，词嵌入与文本神经嵌入的表现始终优于传统TF-IDF表征。在神经与文本嵌入模型的对比中，HealthCLEF数据集上BERT与BioBERT取得了最优效果；而在更大规模的Epistemonikos数据集上，Word2Vec与BERT表现最具竞争力，可见BERT在不同语料库上均展现出最佳的一致性。在主动学习策略层面，结合逻辑回归的不确定性采样策略整体表现最优，优于其余参评方法，且所需迭代次数更少。