Curated Biomedical Abstracts Dataset - Provenance-Verified, Deduplicated, Refined Labels (v1.0)

Title: Biomedical Abstracts Corpus — Provenance-Verified, Deduplicated, Refined Labels (v1.0) DOI: 10.5281/zenodo.17229456 Date created: 2025-09-29 Creators: Idris Babalola, Adewale Alex Adegoke, Peter Adebayo Odesola Contact: [e-mail] ====================================Dataset overviewThis deposit provides a curated biomedical abstracts corpus derived from a widely used Kaggle community dataset. The aim is a reusable, provenance-verified resource for text-classification tasks across major disease areas (e.g., cardiovascular, digestive system, neoplasms, nervous system). ==================================== Scope & counts Total records in source: 14,438 Duplicates/near-duplicates removed: 6,140 Additional filtering: removed “General Pathological Conditions” class and short abstracts (length-based threshold) Final unique records: 5,901 Final label set: Cardiovascular Diseases, Digestive System Diseases, Neoplasms, Nervous System Diseases ====================================Provenance verification (PubMed spot-check)A random sample of records was cross-referenced against PubMed. 99/100 sampled records returned PMIDs, supporting source traceability and dataset credibility. (See docs pubmed_spotcheck_results.csv included.) Note: Code is archived separately and will also be made available on GitHub soon. ==================================== Methods Deduplication: combined text-similarity and metadata to remove duplicates. Label design: normalized disease labels and retired the broad “General Pathological Conditions” class after six diagnostics checks(e.g i. lexical overlap, ii. unsupervised separability based on TF-IDF(1-2) class centroids, iii. intra-class cohesion, distinctiveness estimated via log-odds with an informative Dirichlet prior), to reduce semantic bleed-over that hindered class separability. Short-abstract filter: removed entries below a minimum length threshold to improve training quality. Provenance: PubMed ID recovery via title/metadata matching; PMIDs recorded where found. ====================================Intended use Biomedical NLP: supervised/unsupervised text classification, benchmarking, Reliability & Model uncertainty, LLM use case. Curation & reproducibility: exemplars for deduplication, provenance checks, and label-taxonomy repair. ====================================Licensing This derived dataset (metadata, labels, keys): CC BY-SA 3.0. Original dataset license: CC BY-SA 3.0 (see https://github.com/sebischair/Medical-Abstracts-TC-Corpus & https://huggingface.co/datasets/TimSchopf/medical_abstracts). Redistribution is permitted under ShareAlike terms; please attribute both the original creators and this curated release. Abstract text itself may be subject to third-party rights. Users should respect source terms when retrieving full text. ====================================Funding & competing interestsThis work received no funding. The authors declare no conflicts of interest. The curation and validation were conducted independently and are not endorsed by our current or past affiliations. ==================================== Acknowledgments & original sourceThis corpus is derived from the original community dataset created by https://www.kaggle.com/datasets/chaitanyakck/medical-text. We thank the original creators for making their resource available under CC BY-SA 3.0.Title: 生物医学摘要语料库——来源可验证、去重、标注优化版（v1.0） DOI: 10.5281/zenodo.17229456 Date created: 2025年9月29日 Creators: Idris Babalola, Adewale Alex Adegoke, Peter Adebayo Odesola Contact: [e-mail] ==================================== Dataset overview 本数据集存档基于一款广泛使用的Kaggle社区公开数据集整理优化而来，旨在为覆盖心血管系统、消化系统、肿瘤、神经系统等主要疾病领域的文本分类任务提供可复用、来源可追溯验证的标准化资源。 ==================================== Scope & counts Total records in source: 14,438 Duplicates/near-duplicates removed: 6,140 Additional filtering: 移除了“通用病理状态（General Pathological Conditions）”类别，并基于长度阈值过滤掉过短的摘要文本 Final unique records: 5,901 Final label set: 心血管疾病、消化系统疾病、肿瘤、神经系统疾病 ==================================== Provenance verification (PubMed spot-check) 我们对随机抽取的样本记录与PubMed数据库进行了交叉比对：100条抽样记录中有99条成功匹配到PubMed文献ID（PMID），验证了本数据集的来源可追溯性与可信度。（相关结果详见附件pubmed_spotcheck_results.csv文档） Note: Code is archived separately and will also be made available on GitHub soon. 注：数据集处理代码已单独存档，即将同步发布至GitHub平台。 ==================================== Methods Deduplication: 结合文本相似度与元数据特征完成重复记录的移除 Label design: 经过六项诊断性校验后完成疾病标签的标准化处理，并移除泛化性过强的“通用病理状态（General Pathological Conditions）”类别。六项校验包括：①词汇重叠度分析；②基于TF-IDF（Term Frequency-Inverse Document Frequency，1-2元语法）的类别质心无监督可分性检验；③类内凝聚性检验；以及基于带信息性Dirichlet先验的对数比法评估类别区分度。该流程旨在减少语义溢出问题，提升类别间的可区分性 Short-abstract filter: 移除长度低于设定阈值的记录，以提升下游训练任务的质量 Provenance: 通过标题与元数据匹配恢复PubMed文献ID（PMID），成功匹配的记录均记录对应的PMID ==================================== Intended use Biomedical NLP（生物医学自然语言处理）领域：可用于监督/无监督文本分类、模型基准测试、可靠性与模型不确定性分析，以及大语言模型（LLM）相关应用场景 Curation & reproducibility: 可作为去重、来源核查、标签体系修复等流程的示范案例 ==================================== Licensing 本衍生数据集（含元数据、标签与索引键）采用CC BY-SA 3.0协议授权 Original dataset license: CC BY-SA 3.0（详见链接：https://github.com/sebischair/Medical-Abstracts-TC-Corpus 与 https://huggingface.co/datasets/TimSchopf/medical_abstracts） Redistribution is permitted under ShareAlike terms; please attribute both the original creators and this curated release. 允许基于共享类似（ShareAlike）条款进行再分发，请同时注明原始数据集创作者与本次整理版的来源 Abstract text itself may be subject to third-party rights. Users should respect source terms when retrieving full text. 摘要文本本身可能受第三方版权约束，用户在获取完整文本时需遵守原始数据源的相关条款 ==================================== Funding & competing interests 本研究未获得任何资助。作者声明无利益冲突。本次数据集整理与验证工作均独立完成，与作者当前或过往供职机构无关 ==================================== Acknowledgments & original source 本语料库源自https://www.kaggle.com/datasets/chaitanyakck/medical-text 发布的原始社区数据集。感谢原始数据集创作者以CC BY-SA 3.0协议公开其资源