jang1563/ContradictBio-1138

--- license: cc-by-nc-4.0 task_categories: - text-classification language: - en tags: - biology - biomedicine - contradiction-detection - natural-language-inference - evidence-quality - peer-review - cross-paper size_categories: - 1K<n<10K pretty_name: ContradictBio-1138 dataset_info: features: - name: id dtype: string - name: entry_source dtype: string - name: source_pmid dtype: string - name: source_doi dtype: string - name: paper_title dtype: string - name: claim_a dtype: string - name: claim_b dtype: string - name: is_genuine_contradiction dtype: bool - name: contradiction_type dtype: string - name: confidence dtype: float64 - name: rationale dtype: string - name: abstract_text dtype: string - name: confidence_tier dtype: int64 - name: source_pmid_b dtype: string - name: source_doi_b dtype: string - name: paper_title_b dtype: string - name: abstract_text_b dtype: string splits: - name: train num_examples: 1138 --- # ContradictBio-1138 A biomedical contradiction detection corpus combining within-abstract and cross-paper claim pairs, with 5-category taxonomy and multi-model cross-validation. ## Overview ContradictBio-1138 contains 1,138 biomedical claim pairs labeled for contradiction detection: **338 within-abstract pairs** (from [ContradictBio-338](https://huggingface.co/datasets/jang1563/ContradictBio-338)) plus **800 cross-paper pairs** comparing claims across different publications. Each entry is classified as either **genuine contradiction** or **contextual (non-contradiction)**, with genuine contradictions categorized into 4 types. This corpus was developed as part of [BioTeam-AI](https://github.com/jang1563/bioteam-ai), a multi-agent research automation system for biology. ## Corpus Composition | Source | Entries | Genuine | Contextual | Panel Validated | |--------|---------|---------|------------|-----------------| | **Within-abstract** (v3) | 338 | 123 | 215 | Yes (6-rater PoLL) | | **Cross-paper** (v4) | 800 | 97 | 703 | No (tier 0) | | **Total** | 1,138 | 220 | 918 | | ## Contradiction Taxonomy (5 categories) | Type | Within-abstract | Cross-paper | Total | Description | |------|-----------------|-------------|-------|-------------| | **direct** | 31 | 60 | 91 | Explicit factual disagreement between claims | | **temporal** | 24 | 20 | 44 | Findings that changed over time or across study periods | | **magnitude** | 23 | 1 | 24 | Quantitative disagreement (effect sizes, measurements) | | **methodological** | 45 | 15 | 60 | Contradictions arising from different experimental approaches | | **contextual** | 215 | 704 | 919 | Apparent contradictions explained by differing conditions (negative class) | ## Quality Validation ### Within-Abstract Entries (v3, 338 pairs) Validated using a Panel of LLM Evaluators (PoLL) method ([Verga et al. 2024](https://arxiv.org/abs/2404.18796)) with 3 models x 2 prompt strategies: | Model | Prompt | Precision | Recall | F1 | Parse Fail% | |-------|--------|-----------|--------|-----|-------------| | Gemini 2.5 Flash | baseline | 0.619 | 0.645 | 0.632 | 0% | | DeepSeek V3.2 | contrastive | 0.593 | 0.854 | **0.700** | 0% | | Llama 4 Scout | contrastive | 0.599 | 0.932 | **0.729** | 32.5% | **Key finding**: Contrastive prompt design drives recall from 0.16-0.65 to 0.85-0.97 across all model families. ### Tiered Confidence Labels | Tier | Criteria | Entries | Gold Match | Source | |------|----------|---------|------------|--------| | **Tier 0** | Unrated | 800 | N/A | Cross-paper (v4) | | **Tier 1** | >= 5/6 raters agree | ~132 | **94.2%** | Within-abstract (v3) | | **Tier 2** | 4/6 raters agree | ~111 | 82.0% | Within-abstract (v3) | | **Tier 3** | Split / few agree | ~95 | Needs review | Within-abstract (v3) | ## Data Format Each entry in the JSONL file contains: ```json { "id": "V4-CROSS-DIR-0010", "entry_source": "cross_paper", "source_pmid": "38749421", "source_doi": "10.1016/j.molcel.2024.04.017", "paper_title": "Micronuclei induced by radiation...", "claim_a": "Recent studies have suggested that micronuclei...", "claim_b": "The role of THEM4 in Akt signaling...", "is_genuine_contradiction": true, "contradiction_type": "direct", "confidence": 0.95, "rationale": "The claims address different biological systems...", "abstract_text": "Full abstract of paper A from PubMed...", "confidence_tier": 0, "source_pmid_b": "39011675", "source_doi_b": "10.1631/jzus.B2300457", "paper_title_b": "Roles of THEM4 in the Akt pathway...", "abstract_text_b": "Full abstract of paper B from PubMed..." } ``` ### Fields | Field | Type | Description | |-------|------|-------------| | `id` | string | Unique identifier (`V3-{TYPE}-{NNN}` for within-abstract, `V4-CROSS-{TYPE}-{NNN}` for cross-paper) | | `entry_source` | string | `"within_abstract"` or `"cross_paper"` | | `source_pmid` | string | PubMed ID of the first (or only) source paper | | `source_doi` | string | DOI of the first source paper | | `paper_title` | string | Title of the first source paper | | `claim_a` | string | First extracted claim | | `claim_b` | string | Second claim that may contradict `claim_a` | | `is_genuine_contradiction` | bool | `true` = genuine contradiction, `false` = contextual | | `contradiction_type` | string | One of: `direct`, `temporal`, `magnitude`, `methodological`, `contextual` | | `confidence` | float | Annotation confidence score (0.0-1.0) | | `rationale` | string | Explanation of why the pair is/isn't a contradiction | | `abstract_text` | string | Full abstract text of paper A from PubMed | | `confidence_tier` | int | 0 = unrated (v4), 1 = high (>=5/6 agree), 2 = medium (4/6), 3 = uncertain (<=3/6) | | `source_pmid_b` | string | PubMed ID of the second paper (cross-paper only; empty for within-abstract) | | `source_doi_b` | string | DOI of the second paper (cross-paper only) | | `paper_title_b` | string | Title of the second paper (cross-paper only) | | `abstract_text_b` | string | Full abstract of paper B (cross-paper only) | ## Usage ```python from datasets import load_dataset dataset = load_dataset("jang1563/ContradictBio-1138") # Use only panel-validated within-abstract pairs (Tier 1 = highest quality) tier1 = dataset["train"].filter(lambda x: x["confidence_tier"] == 1) print(f"Tier 1 (validated): {len(tier1)} entries") # Use only cross-paper pairs cross = dataset["train"].filter(lambda x: x["entry_source"] == "cross_paper") print(f"Cross-paper pairs: {len(cross)}") # Filter genuine contradictions genuine = dataset["train"].filter(lambda x: x["is_genuine_contradiction"]) print(f"Genuine contradictions: {len(genuine)}") # Access both abstracts for cross-paper entries for ex in cross.select(range(3)): print(f"[{ex['contradiction_type']}] Paper A: {ex['paper_title'][:60]}...") print(f" vs Paper B: {ex['paper_title_b'][:60]}...") ``` ## Intended Use - **Benchmarking** contradiction detection systems (within-abstract and cross-paper) - **Training** classifiers for biomedical claim pair analysis - **Evaluating** prompt strategies for scientific claim analysis - **Research** on evidence quality, scientific disagreement, and literature consistency ## Related Datasets - [ContradictBio-338](https://huggingface.co/datasets/jang1563/ContradictBio-338) — the within-abstract subset with full 6-rater cross-validation details ## Limitations - Cross-paper entries (800) have not undergone multi-rater panel validation (confidence_tier = 0) - Gold labels created by a single annotator with LLM-assisted cross-validation for within-abstract subset only - Cross-paper pairs may include claims from unrelated biological domains - 1 cross-paper entry (V4-CROSS-DIR-0138) has `is_genuine_contradiction=true` but `contradiction_type=contextual` due to a labeling inconsistency in the v4 generation pipeline - Some within-abstract entries use non-standard ID format (`V3-MET-C{NNN}`) - 8/338 within-abstract entries have empty `source_doi`; all entries are fully citable via `source_pmid` ## Citation ```bibtex @software{kim2026bioteamai, title = {BioTeam-AI: Personal AI Science Team for Biology Research}, author = {Kim, JangKeun}, year = {2026}, url = {https://github.com/jang1563/bioteam-ai}, license = {MIT} } ``` ## License [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — Attribution-NonCommercial 4.0 International. **You are free to:** - Use, share, and adapt this dataset for **research, education, and non-profit purposes** - Cite this work in academic publications **You may NOT:** - Use this dataset for **commercial purposes** without explicit written permission from the author For commercial licensing inquiries, contact the author via the [BioTeam-AI repository](https://github.com/jang1563/bioteam-ai).许可证：CC BY-NC 4.0 任务类别： - 文本分类 语言：英语 标签： - 生物学 - 生物医学 - 矛盾检测 - 自然语言推理 - 证据质量 - 同行评审 - 跨论文 规模类别：1000 < 样本量 < 10000 友好名称：ContradictBio-1138 数据集信息： 字段列表： - 名称：id 数据类型：字符串 - 名称：entry_source 数据类型：字符串 - 名称：source_pmid 数据类型：字符串 - 名称：source_doi 数据类型：字符串 - 名称：paper_title 数据类型：字符串 - 名称：claim_a 数据类型：字符串 - 名称：claim_b 数据类型：字符串 - 名称：is_genuine_contradiction 数据类型：布尔型 - 名称：contradiction_type 数据类型：字符串 - 名称：confidence 数据类型：64位浮点型 - 名称：rationale 数据类型：字符串 - 名称：abstract_text 数据类型：字符串 - 名称：confidence_tier 数据类型：64位整型 - 名称：source_pmid_b 数据类型：字符串 - 名称：source_doi_b 数据类型：字符串 - 名称：paper_title_b 数据类型：字符串 - 名称：abstract_text_b 数据类型：字符串 数据划分： - 名称：训练集 样本数：1138 # ContradictBio-1138 生物医学矛盾检测语料库，整合摘要内部与跨论文主张对，包含5类分类体系与多模型交叉验证方案。 ## 概述 ContradictBio-1138包含1138条经矛盾检测标注的生物医学主张对：**338条摘要内部主张对**（源自[ContradictBio-338](https://huggingface.co/datasets/jang1563/ContradictBio-338)）与**800条跨论文主张对**（比对不同出版物中的研究主张）。 每条条目被分类为**真实矛盾**或**语境性非矛盾**，其中真实矛盾可进一步划分为4个子类型。 本语料库作为[BioTeam-AI](https://github.com/jang1563/bioteam-ai)的一部分开发而成，后者是一款面向生物学研究的多智能体（AI Agent）研究自动化系统。 ## 语料库构成 | 数据来源 | 条目数 | 真实矛盾数 | 语境性非矛盾数 | 经专家组验证 | |------------------|--------|------------|----------------|--------------| | **摘要内部（v3）** | 338 | 123 | 215 | 是（6评委会大语言模型评估法（Panel of LLM Evaluators，简称PoLL）） | | **跨论文（v4）** | 800 | 97 | 703 | 否（等级0） | | **总计** | 1138 | 220 | 918 | | ## 矛盾分类体系（5大类） | 类型 | 摘要内部 | 跨论文 | 总计 | 描述 | |----------------|----------|--------|------|----------------------------------------------------------------------| | **直接型** | 31 | 60 | 91 | 主张间存在明确的事实分歧 | | **时间型** | 24 | 20 | 44 | 随时间或研究周期变化的研究发现 | | **量级型** | 23 | 1 | 24 | 定量结果分歧（如效应量、测量值） | | **方法论型** | 45 | 15 | 60 | 因实验方法差异产生的矛盾 | | **语境型** | 215 | 704 | 919 | 可通过实验条件差异解释的表观矛盾（负类，即非真实矛盾） | ## 质量验证 ### 摘要内部条目（v3，338对） 采用大语言模型评估委员会（Panel of LLM Evaluators，PoLL）方法进行验证（[Verga et al. 2024](https://arxiv.org/abs/2404.18796)），实验设置为3种模型×2种提示策略： | 模型 | 提示策略 | 精确率 | 召回率 | F1值 | 解析失败率 | |------------------|------------|--------|--------|--------|------------| | Gemini 2.5 Flash | 基准策略 | 0.619 | 0.645 | 0.632 | 0% | | DeepSeek V3.2 | 对比提示 | 0.593 | 0.854 | **0.700** | 0% | | Llama 4 Scout | 对比提示 | 0.599 | 0.932 | **0.729** | 32.5% | **核心发现**：对比提示设计可将所有模型家族的召回率从0.16~0.65提升至0.85~0.97。 ### 分层置信度标签 | 标签等级 | 判定标准 | 条目数 | 金标准匹配率 | 数据来源 | |----------|------------------------------|--------|--------------|------------------| | **等级0** | 未标注 | 800 | 无可用数据 | 跨论文（v4） | | **等级1** | ≥5/6名评估者达成一致意见 | 约132 | **94.2%** | 摘要内部（v3） | | **等级2** | 4/6名评估者达成一致意见 | 约111 | 82.0% | 摘要内部（v3） | | **等级3** | 评估意见分歧/少数评估者一致 | 约95 | 需复核 | 摘要内部（v3） | ## 数据格式 JSONL文件中的每条条目包含以下字段： json { "id": "V4-CROSS-DIR-0010", "entry_source": "cross_paper", "source_pmid": "38749421", "source_doi": "10.1016/j.molcel.2024.04.017", "paper_title": "辐射诱导的微核...", "claim_a": "近期研究表明微核...", "claim_b": "THEM4在Akt信号通路中的作用...", "is_genuine_contradiction": true, "contradiction_type": "direct", "confidence": 0.95, "rationale": "两项主张针对不同的生物系统...", "abstract_text": "PubMed收录的论文A完整摘要...", "confidence_tier": 0, "source_pmid_b": "39011675", "source_doi_b": "10.1631/jzus.B2300457", "paper_title_b": "THEM4在Akt通路中的作用...", "abstract_text_b": "PubMed收录的论文B完整摘要..." } ### 字段说明 | 字段名 | 数据类型 | 说明 | |---------------------|------------|----------------------------------------------------------------------| | `id` | 字符串 | 唯一标识符（摘要内部条目格式为`V3-{TYPE}-{NNN}`，跨论文条目格式为`V4-CROSS-{TYPE}-{NNN}`） | | `entry_source` | 字符串 | 取值为`"within_abstract"`（摘要内部）或`"cross_paper"`（跨论文） | | `source_pmid` | 字符串 | 首篇（或单篇）源论文的PubMed ID | | `source_doi` | 字符串 | 首篇源论文的数字对象标识符（DOI） | | `paper_title` | 字符串 | 首篇源论文的标题 | | `claim_a` | 字符串 | 第一条提取出的研究主张 | | `claim_b` | 字符串 | 可能与`claim_a`存在矛盾的第二条研究主张 | | `is_genuine_contradiction` | 布尔型 | `true`表示真实矛盾，`false`表示语境性非矛盾 | | `contradiction_type`| 字符串 | 取值为：`direct`（直接型）、`temporal`（时间型）、`magnitude`（量级型）、`methodological`（方法论型）、`contextual`（语境型） | | `confidence` | 浮点型 | 标注置信度得分（取值范围0.0~1.0） | | `rationale` | 字符串 | 解释该主张对是否存在矛盾的依据 | | `abstract_text` | 字符串 | PubMed收录的论文A的完整摘要文本 | | `confidence_tier` | 整型 | 0=未标注（v4版本），1=高置信度（≥5/6名评估者一致），2=中等置信度（4/6名评估者一致），3=低置信度（≤3/6名评估者一致） | | `source_pmid_b` | 字符串 | 第二篇论文的PubMed ID（仅跨论文条目有效；摘要内部条目留空） | | `source_doi_b` | 字符串 | 第二篇论文的DOI（仅跨论文条目有效） | | `paper_title_b` | 字符串 | 第二篇论文的标题（仅跨论文条目有效） | | `abstract_text_b` | 字符串 | 第二篇论文的完整摘要文本（仅跨论文条目有效） | ## 使用示例 python from datasets import load_dataset dataset = load_dataset("jang1563/ContradictBio-1138") # 仅使用经专家组验证的高置信度摘要内部条目（等级1） tier1 = dataset["train"].filter(lambda x: x["confidence_tier"] == 1) print(f"高置信度等级1条目：{len(tier1)} 条") # 仅使用跨论文条目 cross = dataset["train"].filter(lambda x: x["entry_source"] == "cross_paper") print(f"跨论文主张对：{len(cross)} 条") # 筛选真实矛盾条目 genuine = dataset["train"].filter(lambda x: x["is_genuine_contradiction"]) print(f"真实矛盾条目：{len(genuine)} 条") # 访问跨论文条目的两篇论文摘要 for ex in cross.select(range(3)): print(f"[{ex['contradiction_type']}] 论文A：{ex['paper_title'][:60]}...") print(f" vs 论文B：{ex['paper_title_b'][:60]}...") ## 预期用途 - **基准测试**：矛盾检测系统（涵盖摘要内部与跨论文场景） - **模型训练**：生物医学主张对分析分类器 - **方法评估**：科学主张分析的提示策略优化 - **科学研究**：证据质量、科学分歧与文献一致性相关研究 ## 相关数据集 - [ContradictBio-338](https://huggingface.co/datasets/jang1563/ContradictBio-338) — 包含完整6评委会交叉验证细节的摘要内部子集语料库 ## 局限性 - 跨论文条目（共800条）未经过多评委会专家组验证，置信度等级为0 - 金标准标签仅在摘要内部子集中由单标注员结合大语言模型辅助交叉验证生成 - 跨论文条目可能包含来自无关生物领域的研究主张 - 1条跨论文条目（V4-CROSS-DIR-0138）存在标注不一致问题：`is_genuine_contradiction=true`但`contradiction_type=contextual`，源于v4生成流水线的标注误差 - 部分摘要内部条目使用非标准ID格式（`V3-MET-C{NNN}`） - 8/338条摘要内部条目缺失`source_doi`字段，但所有条目均可通过`source_pmid`实现完整引用 ## 引用格式 bibtex @software{kim2026bioteamai, title = {BioTeam-AI: 面向生物学研究的个性化AI科研团队}, author = {Kim, JangKeun}, year = {2026}, url = {https://github.com/jang1563/bioteam-ai}, license = {MIT} } ## 许可证 [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — 署名-非商业性使用4.0国际许可协议。 **您可自由进行以下操作：** - 出于**研究、教育与非营利目的**使用、分享和改编本数据集 - 在学术出版物中引用本工作 **禁止行为：** - 未经作者书面许可，将本数据集用于**商业用途** 商业许可咨询请通过[BioTeam-AI仓库](https://github.com/jang1563/bioteam-ai)联系作者。