Learning from Disagreement in Digital Pathology: Study-Level Extraction, Taxonomy Coding, and Analysis Materials

This dataset contains the study-level data and reproducibility materials supporting the systematic methodological review “Learning from Disagreement in Digital Pathology: A Systematic Review and Operational Taxonomy of Multi-Rater Disagreement in Artificial Intelligence Studies.” The review examined how human-rater disagreement is represented, quantified, aggregated, incorporated into artificial-intelligence model development and evaluation, and explicitly modeled across digital-pathology studies. The repository includes structured data for 345 included studies, comprising the final study-level extraction dataset, independent reviewer taxonomy assignments, taxonomy agreement and conflict analyses, a complementary multi-label disagreement analysis, harmonized annotator-characteristics coding, mutually exclusive external-testing classification, review-specific methodological-appraisal materials, and exploratory secondary analyses. The five-level operational taxonomy distinguishes: Level 0, disagreement not characterized in the artificial-intelligence workflow; Level 1, disagreement quantified only; Level 2, ratings aggregated to a single reference; Level 3, disagreement incorporated without explicit annotator modeling; and Level 4, explicit annotator-specific modeling. Additional materials document the operational definitions used for the methodological appraisal, the boundary between review eligibility and Level 0 classification, and the harmonization rules applied to annotator characteristics and external-testing categories. These materials are provided to support transparency, reproducibility, secondary methodological research, and future comparisons of disagreement-aware artificial-intelligence approaches in digital pathology. The taxonomy represents the operational use of disagreement information and is not a ranking of methodological quality. Exploratory statistical analyses should be interpreted as hypothesis-generating rather than confirmatory. Copyrighted full-text articles are not redistributed in this repository.