TCGA-242-Pathology-Benchmark: External Validation Dataset for "The Digital Registrar"

This dataset contains the Independent External Validation (IEV) cohort used in the study: Digital Registrar: A Schema-First Framework for Multi-Cancer Privacy-Preserving Pathology Abstraction via Local LLMs. Diagnostics. 2026; 16(11):1644. It comprises 242 de-identified surgical pathology reports selected from The Cancer Genome Atlas (TCGA) program, extracted by Kefeli, J. & Tatonetti, N. (2025, please see reference) covering five distinct categories: Breast, Colorectal, Esophagus, Stomach, and "Others" (including synchronous bilateral cancers). This benchmark serves as a "Gold Standard" for evaluating Large Language Models (LLMs) on clinical extraction tasks, providing raw text, expert-verified ground truth annotations, and baseline model outputs. If you use this code or the dataset in your research, please cite: Chow N-H, Chang H, Chen H-K, Lin C-Y, Liu Y-L, Tseng P-Y, Shiu L-J, Chu Y-W, Chung P-C, Chang K-P. Digital Registrar: A Schema-First Framework for Multi-Cancer Privacy-Preserving Pathology Abstraction via Local LLMs. Diagnostics. 2026; 16(11):1644. https://doi.org/10.3390/diagnostics16111644 BibTeX @Article{diagnostics16111644, AUTHOR = {Chow, Nan-Haw and Chang, Han and Chen, Hung-Kai and Lin, Chen-Yuan and Liu, Ying-Lung and Tseng, Po-Yen and Shiu, Li-Ju and Chu, Yen-Wei and Chung, Pau-Choo and Chang, Kai-Po}, TITLE = {Digital Registrar: A Schema-First Framework for Multi-Cancer Privacy-Preserving Pathology Abstraction via Local LLMs}, JOURNAL = {Diagnostics}, VOLUME = {16}, YEAR = {2026}, NUMBER = {11}, ARTICLE-NUMBER = {1644}, URL = {https://www.mdpi.com/2075-4418/16/11/1644}, ISSN = {2075-4418}, ABSTRACT = {Background/Objectives: Free-text surgical pathology reports hinder automated cancer registry entry and secondary analytics. This study introduces a clinically governed schema layer for interoperability, testing whether a locally-deployable Large Language Model (LLM) pipeline can deliver robust registry-grade extraction across institutions. Methods: We developed a College of American Pathologists (CAP)-aligned clinical ontology encompassing 10 cancer types, 192 per-organ scalar fields, key biomarkers, and nested structures for lymph nodes and margins. Encoded via Declarative Self-improving Python (DSPy) signatures with grammar-constrained decoding using DSPy v3.2.1, this model-agnostic pipeline was benchmarked on 893 internal reports against a pathologist-adjudicated gold standard. External validation utilized 242 The Cancer Genome Atlas (TCGA) reports. Hardware feasibility was confirmed on a single 48-gigabyte (GB) Graphics Processing Unit (GPU), ensuring suitability for privacy-preserving on-premises deployment. Results: Using the gpt-oss-20b model, the framework achieved 92.0% macro-mean exact-match accuracy on internal data, demonstrating near-perfect run-to-run reliability. Critical prognostic indicators, including breast estrogen receptor/progesterone receptor (ER/PR) (98.7%) and margin positivity (>93%), maintained high fidelity. On the external TCGA cohort, accuracy was 77.5%, rising to 88.0% after excluding structurally silent fields absent in older narratives. Operationally, the model processed reports in 40–70 s, optimally balancing speed and accuracy. Conclusions: This schema-first abstraction layer successfully decouples clinical logic from specific Artificial Intelligence (AI) models. By reliably transforming narrative reports into machine-readable structures, it establishes a portable privacy-preserving foundation for automated cancer surveillance, institutional data reuse, and future multimodal clinical systems.}, DOI = {10.3390/diagnostics16111644} }