Reproducible Evaluation of Open-Source Tools for Prostate Segmentation on Public Datasets

Segmentation of the prostate and surrounding regions is important for a variety of clinical and research applications. Our goal is to evaluate the generalizability of publicly available state-of-the-art AI models on publicly available datasets. To compare the AI generated segmentations to the available manually annotated ground-truth, quantitative measures such as Dice Coefficient and Hausdorff distance, along with shape radiomics features, were analyzed. Our study also aims to show how cloud-based tools can be used to analyze, store, and visualize evaluation results. Three open-source pre-trained AI prostate segmentation tools were evaluated against expert annotations, on three publicly available MRI prostate collections, available in NCI Imaging Data Commons[1]. Two pre-trained models originate from the nnU-Net framework[2], the last pre-trained model originates from Prostate158 paper[4]. ProstateX[5], QIN-Prostate-Repeatability[6] and PROSTATE-MRI-US-Biopsy[7]. Expert annotations of the the whole prostate gland, peripheral zone (PZ) and transition zone (TZ) of the prostate are available for ProstateX collection, whole prostate gland and PZ for QIN-Prostate-Repeatability collection, and whole prostate gland for PROSTATE-MRI-US-Biopsy collection. We rely on the DICOM standard to encode our segmentation and radiomics results. The DICOM standard aims to achieve interoperability and FAIR[10] principles. Encoding our results in DICOM representation allows us to leverage DICOM-reliant tools, such as Google Cloud Computing tools for storage,computation, analysis and visualization. Open-source DICOM-based visualization tools such as OHIF[8] viewer can also be used to look qualitatively at the AI and expert annotations and the referenced images.. DICOM Segmentation objects are used to encode the AI models predictions, using dcmqi[11], DICOM Structured Reports on the other hand are used to encode radiomics features[3] extracted from the AI and expert annotations, using dcmqi and highdicom[12]. This dataset is organized in three parts: AI_SEGMENTATIONS_DICOM.zip, AI_STRUCTURED_REPORTS_DICOM.zip and EXPERT_SRUCTURED_REPORTS_DICOM..zip. All zip files contain DICOM objects only, sorted based on DICOM attributes, following this pattern: PatientID/ └───Modality-%StudyInstanceUID/ └───%SeriesInstanceUID-%SeriesDescription.dcm. AI_SEGMENTATIONS_DICOM.zip contains all the pre-trained AI models evaluated segmentation results, encoded as DICOM Segmentation objects. AI_STRUCTURED_REPORTS_DICOM..zip contains firstorder and shape radiomics features extracted for the AI segmentation results, such as Segmentation Volume, encoded as DICOM Structured Reports. EXPERT_SRUCTURED_REPORTS_DICOM.zip contains firstorder and shape radiomics features extracted for the expert annotations (for ProstateX, QIN-Prostate-Repeatability and PROSTATE-MRI-US-Biopsy collections) stored a DICOM Structured Reports objects. Code repository containing evaluation cloud-based notebooks and results/metadata .csv tables is available here:https://github.com/ImagingDataCommons/idc-prostate-mri-analysis Additional Notes This project has been funded in whole or in part with Federal funds from the NCI, NIH, under task order no. HHSN26110071 under contract no. HHSN261201500003l.https://portal.imaging.datacommons.cancer.gov/ nnU-Net: https://github.com/MIC-DKFZ/nnUNet Prostate158: https://github.com/Project-MONAI/model-zoo/tree/dev/models/prostate_mri_anatomyPyradiomics: https://github.com/AIM-Harvard/pyradiomics Highdicom: https://github.com/herrmannlab/highdicom DCMQI: https://github.com/QIICR/dcmqi Github repo: https://github.com/ImagingDataCommons/idc-prostate-mri-analysis Related information ProstateX - https://doi.org/10.7937/K9TCIA.2017.MURS5CL QIN-Prostate-Repeatability - https://doi.org/10.7937/K9/TCIA.2018.MR1CKGNDPROSTATE-MRI-US-BIOPSY - https://doi.org/10.7937/TCIA.2020.A61IOC1A References [1] Fedorov A, Longabaugh WJ, Pot D, Clunie DA, Pieper S, Aerts HJ, Homeyer A, Lewis R, Akbarzadeh A, Bontempi D, Clifford W. NCI imaging data commons. Cancer research. 2021 Aug 8;81(16):4188. [2] Isensee F, Jaeger PF, Kohl SA, Petersen J, Maier-Hein KH. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods. 2021 Feb;18(2):203-11. [3] Van Griethuysen JJ, Fedorov A, Parmar C, Hosny A, Aucoin N, Narayan V, Beets-Tan RG, Fillion-Robin JC, Pieper S, Aerts HJ. Computational radiomics system to decode the radiographic phenotype. Cancer research. 2017 Nov 1;77(21):e104-7. [4] Adams, Lisa C., Marcus R. Makowski, Günther Engel, Maximilian Rattunde, Felix Busch, Patrick Asbach, Stefan M. Niehues, et al. 2022. “Prostate158 - An Expert-Annotated 3T MRI Dataset and Algorithm for Prostate Cancer Detection.” Computers in Biology and Medicine 148 (September): 105817. [5] Natarajan, S., Priester, A., Margolis, D., Huang, J., & Marks, L. (2020). Prostate MRI and Ultrasound With Pathology and Coordinates of Tracked Biopsy (Prostate-MRI-US-Biopsy) (version 2) [Data set]. The Cancer Imaging Archive. DOI: 10.7937/TCIA.2020.A61IOC1A [6] Fedorov, A; Schwier, M; Clunie, D; Herz, C; Pieper, S; Kikinis, R; Tempany, C; Fennessy, F. (2018). Data From QIN-PROSTATE-Repeatability. The Cancer Imaging Archive. DOI: 10.7937/K9/TCIA.2018.MR1CKGND [7] Natarajan, S., Priester, A., Margolis, D., Huang, J., & Marks, L. (2020). Prostate MRI and Ultrasound With Pathology and Coordinates of Tracked Biopsy (Prostate-MRI-US-Biopsy) (version 2) [Data set]. The Cancer Imaging Archive. DOI: 10.7937/TCIA.2020.A61IOC1A [8] Open Health Imaging Foundation Viewer: An Extensible Open-Source Framework for Building Web-Based Imaging Applications to Support Cancer Research. Erik Ziegler, Trinity Urban, Danny Brown, James Petts, Steve D. Pieper, Rob Lewis, Chris Hafey, and Gordon J. Harris [9] Clark K, Vendt B, Smith K, Freymann J, Kirby J, Koppel P, Moore S, Phillips S, Maffitt D, Pringle M, Tarbox L. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. Journal of digital imaging. 2013 Dec;26(6):1045-57. [10] Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, Blomberg N, Boiten JW, da Silva Santos LB, Bourne PE, Bouwman J. The FAIR Guiding Principles for scientific data management and stewardship. Scientific data. 2016 Mar 15;3(1):1-9. [11] Herz C, Fillion-Robin JC, Onken M, Riesmeier J, Lasso A, Pinter C, Fichtinger G, Pieper S, Clunie D, Kikinis R, Fedorov A. DCMQI: an open source library for standardized communication of quantitative image analysis results using DICOM. Cancer research. 2017 Nov 1;77(21):e87-90. [12] Bridge CP, Gorman C, Pieper S, Doyle SW, Lennerz JK, Kalpathy-Cramer J, Clunie DA, Fedorov AY, Herrmann MD. Highdicom: A python library for standardized encoding of image annotations and machine learning model outputs in pathology and radiology. Journal of Digital Imaging. 2022 Aug 22:1-9.