Supplementary Material for: From development to implementation: a systematic review on the current maturity status of artificial intelligence models for patients with colorectal cancer liver metastases

Introduction
Artificial intelligence (AI) is increasingly being researched and developed in the medical field and holds potential to transform healthcare after successful implementation. For patients with colorectal cancer liver metastases (CRLM), many AI models have been developed, but knowledge about translation of these models in the clinical workflow is lacking. Therefore, this systematic review aims to provide a contemporary overview of the current maturity status of AI models for patients with CRLM.

Methods
A systematic search of the literature until November 2, 2023 was conducted in PubMed, Embase.com and Clarivate Analytics/Web of Science Core Collection to identify eligible studies. Studies using AI and/or radiomics for patients with CRLM were considered eligible. Data on the study aim, study design, size of dataset, country, type of AI application, level of validation and clinical implementation status (NASA technology readiness levels) were collected. Risk of bias and applicability of the individual studies were evaluated using the Prediction model Risk Of Bias ASsessment Tool (PROBAST).

Results
A total of 117 studies were included. Ninety-seven studies (83%) were published in the last five years. The most common study design was retrospective (96%). Thirty-five studies (30%) utilized a dataset of fewer than 50 patients with CRLM. Internal validation was performed in 63% of studies, external validation in 17%. The remaining studies did not report validation. Half of the studies were classified as high risk of bias. None of the included studies performed real-time testing, workflow integration, clinical testing or clinical integration.

Conclusion
Although a rapid increase in research describing the development of AI models for patients with CRLM is observed in recent years, not a single AI model has been translated into clinical practice.