Supplementary Material for: Machine Learning Risk Prediction for Treated Retinopathy of Prematurity in Infants

Introduction: Retinopathy of prematurity (ROP) is a leading cause of childhood blindness. However, current screening guidelines may be overly broad, necessitating better models to detect high-risk infants. Methods: From a multicenter cohort of 103,701 infants (3301 [3.2%] treated for ROP) discharged from 298 neonatal intensive care units from 2006–2017 with birth weight ≤ 1500 grams or gestational age ≤ 30 weeks, we used clinically relevant variables to develop machine learning (ML) models at 2-week intervals from postnatal day 14 to 140 to stratify infants by ROP treatment timing. We assessed model performance by concordance index (C-index), area under the receiver operating characteristic curve (AUROC), and average precision (AP), validated performance in a cohort of 25,105 infants across 231 sites from 2018–2020, and compared model performance to a logistic regression (LR) model. Results: In the validation cohort, the day-28 ML model outperformed the LR model by AUROC (0.916 [0.905–0.926] vs 0.903 [0.892–0.914]; p<0.001) and AP (0.190 [0.167–0.217] vs 0.160 [0.140–0.183]; p<0.001). Using the ML model at a 100% sensitivity threshold would have negative predictive value of 99.9% and could reduce the number of infants needing screening by 12% compared to current guidelines. Conclusion: ML models can effectively predict the need for ROP treatment and stratify infants by risk, potentially reducing unneeded screening. Future work is needed to translate model-based ROP predictions to the clinical setting.