Supporting data for "The need to approximate the use-case in clinical machine learning"

The availability of smartphone and wearable sensor technology is leading to a rapid accumulation of human subject data, and machine learning is emerging as a technique to map that data into clinical predictions. As machine learning algorithms are increasingly used to support clinical decision making, it is vital to reliably quantify their prediction accuracy. Cross-validation is the standard approach where the accuracy of such algorithms is evaluated on data the algorithm has not seen during training. However, for this procedure to be meaningful, the relationship between the training and validation set should mimic the relationship between the training set and the dataset expected for the clinical use. Here we compared two popular cross-validation methods: record-wise and subject-wise. The subject-wise procedure mirrors the clinically relevant use-case scenario of diagnosing/identifying patterns in newly recruited subjects. The record-wise strategy has no such interpretation. Using both a publicly available dataset and a simulation, we found that record-wise cross-validation often massively overestimates the prediction accuracy of the algorithms. We also conducted a systematic review of the relevant literature, and found that this overly optimistic method is used by almost half of the retrieved studies that used accelerometers, wearable sensors, or smartphones to predict clinical outcomes. As we move towards an era of machine learning based diagnosis and treatment, using proper methods to evaluate their accuracy is crucial, as results that are overly optimistic can mislead both clinicians and data scientists.