Supporting data for "Statistical quantification of confounding bias in machine learning models"

The lack of non-parametric statistical tests for confounding bias significantly hampers the development of robust, valid and generalizable predictive models in many fields of research. Here I propose the <i>partial confounder test</i>, which, for a given confounder variable, probes the null hypotheses of the model being <i>unconfounded.</i> <br>The test provides a strict control for Type I errors and high statistical power, even for non-normally and non-linearly dependent predictions, often seen in machine learning. Applying the proposed test on models trained on large-scale functional brain connectivity data (N=1865) (i) reveals previously unreported confounders and (ii) shows that state-of-the-art confound mitigation approaches may fail preventing confounder bias in several cases. <br>The proposed test (implemented in the package <i>mlconfound1</i>) can aid the assessment and improvement of the generalizability and neurobiological validity of predictive models and, thereby, foster the development of clinically useful machine learning biomarkers