Out-of-Sample <i>R</i>²: Estimation and Inference

Out-of-sample prediction is the acid test of predictive models, yet an independent test dataset is often not available for assessment of the prediction error. For this reason, out-of-sample performance is commonly estimated using data splitting algorithms such as cross-validation or the bootstrap. For quantitative outcomes, the ratio of variance explained to total variance can be summarized by the coefficient of determination or in-sample <i>R</i>², which is easy to interpret and to compare across different outcome variables. As opposed to in-sample <i>R</i>², out-of-sample <i>R</i>² has not been well defined and the variability on out-of-sample R̂2 has been largely ignored. Usually only its point estimate is reported, hampering formal comparison of predictability of different outcome variables. Here we explicitly define out-of-sample <i>R</i>² as a comparison of two predictive models, provide an unbiased estimator and exploit recent theoretical advances on uncertainty of data splitting estimates to provide a standard error for R̂2. The performance of the estimators for <i>R</i>² and its standard error are investigated in a simulation study. We demonstrate our new method by constructing confidence intervals and comparing models for prediction of quantitative <i>Brassica napus</i> and <i>Zea mays</i> phenotypes based on gene expression data. Our method is available in the R-package <i>oosse</i>.