A Goodness-of-Fit Assessment for General Learning Procedures in High Dimensions

Black-box learners have demonstrated remarkable success across various fields due to their high predictive accuracy. However, the complexity of their learning procedures poses significant challenges in evaluating whether a given learner has achieved optimal performance on datasets with unknown data-generating mechanisms. We propose a general goodness-of-fit test for assessing different learning procedures involving high-dimensional predictors, encompassing methods from classical linear regression to advanced neural networks. Our goodness-of-fit test leverages data-splitting, using the test set to evaluate the black-box learner trained on the training set. By examining the cumulative covariance of the residuals, our method can effectively handle high-dimensional predictors. Extensive simulations and three real data analyses validate the effectiveness of our method. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.