Calibrated Model Criticism Using Split Predictive Checks

Assessing how well a Bayesian model generalizes to unobserved data is essential, yet existing general-purpose model checks are either not properly calibrated (as in posterior predictive checks) or fail to be sufficiently general for practical use (e.g., due to requiring model-specific derivations). We propose split predictive checks (SPCs) as a simple, general-purpose class of predictive checks that maintain the usability of posterior predictive checks while directly targeting predictive generalization. SPCs work by splitting the data into training and test subsets, then fitting the model to the former and evaluating predictive discrepancies on the latter. We develop an asymptotic theory for two variants – single SPCs and divided SPCs – and show that, unlike posterior predictive checks, both yield asymptotically calibrated (hence interpretable) p-values. Our results show that single SPCs work well at identifying substantial misspecification, while divided SPCs are sensitive even to subtle departures from modeling assumptions. Through simulation studies and real-data applications, we show that SPCs provide reliable, flexible, and computationally efficient assessments of Bayesian model fit, often revealing issues with predictive generalization missed by other predictive checks.