Bayesian versus frequentist approaches in multilevel single-case designs: on type I error rate and power [Author Accepted Manuscript]

Single-case designs (SCEDs) assess intervention effects through repeated measurements on one or a few individuals. Multilevel models nest repeated measures within individuals and have gained popularity for inferential analysis in SCEDs, in combination with expert knowledge of the clinicians and applied researchers. However, researchers often face model specification challenges without knowing the true population model underlying their data. This study evaluates how model selection criteria (AIC, BIC, WAIC, LOO) conditioned on the selected model impact statistical power and Type I error rates in intervention effects, reflecting the ecological reality where practitioners do not know the true model. A Monte Carlo simulation modelled data of AB designs varying sample size, measurement points, intervention effects, and random effect structures. Results indicated that frequentist criteria performed well in simpler models in terms of power, while Bayesian approaches showed greater robustness with respect to Type I error control. The findings provide practical insights on multilevel model selection under real-world conditions, highlighting Bayesian methods as a robust alternative for applied researchers handling small sample sizes and complex data structures. CRP was supported by the “Ayudas al Fomento de la Investigación en Másteres Oficiales 2019-2020” and “Ayudas al Fomento de la Investigación en Másteres Oficiales 2020-2021” by the Universidad Autónoma de Madrid. CRP is also supported by “Contratos predoctorales para la Formación de Personal Investigador FPI-UAM 2022”, Universidad Autónoma de Madrid, Spain. reviewed acceptedVersion