Supplementary file 1_Null hypothesis significance testing vs. Bayesian inference using generalized linear mixed models with binary outcomes: a case study under practical design constraints.pdf

Empirical investigation requires dealing with fundamental uncertainty. In experimental psychology, research questions are often addressed using Null Hypothesis Significance Testing (NHST), an approach rooted in the frequentist statistical tradition. In scenarios that do not consent to reject the null hypothesis using the NHST paradigm (i.e., results are non-significant), researchers may be tempted to reframe their analysis in the Bayesian framework, either as a complementary alternative or alongside the original NHST approach. In fact, the Bayesian approach is gaining increasing appeal in the social sciences as an alternative to the frequentist NHST framework, and Bayesian methods for hypothesis testing (i.e., the Bayes Factor) can be used to help determine whether a failure to reject the null hypothesis reflects merely insufficient evidence for the alternative hypothesis or provides affirmative evidence for the (point) null hypothesis. Nevertheless, using the two approaches interchangeably carries the risk of conceptual confusion, as NHST and Bayesian frameworks address different inferential questions. This study provides an empirical, real-world opportunity to examine how NHST and Bayesian methods can be applied to the same hypothesis test when using Generalized Linear Mixed Models with a Binary Outcome. Importantly, this application incorporates common experimental constraints into the design-analysis planning, defining a reachable, realistic albeit underpowered sample size, assuming the classical 0.80 power threshold. This research report provides a valuable opportunity to examine how Bayesian and NHST approaches potentially differ in their workflow, performance, and inferential interpretation under realistic experimental conditions.