The posterior probability of a null hypothesis being true given a statistically significant result

When researchers carry out a null hypothesis significance test, it is tempting to assume that a statistically significant result lowers Prob(H0), the probability of the null hypothesis being true. Technically, such a state- ment is meaningless for various reasons: e.g., the null hypothesis does not have a probability associated with it. However, it is possible to relax certain assumptions to compute the posterior probability Prob(H0) under repeated sampling. We show in a step-by-step guide that the intuitively appealing belief, that Prob(H0) is low when significant results have been obtained under repeated sampling, is in general incorrect and depends greatly on: (a) the prior probability of the null being true; (b) type-I error rate, (c) type-II error rate, and (d) replication of a result. Through step-by-step simulations using open-source code in the R System of Statistical Computing, we show that uncertainty about the null hypothesis being true often remains high despite a significant result. To help the reader develop intuitions about this common misconception, we provide a Shiny app (https://danielschad.shinyapps.io/probnull/). We expect that this tutorial will help researchers better understand and judge results from null hypothesis significance tests.

在进行零假设显著性检验时，研究者往往会倾向于假设一个统计上显著的结果降低了零假设（H0）为真的概率，Prob(H0)。然而，从技术角度来看，这样的陈述因多种原因而毫无意义：例如，零假设并不与任何概率相关联。尽管如此，通过放宽某些假设，我们可以在重复抽样的情况下计算后验概率Prob(H0)。我们在一个逐步指南中展示，这种直观上令人信服的信念——即在重复抽样中获得显著结果时，Prob(H0)很低——在一般情况下是错误的，并且很大程度上取决于：(a) 零假设为真的先验概率；(b) 第一类错误率；(c) 第二类错误率；(d) 结果的重复性。通过使用R系统统计计算的开放源代码进行逐步模拟，我们表明，即使在获得显著结果的情况下，对零假设为真的不确定性往往仍然很高。为了帮助读者形成对此种常见误解的直觉，我们提供了一个Shiny应用程序（https://danielschad.shinyapps.io/probnull/）。我们期望这篇教程将帮助研究者更好地理解和评估零假设显著性检验的结果。