On the Reproducibility of Psychological Science

Investigators from a large consortium of scientists recently performed a multi-year study in which they replicated 100 psychology experiments. Although statistically significant results were reported in 97% of the original studies, statistical significance was achieved in only 36% of the replicated studies. This article presents a reanalysis of these data based on a formal statistical model that accounts for publication bias by treating outcomes from unpublished studies as missing data, while simultaneously estimating the distribution of effect sizes for those studies that tested nonnull effects. The resulting model suggests that more than 90% of tests performed in eligible psychology experiments tested negligible effects, and that publication biases based on <i>p</i>-values caused the observed rates of nonreproducibility. The results of this reanalysis provide a compelling argument for both increasing the threshold required for declaring scientific discoveries and for adopting statistical summaries of evidence that account for the high proportion of tested hypotheses that are false. Supplementary materials for this article are available online.

某大型科学家联盟的研究人员近期开展了一项历时多年的研究，对100项心理学实验进行了重复验证。尽管原研究中有97%报告了具有统计学显著性的结果，但重复研究中仅36%获得了统计学显著性。本文基于正式统计模型对上述数据进行了重新分析：该模型将未发表研究的结果视为缺失数据以校正发表偏倚，同时对检验非零效应的研究的效应量分布进行估计。所得模型表明，在符合纳入标准的心理学实验中，超过90%的检验所针对的效应均可忽略；而基于p值的发表偏倚正是导致观测到的实验不可重复率的原因。本次重新分析的结果为两项举措提供了有力支撑：一是提高宣布科学发现所需的判定阈值，二是采用能够考量被检验假设中占比极高的虚假假设的证据统计汇总方法。本文的补充材料可在线获取。