Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.

科学发现的可复制性对于科学进步至关重要。本研究对2010年至2015年间发表在《自然》和《科学》杂志上的21项社会科学实验研究进行了系统性的复制。这些复制遵循了原始作者审核并通过的 分析计划，并在复制前进行了预注册。这些复制研究具有高效力，样本量平均比原始研究高约五倍。我们发现，有13项（62%）研究的结果与原始研究呈现显著且方向一致的效果，且复制研究的效应量平均约为原始效应量的50%。对于补充的可复制性指标，12项（57%）至14项（67%）研究之间存在差异。与这些结果一致，贝叶斯分析中估计的真实阳性率为67%，真实阳性的相对效应量估计为71%，这表明假阳性以及真实阳性效应量的膨胀均导致了不可完全复制的现象。此外，我们发现同行对可复制性的信念与可复制性密切相关，这表明研究界可以预测哪些结果将会得到复制，且未能复制的结果并非仅仅是偶然所致。