R code to create Figures 3 - 7 from How best to quantify replication success? A simulation study on the comparison of replication success metrics

To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses.

为破解学界持续热议的可信度危机，重复开展研究的做法正日益普及。目前已有多种频率学派（frequentist）与贝叶斯学派（Bayesian）指标被提出，用于评估重复研究是否成功，但学界对哪种方法能最准确地刻画重复研究的成功性仍知之甚少。本研究为首例在结合发表偏倚（publication bias）因素且已知底层真实情况的前提下，对比多种量化重复研究成功性指标做出正确推断能力的尝试之一。研究结果显示，整体而言贝叶斯指标的表现略优于频率学派指标。通常来说，元分析（meta-analytic）方法的表现略优于针对单一研究的评估指标，但在极端发表偏倚场景下，这一优势格局会发生反转。