R code to create Figures 3 - 7 from How best to quantify replication success? A simulation study on the comparison of replication success metrics

To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses.

为应对学界热议的可重复性信任危机，重复研究的开展正愈发普遍。目前已有多种频率学派（frequentist）与贝叶斯学派（Bayesian）的评估指标被提出，用于判定重复研究是否成功，但学界对何种方法最能准确衡量重复研究的成功性仍知之甚少。本研究是首批尝试之一，旨在对比多种量化重复研究成功性的指标，在已知真实研究结果的前提下评估其做出正确推断的能力，同时纳入发表偏倚（publication bias）这一因素进行考量。研究结果显示，整体而言贝叶斯指标的表现略微优于频率学派指标。通常来说，元分析（meta-analytic）类方法的表现略微优于单研究评估指标，但在极端发表偏倚的场景下，这一优势格局会发生反转。