R results/data after simulation from How best to quantify replication success? A simulation study on the comparison of replication success metrics

To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses.

为应对学界广泛热议的信任危机，重复研究的开展正愈发普遍。目前已有多种频率学派（frequentist）与贝叶斯学派（Bayesian）的评估指标被提出，用于判断重复研究是否成功，但学界对何种方法最能准确衡量重复研究的成功性仍知之甚少。本研究为首批尝试之一，旨在对比多种衡量重复研究成功性的定量指标，在已知真实研究效应、同时纳入发表偏倚（publication bias）的情境下，评估其得出正确推论的能力。研究结果显示，整体而言贝叶斯指标的表现略优于频率学派指标。通常情况下，元分析（meta-analytic）方法的表现略优于针对单一项研究的评估指标，但在极端发表偏倚的场景中，这一优势格局会发生反转。