Variation and Covariation in Large-Scale Replication Projects: An Evaluation of Replicability

Over the last decade, large-scale replication projects across the biomedical and social sciences have reported relatively low replication rates. In these large-scale replication projects, replication has typically been evaluated based on a single replication study of some original study and dichotomously as successful or failed. However, evaluations of replicability that are based on a single study and are dichotomous are inadequate, and evaluations of replicability should instead be based on multiple studies, be continuous, and be multi-faceted. Further, such evaluations are in fact possible due to two characteristics shared by many large-scale replication projects. In this article, we provide such an evaluation for two prominent large-scale replication projects, one which replicated a phenomenon from cognitive psychology and another which replicated 13 phenomena from social psychology and behavioral economics. Our results indicate a very high degree of replicability in the former and a medium to low degree of replicability in the latter. They also suggest an unidentified covariate in each, namely ocular dominance in the former and political ideology in the latter, that is theoretically pertinent. We conclude by discussing evaluations of replicability at large, recommendations for future large-scale replication projects, and design-based model generalization. Supplementary materials for this article are available online.

近十年来，生物医学与社会科学领域的大规模重复研究项目均报告了相对偏低的重复率。在这类大规模重复研究项目中，重复研究的评估通常仅基于对某一原始研究的单次重复实验，并以二分法将其划分为成功或失败。然而，仅基于单次研究的二分法可复现性评估并不充分，可复现性评估应基于多项研究、采用连续度量方式并从多维度展开。此外，得益于诸多大规模重复研究项目共有的两项特征，这类多维度评估实际上是可行的。本文针对两项极具影响力的大规模重复研究项目开展了此类评估：其中一项重复了认知心理学领域的某一现象，另一项则重复了社会心理学与行为经济学领域的13项现象。研究结果显示，前者的可复现程度极高，后者则处于中等至偏低水平。研究还发现，两项研究各自存在一项未被探明的协变量：前者为眼优势（ocular dominance），后者为政治意识形态（political ideology），二者均具备理论相关性。最后，本文就通用的可复现性评估、未来大规模重复研究项目的设计建议，以及基于研究设计的模型泛化展开了讨论。本文的补充材料可在线获取。