Predicting replication outcomes in the Many Labs 2 study

Understanding and improving reproducibility is crucial for scientific progress. Prediction markets and related methods of eliciting peer beliefs are promising tools to predict replication outcomes. We invited researchers in the field of psychology to judge the replicability of 24 studies replicated in the large scale Many Labs 2 project. We elicited peer beliefs in prediction markets and surveys about two replication success metrics: the probability that the replication yields a statistically significant effect in the original direction (p<0.001), and the relative effect size of the replication. The prediction markets correctly predicted 75% of the replication outcomes, and were highly correlated with the replication outcomes. Survey beliefs were also significantly correlated with replication outcomes, but had higher prediction errors. The prediction markets for relative effect sizes attracted little trading and thus did not work well. The survey beliefs about relative effect sizes performed better and were significantly correlated with observed relative effect sizes. These results suggest that replication outcomes can be predicted and that the elicitation of peer beliefs can increase our knowledge about scientific reproducibility and the dynamics of hypothesis testing.

理解与提升可重复性对于科学进步至关重要。预测市场及其相关方法在预测复制结果方面具有广阔的应用前景。我们邀请了心理学领域的学者对在大型 Many Labs 2 项目中复制的 24 项研究进行可重复性评估。我们通过预测市场和调查收集了同行意见，关于两项复制成功指标：复制研究产生原始方向统计显著效应的概率（p<0.001），以及复制的相对效应量。预测市场正确预测了 75% 的复制结果，且与复制结果高度相关。调查信念也与复制结果显著相关，但预测误差较高。相对效应量的预测市场吸引的交易很少，因此效果不佳。相对效应量的调查信念表现更佳，且与观察到的相对效应量显著相关。这些结果表明，复制结果是可以预测的，而同行意见的收集能够增强我们对科学可重复性及其假设检验动态的认识。