Are replication rates the same across academic fields? community forecasts from the DARPA SCORE program

The DARPA program “Systematizing Confidence in Open Research and Evidence” (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioral sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in Psychology and in Education (average survey response of 42% for both fields). These results reveal insights into the academic community’s views...