Are replication rates the same across academic fields? community forecasts from the DARPA SCORE program

The DARPA program “Systematizing Confidence in Open Research and Evidence” (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioral sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in Psychology and in Education (average survey response of 42% for both fields). These results reveal insights into the academic community’s views of the replication crisis, including for research fields for which no large-scale replication studies have been undertaken yet.
Methods
This dataset was collected via online platform over the course two weeks in August 2019. The participants of the surveys and prediction markets were recruited through blog posts, twitter and emailing lists, primarily aimed at academics, however anyone was free to join. The data was exported from the online platforms and anonymised before processing and analysis.

The datatables were processed into formats more suited to analysis, and columns name changed to be more intuitive. New columns were also appended which provided information, for instance survey question wording was included as well as question code. All ‘test’ surveys (or surveys completed by admins) were removed before analysis.