The Impact of Open Science Practices on Research Methodology: A Case Study for Research in Judgment and Decision Making

In 2011, the journal Judgment and Decision Making introduced as one of the first journals in psychology a standard request for data at submission of an article, which are also used for checks in the review process. In a sample of 71 articles published in this journal we investigate the effectiveness of this policy with respect to the prevalence of direct and indirect open science measures and their development between 2012 and 2018. For 100% of the articles data was available, 80% of the original authors responded positively to student requests for cooperation in conducting replications of their publications, for 94% of the articles materials were available or shared by the authors on request, and 96% of the original analyses were reproducible also by students, 30% of them with minor deviations or after further clarification. The usage of a priori power analyses (10%) and the reporting of effect sizes (66%) were considerably lower but increased over time. For only 4% of the articles analysis scripts were directly available, and none of the studies pre-registered hypotheses or used a pre-registered report format. There was no indication of the usage of small underpowered studies (Md(N) = 193, average effect size r = 0.30) and the p-values showed the expected right-skewed distribution without a bunching of p-values just below p = .05. Overall, adoption rates of open science principle are higher than in other fields and the journals policies were successful in fostering adherence to open science principles. A general culture of openness to reproducing analyses and replication of findings was established, which allows for a cumulative development of knowledge to which also student research can contribute.