Looking for evidence that publish-or-perish reinforces publication bias and gamification of publishing strategy

This study analyzes the reliability of articles published by Polish researchers between the years 2017 and 2021 in search for the evidence of spurious publishing incentives related to the ministerial requirements. In 2017, a legislation was passed which made it a requirement for each psychological institute to present 3 publications per scientist, but a maximum of 4 authored by one person employed in the facility at the end of the evaluation period (2017-2021). The institutes have been rated on the basis of a narrow set of criteria scores for publications. Journals’ scores were based on the ministerial list, which assigns specific scores to chosen journals; and are loosely associated with their IF or SJR.

We propose interpreting this situation as a quasi-experimental setup, where researchers, particularly those with fewer publications, are encouraged to publish more articles in well-rated journals.

To analyze the articles, we have first scraped the ORCID profiles of the scientists employed in the psychological institutes that took part in the evaluation. Based on these ORCIDs, we have collected articles that were published in these institutions in the period of interest, along with the journals in which they were published. Due to numerous deficiencies and potential reporting bias in the ORCID dataset (having an ORCID account was obligatory only for a short period), further extension for another dataset is planned.

The statistics related to significance will be extracted from the text using the statcheck software (Epskamp, S. & Nuijten, M. B. (2016). statcheck: Extract statistics from articles and recompute p values. Retrieved from http://CRAN.R-project.org/package=statcheck. (R package version 1.2.2)) and used as input into the z-curve software (Bartoš, F., & Schimmack, U. (2022). Z-curve 2.0: Estimating replication rates and discovery rates. Meta-Psychology, 6.). The z-curve analysis will be performed separately for compared groups rearing Observed Discover Rate (ODR) and Expected Discovery Rate (EDR) scores. We will also analyze the distribution of papers across the different journals, and its relation to the scores on the ministerial list, as well as whether the studies were pre registered, and what statistical analysis they used. We also consider extending the analysis to include articles’ focal hypotheses.