The Reproducibility in Reliability Generalization Meta-Analysis

Background One of the principles of scientific method is the ability of an experiment to be reproducible. This means that the results of a study should be the same if the research is repeated by other researchers, following the same procedure. It is essential that the empirical findings be replicable by other researchers. In recent years, the reproducibility of the experimental results has been questioned by many studies. This problem has become increasingly evident, especially regarding to research fields such as psychology. Objectives Considering all this information, our aim is to examine how the reproducibility drama affects into our area of study: The Reliability Generalization Meta-Analysis. In order to do it, we want to prove whether the results of RG meta-analysis can be reproduced following the described procedure in each meta-analysis and applying better analytic strategies. Method At the current stage where we are, we have collected 145 databases of reliability generalization studies which presented the data of primary studies to repeat the described analysis in the paper, proving so the degree of concordance between those results and ours. Our reproducibility study has focused on the average Alpha Coefficient, its 95% confidence interval and the heterogeneity I2 index. These analyses have been carried out, essentially, with metafor package of R. In some cases, we have also used the same software presented by the authors of the paper. Results Through this research we expect to obtain that the results of RG meta-analysis are reproducible. We expect to find that our results and the previous ones are practically identical. Moreover, we expect that much of the reproducibility problems are due to a lack of transparency in specifying the analytical strategy that has been chosen to be used. The results of our reproducibility study are discussed, as well as their implications for improving the reporting practices of reliability generalization meta-analyses. This research has been funded with a grant from the Ministerio de Ciencia e Innovación of the Spanish Government and by FEDER funds (project no PID2019-104080GB-I00). unknown unknown