Publication Bias Examined in Meta-Analyses from Psychology and Medicine: A Meta-Meta-Analysis

Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8% of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. Publication bias tests did not reveal evidence for bias in the homogeneous subsets. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. However, a Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). Our findings are in line with, in its most extreme case, publication bias ranging from no bias until only 5% statistically nonsignificant effect sizes being published. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9% and 18.9% for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medicine.

发表偏差对于研究整体的可信度，尤其是对于系统评价而言，构成了一项重大问题，因为它导致了效果的过度估计，并可能暗示了非存在的效应的存在。尽管普遍认同发表偏差的存在，但目前对于其如何影响不同科学文献的了解尚不充分。我们对纳入《心理通报》发表的83篇系统评价以及《Cochrane系统评价数据库》中的499篇系统评价的初级研究的大型数据集进行了发表偏差的证据考察（这些系统评价代表了心理学领域的系统评价）。由于发表偏差的方法在真实效应量异质的情况下缺乏良好的统计特性，因此我们对系统评价中包含的所有同质性子集（占《心理通报》发表的系统评价子集的3.8%）进行了发表偏差的评估。发表偏差测试并未在同质性子集中揭示偏差的证据。过度估计的现象最小，但具有统计学意义，为发表偏差的存在提供了证据，这种偏差似乎在两个领域都存在。然而，一项蒙特卡洛模拟研究表明，创建同质性子集为发表偏差方法带来了挑战，因为子集中效应量的数量相对较小（效应量数量的中位数等于6）。我们的发现与最极端的情况相一致，即从无偏差到只有5%的统计上非显著效应量被发表。这些以及其他发现，结合统计上显著的主要效应量百分比（分别为《心理通报》和CDSR发表的子集中的28.9%和18.9%），导致我们得出结论：在研究中的同质性子集中发表偏差的证据薄弱，但暗示了心理学和医学中都存在轻微的发表偏差。