Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size

BackgroundThe p value obtained from a significance test provides no information about the magnitude or importance of the underlying phenomenon. Therefore, additional reporting of effect size is often recommended. Effect sizes are theoretically independent from sample size. Yet this may not hold true empirically: non-independence could indicate publication bias.MethodsWe investigate whether effect size is independent from sample size in psychological research. We randomly sampled 1,000 psychological articles from all areas of psychological research. We extracted p values, effect sizes, and sample sizes of all empirical papers, and calculated the correlation between effect size and sample size, and investigated the distribution of p values.ResultsWe found a negative correlation of r = −.45 [95% CI: −.53; −.35] between effect size and sample size. In addition, we found an inordinately high number of p values just passing the boundary of significance. Additional data showed that neither implicit nor explicit power analysis could account for this pattern of findings.ConclusionThe negative correlation between effect size and samples size, and the biased distribution of p values indicate pervasive publication bias in the entire field of psychology.

研究背景
显著性检验所得的p值，无法反映所研究现象的效应量大小与实际重要性。因此，学界通常建议补充报告效应量（effect size）。从理论层面而言，效应量与样本量相互独立。但在实证研究中这一结论未必成立：二者若不独立，可能提示存在发表偏倚。

研究方法
本研究旨在探究心理学研究中效应量与样本量是否相互独立。我们从心理学各研究领域中随机抽取了1000篇心理学学术论文，提取了所有实证研究论文的p值、效应量与样本量，计算了效应量与样本量之间的相关系数，并分析了p值的分布特征。

研究结果
结果显示，效应量与样本量之间存在显著负相关，相关系数r = −0.45，95%置信区间为[−0.53, −0.35]。此外，我们发现恰好达到显著性临界值的p值数量异常偏高。进一步分析表明，无论是隐式还是显式功效分析（power analysis），均无法解释该研究结果的分布模式。

研究结论
效应量与样本量的负相关关系，以及p值分布的偏倚特征，均表明心理学整个领域普遍存在发表偏倚。