Post-hoc power analyses of Head et al. (2015).

We extend questionable research practices (QRPs) research by conducting a robust, large-scale analysis of p-hacking in organizational research. We leverage a manually curated database of more than 1,000,000 correlation coefficients and sample sizes, with which we calculate exact p-values. We test for the prevalence and magnitude of p-hacking across the complete database as well as various subsets of the database according to common bivariate relation types in the organizational literature (e.g., attitudes-behaviors). Results from two analytical approaches (i.e., z-curve, critical bin comparisons) were consistent in both direction and significance in nine of 18 datasets. Critical bin comparisons indicated p-hacking in 12 of 18 subsets, three of which reached statistical significance. Z-curve analyses indicated p-hacking in 11 of 18 subsets, two of which reached statistical significance. Generally, results indicated that p-hacking is detectable but small in magnitude. We also tested for three predictors of p-hacking: Publication year, journal prestige, and authorship team size. Across two analytic approaches, we observed a relatively consistent positive relation between p-hacking and journal prestige, and no relationship between p-hacking and authorship team size. Results were mixed regarding the temporal trends (i.e., evidence for p-hacking over time). In sum, the present study of p-hacking in organizational research indicates that the prevalence of p-hacking is smaller and less concerning than earlier research has suggested.

本研究拓展了可疑研究操作（questionable research practices, QRPs）领域的研究范畴，针对组织学研究中的p值操纵现象开展了大规模、严谨的系统性分析。本研究依托人工整理的、包含超100万条相关系数与样本量的数据库，据此计算精确p值。本研究针对完整数据库，以及依据组织学文献中常见双变量关系类型（如态度-行为关系）划分的各类数据库子集，检验了p值操纵的流行程度与影响强度。两种分析方法——即z曲线（z-curve）与临界组比较（critical bin comparisons）——的结果显示，在18个数据集中有9个在效应方向与显著性水平上保持一致。临界组比较分析显示，18个子集中有12个存在p值操纵现象，其中3个达到统计学显著性水平。z曲线分析则表明，18个子集中有11个存在p值操纵现象，其中2个达到统计学显著性水平。总体而言，研究结果显示p值操纵现象可被检测出，但其影响程度较为有限。本研究同时检验了三类p值操纵的预测因素：发表年份、期刊声望以及作者团队规模。基于两种分析方法，本研究均观察到p值操纵与期刊声望之间存在相对稳定的正向关联，而p值操纵与作者团队规模之间则无显著关联。关于时间趋势（即p值操纵随时间的变化规律）的研究结果则存在分歧。综上，本项针对组织学研究中p值操纵现象的研究表明，p值操纵的流行程度低于此前相关研究的结论，且其受关注程度也相对更低。