Detecting Evidential Value and P-Hacking With the P-curve tool: A Word of Caution

Simonsohn, Nelson, and Simmons (2014a) proposed p-curve – the distribution of statistically significant p-values for a set of studies – as a tool to assess the evidential value of these studies. They argued that, whereas right-skewed p-curves indicate true underlying effects, left-skewed p-curves indicate selective reporting of significant results from a much larger set of tests conducted on the same data when there is no true effect (“p-hacking”). We first review research that criticized the first claim by showing that null effects may indeed produce right-skewed p-curves under some conditions. We then question the second claim by showing that not only selective reporting but also selective non-reporting of significant results (e.g., of an ANCOVA for randomized 2-groups designs) due to a significant outcome of a more popular alternative test of the same hypothesis (e.g., a two-group t-test) may produce left-skewed p-curves, even if all studies included in a p-curve reflect true effects. Thus, although it is true that left-skewed p-curves indicate selection bias, it is possible that the bias is due to studies excluded from the p-curve rather than to those included in it. Hence, just as right-skewed p-curves do not necessarily imply evidential value, left-skewed p-curves do not necessarily imply p-hacking and absence of true effects in the studies involved.