pvaluesim.m

<div>These are matlab files to show the effect of p-hacking through a visual simulation. It simulates a natural experiments with two groups with the default of n = 20 each, both drawn from the standard normal distribution. Default is 1000 simulations and the first 15 are visualized. The 95% confidence intervals of the difference between the two group means are shown (green means that zero is included in the 95% confidence interval, red means that zero is not included in the 95% confidence interval). It is recorded how often in the simulation the null hypothesis is rejected at a 5% level.<div><br></div><div>The effect of the following four p-hacking cheats can be simulated:</div><div><br></div><div>A: Not doing the test once, but test two for two variables and their mean.</div><div>B: If no no significant effect is found, add 10 cases to each of the experimental groups</div><div>C: Add tests with a covariate (such as gender) and a test of an interaction effect of gender and the experimental condition.</div><div>D: Add a third experimental condition and repeat the tests for any pair of experimental condition. <br><div><br></div><div>In matlab, run<div><br></div><div>pvaluesim</div><div><br></div><div>to start the GUI. Note that higher values than 5% are found because essentially multiple tests are done without correcting for the number of tests.</div><div><br></div><div>This simulation is the implementation of:</div><div><br></div><div>Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. <i>Psychological science</i>, <i>22</i>(11), 1359-1366.</div></div></div></div>