P-Hacking: A Talk and Further Thoughts

A week ago I gave a talk to Marcus Munafo's group at the University of Bristol on the subject of P-hacking. The presentation is now available online. If you don't have time to watch the whole thing, I'd recommend the section from 14:50 to 22:00 - that's my live demonstration of how, with a few minutes of P-hacking, you can find a nicely significant correlation ( p = 0.006 ) between two variables that have no relationship whatsoever. [embed]https://www.youtube.com/watch?v=A0vEGuOMTyA[/embed] Here's a few further thoughts on this topic. As I noted in my talk, there are various proposed methods of detecting p-hacking within a set of studies or a whole literature. These methods tend to rely on looking at the distribution of published p-values, or p-curve. Assuming that there really is a significant effect, p-values should be clustered around 0. Under the null hypothesis of no effect, p-values are evenly distributed ...