Register for an account


Enter your name and email address below.

Your email address is used to log in and will not be shared or sold. Read our privacy policy.


Website access code

Enter your access code into the form field below.

If you are a Zinio, Nook, Kindle, Apple, or Google Play subscriber, you can enter your website access code to gain subscriber access. Your website access code is located in the upper right corner of the Table of Contents page of your digital edition.


The F Problem With The P-Value Sciences

Neuroskeptic iconNeuroskepticBy NeuroskepticOctober 17, 2013 2:00 AM


Sign up for our email newsletter for the latest science news

There is a problem in science today. I've written a lot about how to cure it, but in this post I want to outline the nature of the disease as I see it. The problem goes by many names:

So I'm going to call it the f problem for short. I like to visualize f as a forking path. Given any particular set of raw data, a researcher faces a series of choices about how to turn it into a 'result'. There are choices over which statistical tests to run, on which variables, after excluding which outliers, and applying which preprocessing... and so on:


The f problem is that researchers can try multiple approaches in private, and select for publication the most desirable ones. Most often, it's statistically significant effects, that match with prior hypotheses, that are desired. Even if there are no real effects of interest in the data, some comparisons will be 'positive' just by chance. Researchers today face pressure to publish 'good results' and are rewarded for doing so - this is what turns f from a theoretical concern into a real one. Yet f is not a problem for all of science. Broadly, f only affects research in which the results take the form of p-values. Thus fields like mathematics, where results are proofs, are immune. But even some p-value sciences manage to escape f. This is because in these fields, the nature of the enterprise means that 'everyone knows' what experiments are being carried out, and how the data ought to be analyzed - in advance. In particle physics, for example, it was public knowledge that CERN was looking for the Higgs Boson, and it was also known exactly how they planned to look for it in the data. Openness meant that was no room for f, because f is all about the scope for hidden flexibility.

    3 Free Articles Left

    Want it all? Get unlimited access when you subscribe.


    Already a subscriber? Register or Log In

    Want unlimited access?

    Subscribe today and save 50%


    Already a subscriber? Register or Log In