Mind

The F Problem With The P-Value Sciences

Neuroskeptic iconNeuroskepticBy NeuroskepticOct 16, 2013 9:00 PM

Newsletter

Sign up for our email newsletter for the latest science news
 

There is a problem in science today. I've written a lot about how to cure it, but in this post I want to outline the nature of the disease as I see it. The problem goes by many names:

So I'm going to call it the f problem for short. I like to visualize f as a forking path. Given any particular set of raw data, a researcher faces a series of choices about how to turn it into a 'result'. There are choices over which statistical tests to run, on which variables, after excluding which outliers, and applying which preprocessing... and so on:

The f problem is that researchers can try multiple approaches in private, and select for publication the most desirable ones. Most often, it's statistically significant effects, that match with prior hypotheses, that are desired. Even if there are no real effects of interest in the data, some comparisons will be 'positive' just by chance. Researchers today face pressure to publish 'good results' and are rewarded for doing so - this is what turns f from a theoretical concern into a real one. Yet f is not a problem for all of science. Broadly, f only affects research in which the results take the form of p-values. Thus fields like mathematics, where results are proofs, are immune. But even some p-value sciences manage to escape f. This is because in these fields, the nature of the enterprise means that 'everyone knows' what experiments are being carried out, and how the data ought to be analyzed - in advance. In particle physics, for example, it was public knowledge that CERN was looking for the Higgs Boson, and it was also known exactly how they planned to look for it in the data. Openness meant that was no room for f, because f is all about the scope for hidden flexibility.

1 free article left
Want More? Get unlimited access for as low as $1.99/month

Already a subscriber?

Register or Log In

1 free articleSubscribe
Magazine Examples
Want more?

Keep reading for as low as $1.99!

Subscribe

Already a subscriber?

Register or Log In

More From Discover
Recommendations From Our Store
Shop Now
Stay Curious
Join
Our List

Sign up for our weekly science updates.

 
Subscribe
To The Magazine

Save up to 70% off the cover price when you subscribe to Discover magazine.

Copyright © 2021 Kalmbach Media Co.