Register for an account

X

Enter your name and email address below.

Your email address is used to log in and will not be shared or sold. Read our privacy policy.

X

Website access code

Enter your access code into the form field below.

If you are a Zinio, Nook, Kindle, Apple, or Google Play subscriber, you can enter your website access code to gain subscriber access. Your website access code is located in the upper right corner of the Table of Contents page of your digital edition.

Mind

Neuroscience Fails Stats 101?

Neuroskeptic iconNeuroskepticBy NeuroskepticSeptember 12, 2011 2:36 AM

Newsletter

Sign up for our email newsletter for the latest science news

According to a new paper, a full half of neuroscience papers that try to do a (very simple) statistical comparison are getting it wrong: Erroneous analyses of interactions in neuroscience: a problem of significance.

Here's the problem. Suppose you want to know whether a certain 'treatment' has an affect on a certain variable. The treatment could be a drug, an environmental change, a genetic variant, whatever. The target population could be animals, humans, brain cells, or anything else.

So you give the treatment to some targets and give a control treatment to others. You measure the outcome variable. You use a t-test of significance to see whether the effect is large enough that it wouldn't have happened by chance. You find that it was significant.

That's fine. Then you try a different treatment, and it doesn't cause a significant effect against the control. Does that mean the first treatment was more powerful than the second?

No. It just doesn't. The only way to find that out would be to compare the two treatments directly - and that would be very easy to do, because you have all the data to hand. If you just compare the two treatments to control you might end up with this scenario:

Both treatments are very similar but one (B) is slightly better so it's significantly different from control, while A isn't. But they're basically the same. It's probably just fluke that B did slightly better than A. If you compared A and B directly you'd find they were not significantly different.

placeholder

An analogy: Passing a significance test is like winning a prize. You can only do it if you're much better than the average. But that doesn't mean you're much better than everyone who didn't win the prize, because some of them will have almost been good enough.

Usain Bolt is the fastest man in the world (when he's not false-starting himself out of races). Much faster than me. But he's not much faster than the second fastest man in the world.

rb2_large_white.png

Nieuwenhuis S, Forstmann BU, & Wagenmakers EJ (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nature neuroscience, 14 (9), 1105-7 PMID: 21878926

    2 Free Articles Left

    Want it all? Get unlimited access when you subscribe.

    Subscribe

    Already a subscriber? Register or Log In

    Want unlimited access?

    Subscribe today and save 70%

    Subscribe

    Already a subscriber? Register or Log In