False-Positive fMRI Hits The Mainstream

Neuroskeptic iconNeuroskeptic
By Neuroskeptic
Jul 7, 2016 7:37 PMNov 20, 2019 1:19 AM

Newsletter

Sign up for our email newsletter for the latest science news
 

A new paper in PNAS has made waves. The article, called

Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates,

comes from Swedish neuroscientists Anders Eklund, Tom Nichols, and Hans Knutsson. According to many of the headlines that greeted "Cluster failure", the paper is a devastating bombshell that could demolish the whole field of functional magnetic resonance imaging (fMRI):

Bug in fMRI software calls 15 years of research into question (Wired) A bug in fMRI software could invalidate 15 years of brain research. This is huge. (ScienceAlert) New Research Suggests That Tens Of Thousands Of fMRI Brain Studies May Be Flawed (Motherboard)

So what's going on here, and is it really this serious? The first thing to note is that the story isn't really new. I've been covering Eklund et al.'s work on the false-positive issue since 2012 (1,2,3,4). Over that time, Eklund and his colleagues have developed the argument that several commonly used fMRI analysis software tools suffer from a basic flaw which leads to elevated false-positive rates when it comes to finding activations associated with tasks or stimuli i.e. finding which brain area 'lights up' during particular tasks. The new paper is just the culmination of this program, and the results - that up to 70% of analyses produce at least one false positive, depending on the software and conditions - won't come as a surprise to anyone who has been following the issue Although there is one unexpected point in "Cluster failure": Eklund et al. reveal that they discovered a different kind of bug in one of the software packages, called AFNI:

A 15-y-old bug was found in [AFNI's tool] 3dClustSim while testing the three software packages (the bug was fixed by the AFNI group as of May 2015, during preparation of this manuscript). The bug essentially reduced the size of the image searched for clusters, underestimating the severity of the multiplicity correction and overestimating significance (i.e., 3dClustSim FWE P values were too low)

This is a new and important issue, but this new bug only applies to AFNI, not other widely-used packages such as FSL and SPM. As to the question of how serious this is, in my view, it's very serious, but it doesn't "invalidate 15 years of brain research" as the headline had it. For one thing, the issue only affects fMRI, and most brain research does not use fMRI. Moreover, Eklund et al.'s findings don't call all fMRI studies into question - the problem only affects activation mapping studies. Yet while these experiments are common, they are far from the only application of fMRI. Studies of functional connectivity or multi-voxel pattern analysis (MVPA) are increasingly popular and they're not, as far as I can see, likely to be affected. Finally, it's important to remember that "70% chance of finding at least one false positive" does not imply that "70% of positives are false". If there are lots of true positives, only a minority of positives will be false. It's impossible to directly know the true positive rate, however. Update 15th July 2016: Tom Nichols, one of the authors of 'Cluster Failure', reports that he's requested some corrections to the paper in order to remove some of the statements that led to "misinterpretations" of the study (i.e. to those hyped headlines). However, PNAS did not agree to the correction, so Nichols has posted it on PubMed Commons, here.

Eklund A, Nichols TE, & Knutsson H (2016). Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Proceedings of the National Academy of Sciences of the United States of America PMID: 27357684

1 free article left
Want More? Get unlimited access for as low as $1.99/month

Already a subscriber?

Register or Log In

1 free articleSubscribe
Discover Magazine Logo
Want more?

Keep reading for as low as $1.99!

Subscribe

Already a subscriber?

Register or Log In

More From Discover
Recommendations From Our Store
Stay Curious
Join
Our List

Sign up for our weekly science updates.

 
Subscribe
To The Magazine

Save up to 40% off the cover price when you subscribe to Discover magazine.

Copyright © 2024 Kalmbach Media Co.