We have completed maintenance on DiscoverMagazine.com and action may be required on your account. Learn More

"Cluster Failure": fMRI False Positives Revisited

Neuroskeptic iconNeuroskeptic
By Neuroskeptic
Jul 22, 2018 6:31 PMNov 20, 2019 12:51 AM


Sign up for our email newsletter for the latest science news

Two years ago, a paper by Swedish neuroscientist Anders Eklund and colleagues caused a media storm. The paper, Cluster Failure, reported that the most widely used methods for the analysis of fMRI data are flawed and produce a high rate of false positives. As I said at the time, Cluster Failure wasn't actually making especially new claims because Eklund et al. had been publishing quite similar results years earlier - but it wasn't until Cluster Failure that they attracted widespread attention.

Perhaps Cluster Failure went mainstream is that it was the first of Eklund et al.'s false positive papers to be published in a high-impact journal (PNAS). But another reason is that it contained an alarming statement, namely that "These results question the validity of some 40,000 fMRI studies." This triggered many headlines implying that all of fMRI was suspect. Tom Nichols, one of the Cluster Failure authors, later clarified that 40,000 referred to the total number of fMRI studies out there, and wasn't meant to imply that all of those studies were invalid. He went on to estimate that about 10% of the 40,000 fMRI experiments were at high risk of false positives from the Cluster Failure problem, although another 33% suffer from a different problem (no multiple comparisons correction at all.) Now, Eklund et al. have released a biorXiv preprint looking back on their much-discussed 2016 paper: Cluster Failure Revisited: Impact of First Level Design and Data Quality on Cluster False Positive Rates In the new article, Eklund et al. consider various technical critiques of their previous work, but conclude that they are unfounded. In response to concerns that the high false positive rates in Cluster Failure were a result of "idiosyncratic attributes of our first level designs", they show that elevated false positive rates are also seen with modified designs (first level models with two regressors, and models with two intersubject-randomized regressors). Eklund et al. also further explore how best to analyze fMRI data to avoid false positives. After trying several different approaches, they conclude that a combination of two things is required: non-parametric statistical thresholding, and noise reduction (using ICA FIX). Finally, Eklund et al. revisit the question of how many fMRI papers may be compromised by sub-optimal analysis. They reiterate that about 10% of fMRI papers used a multiple comparisons correction that Cluster Failure showed to be most problematic (p<0.01 cluster defining threshold), and say that any marginally significant results obtained by this method (p close to 0.05) should be "judged with great skepticism". But studies that used no multiple comparisons correction at all are even more dubious, the authors say. In my view, for all of the hyperbole it created, Cluster Failure was a great paper and highlights an issue that shouldn't be ignored. Yet I wonder whether new and different problems may be on the horizon. New analysis tools for fMRI have emerged in the past few years, which - unlike earlier methods - aren't based on mapping clusters of brain activity. Popular new methods include MVPA and network-based analyses. These approaches are exciting and impressive and they are not (as far as I know) subject to the problems identified in Cluster Failure. But then again, they are new enough that they might have undiscovered issues of their own. In 2026, will we be worrying about the implications of a bombshell paper called Network Failure...?

1 free article left
Want More? Get unlimited access for as low as $1.99/month

Already a subscriber?

Register or Log In

1 free articleSubscribe
Discover Magazine Logo
Want more?

Keep reading for as low as $1.99!


Already a subscriber?

Register or Log In

More From Discover
Recommendations From Our Store
Shop Now
Stay Curious
Our List

Sign up for our weekly science updates.

To The Magazine

Save up to 40% off the cover price when you subscribe to Discover magazine.

Copyright © 2024 Kalmbach Media Co.