Register for an account

X

Enter your name and email address below.

Your email address is used to log in and will not be shared or sold. Read our privacy policy.

X

Website access code

Enter your access code into the form field below.

If you are a Zinio, Nook, Kindle, Apple, or Google Play subscriber, you can enter your website access code to gain subscriber access. Your website access code is located in the upper right corner of the Table of Contents page of your digital edition.

Mind

"Cluster Failure": fMRI False Positives Revisited

Neuroskeptic iconNeuroskepticBy NeuroskepticJuly 22, 2018 6:31 PM

Newsletter

Sign up for our email newsletter for the latest science news

Two years ago, a paper by Swedish neuroscientist Anders Eklund and colleagues caused a media storm. The paper, Cluster Failure, reported that the most widely used methods for the analysis of fMRI data are flawed and produce a high rate of false positives. As I said at the time, Cluster Failure wasn't actually making especially new claims because Eklund et al. had been publishing quite similar results years earlier - but it wasn't until Cluster Failure that they attracted widespread attention.

brainquest1.jpg

Perhaps Cluster Failure went mainstream is that it was the first of Eklund et al.'s false positive papers to be published in a high-impact journal (PNAS). But another reason is that it contained an alarming statement, namely that "These results question the validity of some 40,000 fMRI studies." This triggered many headlines implying that all of fMRI was suspect. Tom Nichols, one of the Cluster Failure authors, later clarified that 40,000 referred to the total number of fMRI studies out there, and wasn't meant to imply that all of those studies were invalid. He went on to estimate that about 10% of the 40,000 fMRI experiments were at high risk of false positives from the Cluster Failure problem, although another 33% suffer from a different problem (no multiple comparisons correction at all.) Now, Eklund et al. have released a biorXiv preprint looking back on their much-discussed 2016 paper: Cluster Failure Revisited: Impact of First Level Design and Data Quality on Cluster False Positive Rates In the new article, Eklund et al. consider various technical critiques of their previous work, but conclude that they are unfounded. In response to concerns that the high false positive rates in Cluster Failure were a result of "idiosyncratic attributes of our first level designs", they show that elevated false positive rates are also seen with modified designs (first level models with two regressors, and models with two intersubject-randomized regressors). Eklund et al. also further explore how best to analyze fMRI data to avoid false positives. After trying several different approaches, they conclude that a combination of two things is required: non-parametric statistical thresholding, and noise reduction (using ICA FIX). Finally, Eklund et al. revisit the question of how many fMRI papers may be compromised by sub-optimal analysis. They reiterate that about 10% of fMRI papers used a multiple comparisons correction that Cluster Failure showed to be most problematic (p<0.01 cluster defining threshold), and say that any marginally significant results obtained by this method (p close to 0.05) should be "judged with great skepticism". But studies that used no multiple comparisons correction at all are even more dubious, the authors say. In my view, for all of the hyperbole it created, Cluster Failure was a great paper and highlights an issue that shouldn't be ignored. Yet I wonder whether new and different problems may be on the horizon. New analysis tools for fMRI have emerged in the past few years, which - unlike earlier methods - aren't based on mapping clusters of brain activity. Popular new methods include MVPA and network-based analyses. These approaches are exciting and impressive and they are not (as far as I know) subject to the problems identified in Cluster Failure. But then again, they are new enough that they might have undiscovered issues of their own. In 2026, will we be worrying about the implications of a bombshell paper called Network Failure...?

    2 Free Articles Left

    Want it all? Get unlimited access when you subscribe.

    Subscribe

    Already a subscriber? Register or Log In

    Want unlimited access?

    Subscribe today and save 70%

    Subscribe

    Already a subscriber? Register or Log In