Four years ago, neuroscientists became aware of an ominous-sounding manuscript entitled "Voodoo Correlations In Social Neuroscience". This piece was eventually published under a more prosaic name but it still hit home, with nearly 500 citations so far. To me, this paper marked the start of a new era of 'critical' (in the proper sense of thoughtful discussion and reflection) neuroscience, with fMRI researchers becoming more aware that fundamental statistics are as important as ever, despite the amazing technical advances and novel techniques of the 1990s and 2000s.
Now London neuroscientist James Kilner has reminded us that the 'voodoo problem' applies not only to fMRI but also to EEG and MEG, methods for measuring brain electro-magnetic activity: Bias in a common EEG and MEG statistical analysis and how to avoid it The problem, in essence, is about selecting values out of a random population. If you apply a selection criteria to lots of random variables, and pick out only the highest (or lowest) values, then any statistical tests you run on those picked values will probably be biased because they're selected. It sounds simple, but in a complex data analysis, it's surprising how easy it is to select and test without realizing it. In the case of EEG and MEG, the recorded data consists of anywhere from 20 to 250 sensors or electrodes placed around the head. It's not clear however which sensors are most 'interesting' in any given experiment. A common practice is to focus on the electrode at which the largest electrical or magnetic response (ERP) is seen to a given stimulus, but Kilner shows that this is dangerous unless care is taken to make the selection criteria independent of the subsequent analysis. Selection itself is fine, but 'double dipping' or 'circular' testing of the same things that were used as selection criteria (e.g. testing the size of the ERP at the electrode where that ERP is largest) is problematic.
Kilner, J. (2013). Bias in a common EEG and MEG statistical analysis and how to avoid it Clinical Neurophysiology DOI: 10.1016/j.clinph.2013.03.024
