A new paper could prompt a rethink of a technique that's become very hot in neuroscience lately: Confounds in multivariate pattern analysis The authors are Princetonians Michael T. Todd and colleagues, and the method in question is multivariate pattern analysis (MVPA). I've written about this before and there's a blog dedicated to it. MVPA searches for relatively subtle patterns of brain activity, most commonly in fMRI data. For example, a conventional fMRI study might compare activity when someone's looking at a picture, compared to a blank screen, and would find increases of activity in the visual cortex. But MVPA might take two different pictures, and see if there's a pattern of activity that's unique to one picture over the other - even if overall activity in the visual cortex is the same. Neuroscientists have fallen in love with MVPA (and related methods) over the past 5 years, mainly I think because it's promised to let us 'read' the brain: to not just see where in the brain things happen, but to glimpse what information is being represented. In the new paper, Todd et al make a very simple point: all MVPA really shows is that there are places where, in most people's brain, activity differs when they're doing one thing as opposed to another. But there infinite reasons why that might be the case, many of them rather trivial. The authors give the example of two very similar tasks, A and B. We'll say these are imagining apples and imagining bananas. You scan some people doing A and B. You run a standard fMRI analysis, and find that nowhere in the brain shows a difference in activity, on average, between the two (as expected - they are similar.) So you do an MVPA analysis, and now you find an area where the pattern of activity predicts task. Hooray! You've found the part of the brain that encodes the nature of fruit, or something else exciting. Not so fast. The problem is that it might be that some people (say) ate an apple for breakfast that morning and find imagining an apple easier, while others had a banana, and find apples harder. In which case, a part of the brain that lights up when you're concentrating hard on something (and there's a lot of those areas) would light up differently based on task, and would discriminate between the two within a given individual. Here's a hypothetical signal from a brain area that lights up when you're working hard:
Generally, the problem is that MVPA will detect a signal from any area of the brain where activity differs according to how each individual performs the task, but this signal might be rather uninteresting. What's scary is that this problem is insidious. It could easily go undetected. Because if some people find A hard, and others B hard, they wouldcancel out in the average. It would be very easy to look at the average and conclude that difficulty was just not a factor in your experiment. But MVPA by its very nature looks at individual-specific things and doesn't let them cancel out. It's one of those things that's so obvious in retrospect. And it's not just difficulty. If some people find some of your stimuli (let's say) more boring, fun, unpleasant, attractive, or annoying than others, that could show up too. Todd et al show this to be more than theoretical. In an fMRI dataset involving two tasks equally difficult on average, they show that MVPA detects several areas where activity patterns are correlated with task, but this disappears if you control for each subject's reaction times, a proxy for how difficult they found it individually. Such a correction is rarely performed.
The authors conclude by saying that this is not a problem with MVPA, as such. The mathematics are sound and the method does 'work', but the trouble is what the results mean. As they write:
Our concern is limited to
cases in which a brain region revealed by MVPA is interpreted as representing a particular cognitive variable or type of information e.g., “Brain region A represents information X”.
In contrast, claims of the following form are not problematic: “Brain region A can predict behavior Y”
Todd, M., Nystrom, L., & Cohen, J. (2013). Confounds in multivariate pattern analysis: Theory and rule representation case study NeuroImage DOI: 10.1016/j.neuroimage.2013.03.039
